A driver gaze behavior safety reminder method and device suitable for smart cockpit

Abstract

The present invention provides a method and device for driver gaze behavior safety reminders applicable to smart cockpits, comprising the following steps: S1: constructing a human-computer interaction dataset for a smart cockpit touch screen; S2: extracting time series data features; S3: preprocessing the data and constructing a time series matrix; S4: constructing a hybrid deep model CNN-BiLSTM-Attention based on BiLSTM, CNN, and attention mechanism; S5: training, testing, tuning, and integrating CNN-BiLSTM-Attention; and S6: executing a gaze behavior safety reminder method based on the prediction results of the gaze area. Beneficial effect: Based on the prediction results of the driver's gaze area, distraction theory, and safe driving theory, the timing of sending safety reminder information is determined, and the driver is actively interacted with through visual and voice information to prompt the driver to regulate his or her gaze behavior. This method is targeted at smart cockpit full touch screen interaction scenarios, can predict the driver's gaze area, actively interact with the driver to improve driving safety, and can effectively avoid human-computer conflicts.

Description

Technical Field

[0001] The present invention relates to a method and device for reminding a driver's gaze behavior safety in an intelligent cockpit, and belongs to the technical field of road traffic safety. Background Art

[0002] Driver gaze behavior is a key concern in the field of road traffic safety. While existing methods exist for monitoring driver gaze behavior, these methods focus on real-time monitoring and do not predict it and proactively interact with it to improve safety. If an autonomous driving system can accurately predict the driver's gaze area during full touchscreen interaction and proactively issue safety alerts when necessary, prompting the driver to promptly redirect their attention to the road, potential driving risks can be mitigated. Furthermore, in the future era of human-machine co-driving, predicting the driver's gaze area will help intelligent driving systems make the right decisions about when to take over control.

[0003] With the increasing intelligence of vehicles, large touch screens have become the main interface for human-vehicle interaction. This development has created more possibilities for drivers to perform visual interaction tasks. When performing touch screen interaction tasks, the driver's attention will switch back and forth between the touch screen and the road. At the same time, the driver also needs to physically interact by clicking or sliding the controls on the touch screen to keep his operations consistent with the visual focus. This increase in visual and manual workload will aggravate visual and manual distractions, resulting in the driver's inability to perceive potential dangers on the road, thereby causing serious traffic accidents. Therefore, the present invention focuses on the above-mentioned problems and proposes a driver's gaze behavior safety reminder method and device suitable for smart cockpits, which aims to achieve accurate prediction of the driver's attention area, gain insight into the driver's behavior in advance, and actively interact with the driver at the appropriate time and send safety reminder information. Summary of the Invention

[0004] Purpose of the invention: In view of the deficiencies in the prior art, the present invention provides a method and device for providing a safety reminder of the driver's gaze behavior in a smart cockpit, so as to solve the problems mentioned in the above background technology.

[0005] Technical solution: A driver gaze behavior safety reminder method applicable to a smart cockpit, comprising the following steps:

[0006] S1: Constructing a dataset of human-computer interaction for smart cockpit touch screens;

[0007] S2: time series data feature extraction;

[0008] S3: Data preprocessing and constructing time series matrix;

[0009] S4: Construct a hybrid deep model CNN-BiLSTM-Attention based on BiLSTM, CNN and attention mechanism;

[0010] S5: CNN-BiLSTM-Attention training, testing, tuning and integration;

[0011] S6: Execute the gaze behavior safety reminder method based on the prediction result of the gaze area.

[0012] The step S1 is specifically as follows:

[0013] S1.1: Select N0 models with a full-touch screen smart cockpit;

[0014] S1.2: Design a touch screen interaction experiment and select N1 interactive tasks based on the vehicle's interactive functions;

[0015] S1.3: Select N2 drivers of different genders, driving experience, and ages; each driver performs each similar interactive task N3 times; design N4 actual driving routes, each of which has N5 different speeds; the total number of interactive task samples obtained is N0×N1×N2×N3×N4×N5=N T ;

[0016] S1.4: The dual-source camera monitoring system is used to collect the driver's facial image and hand image respectively; the vehicle data collector is used to collect the driver's operation data and vehicle motion state parameters and save them as text data.

[0017] The step S2 is specifically as follows:

[0018] S2.1: Embed visual feature extraction models into the microprocessor: the token-based head pose estimator TokenHPE, the gaze region recognition model iTracker, and the hand keypoint extraction model MediaPipe-Hands;

[0019] S2.2: Extract N from the original data of step S1 T Interactive task process fragments;

[0020] S2.3: Use the data acquisition unit to collect and store data, and send it to the microprocessor unit. In the microprocessor unit, head posture, gaze area, and hand key points are analyzed for each interaction sample;

[0021] S2.4: In the microprocessor, a token-based head pose estimator (TokenHPE) is used to extract the driver's head pose time series. The head pose includes the head pitch angle Y, the head yaw angle P, and the head roll angle R.

[0022] S2.5: In the microprocessor unit, a gaze region estimation model is used to extract a gaze region label G; the value of G is 0 and 1, where 0 represents no gaze on the full touch screen and 1 represents gaze on the full touch screen;

[0023] S2.6: In the microprocessor, use the MediaPipe-Hands hand keypoint extraction model to extract the time series of the driver's hand keypoints.

[0024] S2.7: In the microprocessor unit, extract hand motion parameters from the key points of the driver's hand, including hand movement speed Hms, hand lateral position Hx, hand longitudinal position Hy, and finger bending angle Fb.

[0025] The step S3 is specifically as follows:

[0026] S3.1: Construct the interaction sample parameter matrix. Each interaction sample is represented by the parameter matrix X = {G, Y, P, R, Hms, Hx, Hy, Fb, V, δ, Ll}; where V, δ, L l They represent the vehicle speed, steering wheel angle, and vehicle lateral position extracted from the vehicle data acquisition unit;

[0027] S3.2: Use the Savitzky-Golay algorithm to smooth the parameter matrix X = {G, Y, P, R, Hms, Hx, Hy, Fb, V, δ, Ll};

[0028] S3.3: Use the maximum and minimum method to normalize the parameter matrix X = {G, Y, P, R, Hms, Hx, Hy, Fb, V, δ, Ll};

[0029] S3.4: Generate feature matrix X for input model using sliding window method i and predicted region labels G i ;

[0030] In a certain time interval Δt, the cumulative duration of the interactive task is represented by T, the start of the task is represented by t0, and the end of the task is represented by te, T = te- t0;

[0031] The frequency of the time series data is 30 Hz, which means that there are 30 sets of different feature data in a one-second time interval. If the sliding window step size S = 1s and the prediction time interval Δt = 0.5s, it means that the input feature data length of the model is 30, and it can predict the driver's gaze area G 0.5s later. The values ​​of S and Δt satisfy S ≥ 2Δt. i It is a matrix with a length and width of 30 and 11 respectively, 30 is the time length, 11 is the parameter dimension, G i Is a number with a value of 0 or 1, X i and Gi The time interval is Δt.

[0032] The step S4 is specifically as follows:

[0033] S4.1: Extract the underlying features of the input multidimensional time series data based on the CNN convolutional neural network model;

[0034] The formula is:

[0035]

[0036]

[0037] in represents the convolutional neural network module, Represents the network layer in the convolution module, l=1, 2, 3 represents the number of convolution layers, b represents the network layer bias, W represents the parameter weight, X l Represents the output value of each layer, d represents the convolution kernel parameter, t represents the convolution kernel step size, is the input of the convolutional neural network model, is the output of the convolutional neural network model, the first layer of convolution Input ;

[0038] S4.2: Capturing long-term causal relationships based on the BiLSTM bidirectional long-short-term network model;

[0039] The calculation process of BiLSTM is as follows:

[0040]

[0041]

[0042]

[0043]

[0044]

[0045]

[0046]

[0047]

[0048] Where W and b correspond to the parameter weight and bias respectively, i, f, O represent the input gate, forget gate and output gate respectively, C and H represent the candidate memory unit and forget memory unit respectively;

[0049] S4.3: Combined with the ability of the Attention mechanism to allocate weights, the model can adaptively allocate training weights. The calculation formula is as follows:

[0050]

[0051]

[0052]

[0053]

[0054]

[0055] Among them, X is the input, α is the feature importance weight, w and q are model parameters, and y is the final output result;

[0056] S4.4: Use different BiLSTM modules to extract temporal information from the output of each convolutional layer in the CNN model;

[0057] S4.5: Add separate attention mechanism modules to the forward and backward hidden layers of each parallel BiLSTM module to focus the relevant information within each BiLSTM module and reduce potential interference between key information.

[0058] Adding attention forward and backward The process is as follows:

[0059]

[0060]

[0061]

[0062] in and are the forward and backward hidden layer outputs respectively, n represents the number of hidden layers, U, w, and b represent the parameter weights respectively;

[0063] After calculations from S4.1 to S4.5, the final architecture HDLM of hybrid deep learning can be built. HDLM contains three layers of CNN 、 and , each layer of CNN is connected to an independent BiLSTM module 、 、 , and then each BiLSTM module is connected to the bidirectional attention module , , , and finally through a shallow neural network Output the gaze area prediction result G. The above calculation process is expressed as follows:

[0064]

[0065]

[0066]

[0067]

[0068] Where: y 1 is the output result of the first layer attention module; y 2 is the output result of the second layer attention module; y 3 is the output result of the third-layer attention module; G is the prediction result of the gaze area.

[0069] The step S5 is specifically as follows:

[0070] S5.1: With {X i , G i Perform cyclic training and testing on the HDLM model as a sample, setting the number of iterations to 200, the batch size to 64, and the learning rate to 0.001. Save the results of each round of training and testing after training begins. Repeat step S5.1 until the model test accuracy reaches above 98%, reaching the expected target, and save the model parameter weights at this test accuracy.

[0071] S5.2: Build a hardware-in-the-loop test platform for model testing, including a driving simulator, a virtual simulation environment, a computing server, a six-degree-of-freedom vehicle platform, a vehicle data acquisition unit, a dual-source camera, a driver status monitoring system, a data acquisition unit, a microprocessor, and a logic judgment unit. Recruit Nx drivers to conduct performance verification.

[0072] During the test, the driver sits on a six-degree-of-freedom vehicle platform and drives in a simulated environment, operating the central control screen while driving. The dual-source camera captures real-time images of the driver's face and hands, and the vehicle data collector collects vehicle operation and running data. The data mobile unit transmits the collected data to the microprocessor unit, which calculates the real-time gaze area prediction results.

[0073] S5.3: Conduct Likert scale satisfaction surveys on Nx drivers. When the satisfaction rate reaches above 95%, the proposed method is deemed to meet the design standards and the algorithm is integrated into the microprocessor unit.

[0074] The step S6 is specifically as follows:

[0075] S6.1: Set the visual distraction duration threshold TD0, rear-end collision time threshold TTC0, and vehicle deviation time distance TLC0 based on driver distraction theory and safe driving theory;

[0076] S6.2: Obtain the final HDLM model parameters from step S5.1;

[0077] S6.3: The system starts working, and the data acquisition unit collects data in real time; the DMS system determines the driver's current T X Whether to start controlling the full touch screen at the moment, if yes, proceed to the next step;

[0078] S6.4: Data acquisition unit collects T X to T X +S time period data {G, Y, P, R, Hms, Hx, Hy, Fb, V, δ, Ll}, real-time construction of the model input parameter matrix {X i , G i};

[0079] S6.5: Transform the parameter matrix {X i , G i} is sent to the HDLM algorithm module in the microprocessor to X +S+Δt time to predict the gaze area G;

[0080] S6.6: In the microprocessor, calculate T X +S: The total time TD that the driver continuously stares at the entire touch area;

[0081] Record T X to T X At time +S, the model HDLM predicts the number of times G=1. If the number is NG, the fixation time is NG / 30;

[0082] S6.7: In the microprocessor, calculate T X + TTC of the vehicle and the preceding vehicle at time S;

[0083] TTC = (vf - vs) / d, where vf and vs represent the speeds of the preceding vehicle and the ego vehicle, respectively, and d represents the distance between the preceding vehicle and the ego vehicle.

[0084] S6.8: In the microprocessor, calculate T X +S time interval TLC of the vehicle;

[0085] TLC=y / LV, y represents the lateral distance between the center of the vehicle and the lane line, and LV represents the lateral speed of the vehicle;

[0086] S6.9: When T X +S time TD>2s, and T X+The TTC at time S < TTC0, and T X +The TLC at time S < TLC0, and T X +When G = 1 at time S+Δt, the microprocessing unit controls the decision to give a safety reminder;

[0087] S6.10: The microprocessor controls the vehicle head unit to pop up a prompt message at the central upper position of the touch screen interface; at the same time, the microprocessor controls the vehicle head unit to emit a warning sound from the audio unit;

[0088] S6.11: After the DMS system determines that the driver is driving normally, the system ends its operation.

[0089] A system for giving safety reminders for the driver's gaze behavior in a vehicle intelligent cockpit, including:

[0090] Dual-source camera monitoring system: The dual-source camera monitoring system consists of two RGB cameras installed at different positions. One camera captures the driver's face image, and the other camera is used to capture the driver's hand image; during operation, the dual-source cameras are respectively connected to the data acquisition unit;

[0091] DMS system: It can obtain the driver's behavior state in real time, judge whether the driver starts to execute the touch screen interaction task through the DMS system, is connected to the data acquisition unit through a signal harness, and transmits data to the data acquisition unit;

[0092] Vehicle data collector: The vehicle data collector is a hardware unit for collecting the vehicle's motion state and the driver's operation state; during operation, it collects data in real time and transmits the data to the data acquisition unit through a signal harness;[[ID=X]]

[0093] Data acquisition unit: The data acquisition unit is connected to the vehicle data acquisition unit and the dual-source camera monitoring system; it collects and stores the data matrix for a period of time, waiting for the call of the microprocessing unit;

[0094] Microprocessing unit: One end of the microprocessing unit is connected to the data acquisition unit through an input data interface, and the other end is connected to the safety reminder device through an output data interface;

[0095] The microprocessing unit includes an algorithm module and a logic judgment module. The algorithm module integrates the HDLM model to realize the prediction of the driver's gaze area. It also includes a logic judgment module, whose function is to output a safety reminder signal according to the driver's gaze area, the driver's continuous gaze time, and the driving risk index;

[0096] Safety reminder device: includes a visual reminder module and a voice reminder module, both of which are integrated with the vehicle-mounted assembly; for the visual reminder module, when a safety reminder is required, the reminder information will appear in a small window on the full touch screen, and the content will be displayed in the form of a pop-up information interface; for the voice reminder module, when a safety reminder is required, it will issue a reminder sound in the voice module of the full touch screen.

[0097] Beneficial Effects: The present invention develops a hybrid deep learning model, taking the driver's vehicle control state, vehicle motion state, driver's head posture, driver's hand motion state, and driver's gaze area as model input parameters, and the driver's gaze area as model output. The model is trained, tested, and integrated to predict the driver's gaze behavior. Then, based on the predicted results of the driver's gaze area, distraction theory, and safe driving theory, the timing of sending safety prompts is determined, and the driver is actively interacted with through visual and voice information to prompt the driver to regulate his or her gaze behavior. Compared with traditional methods, this method is targeted at smart cockpit full-touch screen interaction scenarios, can predict the driver's gaze area, actively interact with the driver to improve driving safety, and effectively avoid human-machine conflicts. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0099] Figure 1 This is a flow chart of the driver's gaze behavior safety reminder method of the present invention.

[0100] Figure 2 This is a diagram of the input and output feature construction method of the present invention.

[0101] Figure 3 It is a hybrid deep learning algorithm unit.

[0102] Figure 4 FIG. 2 is a diagram of a gaze area prediction device according to the present invention. DETAILED DESCRIPTION

[0103] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0104] In the description of the present invention, it should be understood that the terms "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as limiting the present invention.

[0105] In the present invention, unless otherwise expressly specified or limited, a first feature being "above" or "below" a second feature may include the first and second features being in direct contact, or may include the first and second features being in contact not directly but through another feature between them. Furthermore, a first feature being "above," "above," and "above" a second feature may include the first feature being directly above or obliquely above the second feature, or may simply mean that the first feature is higher in level than the second feature. A first feature being "below," "below," and "below" a second feature may include the first feature being directly below or obliquely below the second feature, or may simply mean that the first feature is lower in level than the second feature.

[0106] like Figures 1 to 4 As shown, a driver gaze behavior safety reminder method applicable to a smart cockpit includes the following steps:

[0107] S1: Constructing a dataset of human-computer interaction for smart cockpit touch screens;

[0108] S1.1: Select N0 models with a full-touch screen smart cockpit;

[0109] S1.2: Design a touch screen interaction experiment and select N1 interactive tasks based on the vehicle's interactive functions;

[0110] S1.3: Select N2 drivers of different genders, driving experience, and ages; each driver performs each similar interactive task N3 times; design N4 actual driving routes, each of which has N5 different speeds; the total number of interactive task samples obtained is N0×N1×N2×N3×N4×N5=N T ;

[0111] S1.4: The dual-source camera monitoring system is used to collect the driver's facial image and hand image respectively; the vehicle data collector is used to collect the driver's operation data and vehicle motion state parameters and save them as text data.

[0112] S2: time series data feature extraction;

[0113] S2.1: Embed the visual feature extraction model TokenHPE token-based head pose estimator, iTracker gaze region recognition model and MediaPipe-Hands hand key point extraction model into the microprocessor;

[0114] S2.2: Extract N from the original data of step S1 T Interactive task process fragments;

[0115] S2.3: Use the data acquisition unit to collect and store data, and send it to the microprocessor unit. In the microprocessor unit, head posture, gaze area, and hand key points are analyzed for each interaction sample;

[0116] S2.4: In the microprocessor, a token-based head pose estimator (TokenHPE) is used to extract the driver's head pose time series. The head pose includes the head pitch angle Y, the head yaw angle P, and the head roll angle R.

[0117] S2.5: In the microprocessor unit, a gaze region estimation model is used to extract a gaze region label G; the value of G is 0 and 1, where 0 represents no gaze on the full touch screen and 1 represents gaze on the full touch screen;

[0118] S2.6: In the microprocessor unit, use the MediaPipe-Hands hand keypoint extraction model to extract the time series of the driver's hand keypoints;

[0119] S2.7: In the microprocessor unit, extract hand motion parameters from the key points of the driver's hand, including hand movement speed Hms, hand lateral position Hx, hand longitudinal position Hy, and finger bending angle Fb.

[0120] S3: Data preprocessing and constructing time series matrix;

[0121] S3.1: Construct the interaction sample parameter matrix. Each interaction sample is represented by the parameter matrix X = {G, Y, P, R, Hms, Hx, Hy, Fb, V, δ, Ll}; where V, δ, L l They represent the vehicle speed, steering wheel angle, and vehicle lateral position extracted from the vehicle data acquisition unit;

[0122] S3.2: Use the Savitzky-Golay algorithm to smooth the parameter matrix X = {G, Y, P, R, Hms, Hx, Hy, Fb, V, δ, Ll};

[0123] S3.3: Use the maximum and minimum method to normalize the parameter matrix X = {G, Y, P, R, Hms, Hx, Hy, Fb, V, δ, Ll};

[0124] S3.4: Generate feature matrix X for input model using sliding window method i and predicted region labels G i ;

[0125] In a certain time interval Δt, the cumulative duration of the interactive task is represented by T, the start of the task is represented by t0, and the end of the task is represented by te, T = te- t0;

[0126] The frequency of the time series data is 30Hz, which means there are 30 sets of different feature data in a one-second time interval. If the sliding window step size S=1s and the prediction time interval Δt=0.5s, it means that the input feature data length of the model is 30, and it can predict the driver's gaze area G 0.5s later. The values ​​of S and Δt satisfy S≥2Δt; X i It is a matrix with a length and width of 30 and 11 respectively, 30 is the time length, 11 is the parameter dimension, G i Is a number with a value of 0 or 1, X i and G i The time interval is Δt.

[0127] S4: Construct a hybrid deep model CNN-BiLSTM-Attention based on BiLSTM, CNN and attention mechanism;

[0128] S4.1: Extract the underlying features of the input multidimensional time series data based on the CNN convolutional neural network model;

[0129] The formula is:

[0130]

[0131]

[0132] in represents the convolutional neural network module, Represents the network layer in the convolution module, l=1, 2, 3 represents the number of convolution layers, b represents the network layer bias, W represents the parameter weight, X l Represents the output value of each layer, d represents the convolution kernel parameter, t represents the convolution kernel step size, is the input of the convolutional neural network model, is the output of the convolutional neural network model, the first layer of convolution Input .

[0133] S4.2: Capturing long-term causal relationships based on the BiLSTM bidirectional long-short-term network model;

[0134] The calculation process of BiLSTM is as follows:

[0135]

[0136]

[0137]

[0138]

[0139]

[0140]

[0141]

[0142]

[0143] Where W and b correspond to the parameter weight and bias respectively, i, f, O represent the input gate, forget gate and output gate respectively, C and H represent the candidate memory unit and forget memory unit respectively.

[0144] S4.3: Combined with the ability of the Attention mechanism to allocate weights, the model can adaptively allocate training weights. The calculation formula is as follows:

[0145]

[0146]

[0147]

[0148]

[0149]

[0150] Among them, X is the input, α is the feature importance weight, w and q are model parameters, and y is the final output result.

[0151] S4.4: Use different BiLSTM modules to extract temporal information from the output of each convolutional layer in the CNN model;

[0152] S4.5: Add separate attention mechanism modules to the forward and backward hidden layers of each parallel BiLSTM module to focus the relevant information within each BiLSTM module and reduce potential interference between key information.

[0153] Adding attention forward and backward The process is as follows:

[0154]

[0155]

[0156]

[0157] in and are the forward and backward hidden layer outputs respectively, n represents the number of hidden layers, and U, w, and b represent the parameter weights respectively.

[0158] After calculations from S4.1 to S4.5, the final architecture HDLM of hybrid deep learning can be built. HDLM contains three layers of CNN 、 and , each layer of CNN is connected to an independent BiLSTM module 、 、 , and then each BiLSTM module is connected to the bidirectional attention module , , , and finally through a shallow neural network Output the gaze area prediction result G. The above calculation process is expressed as follows:

[0159]

[0160]

[0161]

[0162]

[0163] Where: y 1 is the output result of the first layer attention module; y 2 is the output result of the second layer attention module; y 3 is the output result of the third-layer attention module; G is the prediction result of the gaze area.

[0164] S5: CNN-BiLSTM-Attention training, testing, tuning and integration;

[0165] S5.1: With {X i , G i Perform cyclic training and testing on the HDLM model as a sample, setting the number of iterations to 200, the batch size to 64, and the learning rate to 0.001. Save the results of each round of training and testing after training begins. Repeat step S5.1 until the model test accuracy reaches above 98%, reaching the expected target, and save the model parameter weights at this test accuracy.

[0166] S5.2: Build a hardware-in-the-loop test platform for model testing, including a driving simulator, a virtual simulation environment, a computing server, a six-degree-of-freedom vehicle platform, a vehicle data acquisition unit, a dual-source camera, a driver status monitoring system, a data acquisition unit, a microprocessor, and a logic judgment unit. Recruit Nx drivers to conduct performance verification.

[0167] During the test, the driver sits on a six-degree-of-freedom vehicle platform and drives in a simulated environment, operating the central control screen while driving. The dual-source camera captures real-time images of the driver's face and hands, and the vehicle data collector collects vehicle operation and running data. The data mobile unit transmits the collected data to the microprocessor unit, which calculates the real-time gaze area prediction results.

[0168] S5.3: Conduct a Likert scale satisfaction questionnaire survey on Nx drivers. When the satisfaction rate reaches above 95%, the proposed method is deemed to meet the design standards and the algorithm is integrated into the microprocessor unit.

[0169] S6: executing a gaze behavior safety reminder method based on the prediction result of the gaze area;

[0170] S6.1: Set the visual distraction duration threshold TD0, rear-end collision time threshold TTC0, and vehicle deviation time distance TLC0 based on driver distraction theory and safe driving theory;

[0171] S6.2: Obtain the final HDLM model parameters from step S5.1;

[0172] S6.3: The system starts working, and the data acquisition unit collects data in real time; the DMS system determines the driver's current T X Whether to start controlling the full touch screen at the moment, if yes, proceed to the next step;

[0173] S6.4: Data acquisition unit collects T X to T X +S time period data {G, Y, P, R, Hms, Hx, Hy, Fb, V, δ, Ll}, real-time construction of the model input parameter matrix {X i , G i};

[0174] S6.5: Transform the parameter matrix {X i , G i} is sent to the HDLM algorithm module in the microprocessor to X +S+Δt time to predict the gaze area G;

[0175] S6.6: In the microprocessor, calculate T X+Total time TD that the driver continuously gazes at the full touch area at time S;

[0176] Record T X from T X +At time S, the number of times the model HDLM predicts G = 1. If the number is NG, the gaze time is NG / 30;

[0177] S6.7: In the microprocessor, calculate T X +Time to collision (TTC) between the host vehicle and the leading vehicle at time S;

[0178] TTC = (vf - vs) / d, where vf and vs represent the speeds of the leading vehicle and the host vehicle respectively, and d represents the distance between the leading vehicle and the host vehicle;

[0179] S6.8: In the microprocessor, calculate T X +Time lag to collision (TLC) of the host vehicle at time S;

[0180] TLC = y / LV, where y represents the lateral distance of the vehicle center from the lane line, and LV represents the lateral speed of the vehicle.

[0181] S6.9: When T X +TD at time S > 2s, and T X +TTC at time S < TTC0, and T X +TLC at time S < TLC0, and T X +G = 1 at time S + Δt, the microprocessing unit controls the decision to give a safety reminder;

[0182] S6.10: The microprocessor controls the in-vehicle computer assembly to pop up a prompt message "Please pay attention to the road surface in time!" at the central upper position of the touch screen interface; at the same time, the microprocessor controls the in-vehicle computer assembly to emit a "beep beep beep" sound from the audio unit.

[0183] S6.11: After the DMS system determines that the driver is driving normally, the system ends its operation.

[0184] A system for giving a safety reminder of the driver's gaze behavior in a vehicle intelligent cockpit, including:

[0185] Dual-source camera monitoring system: The dual-source camera monitoring system is composed of two RGB cameras installed at different positions. One camera captures the facial image of the driver, and the other camera is used to capture the hand image of the driver; during operation, the dual-source cameras are respectively connected to the data acquisition unit;

[0186] DMS system: It can obtain the driver's behavior status in real time, determine whether the driver starts to perform a touch screen interaction task through the DMS system, and transmit the data to the data acquisition unit;

[0187] Vehicle data collector: A vehicle data collector is a hardware unit used to collect vehicle motion status and driver operation status. During operation, it collects data in real time and transmits the data to the data collection unit through a signal harness.

[0188] Data acquisition unit: The data acquisition unit is connected to the vehicle data acquisition unit and the dual-source camera monitoring system; it collects and stores the data matrix over a period of time, waiting for the microprocessor unit to call;

[0189] Microprocessing unit: One end of the microprocessing unit is connected to the data acquisition unit through the input data interface, and the other end is connected to the safety prompt device through the output data interface;

[0190] The microprocessor unit includes an algorithm module and a logic judgment module. The algorithm module integrates the core algorithm HDLM of the present invention to achieve the prediction of the driver's gaze area;

[0191] The logic judgment module is used to output safety warning signals based on the driver's gaze area, the driver's continuous gaze time, and the driving risk index;

[0192] Safety reminder device: includes a visual reminder module and a voice reminder module, both of which are integrated with the vehicle-mounted assembly; for the visual reminder module, when a safety reminder is required, the reminder information will appear in a small window on the full touch screen, and the content will be displayed in the form of a pop-up information interface; for the voice reminder module, when a safety reminder is required, it will issue a reminder sound in the voice module of the full touch screen.

[0193] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0194] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A driver gaze behavior safety reminder method applicable to a smart cockpit, characterized in that: The following steps are involved: S1: Constructing a dataset of human-computer interaction for smart cockpit touch screens; S2: time series data feature extraction; S3: Data preprocessing and constructing time series matrix; S4: Construct a hybrid deep model CNN-BiLSTM-Attention based on BiLSTM, CNN and attention mechanism; S5: CNN-BiLSTM-Attention training, testing, tuning and integration; S6: executing a gaze behavior safety reminder method based on the prediction result of the gaze area; The step S1 is specifically as follows: S1.1: Select N0 models with a full-touch screen smart cockpit; S1.2: Design a touch screen interaction experiment and select N1 interactive tasks based on the vehicle's interactive functions; S1.3: Select N2 drivers of different genders, driving experience, and ages; each driver performs each similar interactive task N3 times; design N4 actual driving routes, each of which has N5 different speeds; the total number of interactive task samples obtained is N0×N1×N2×N3×N4×N5=N T ; S1.4: The dual-source camera monitoring system collects facial and hand images of the driver. The vehicle data collector collects driver operation data and vehicle motion state parameters and saves them as text data. S5.1: With {X i , G i } Perform cyclic training and testing on the HDLM model as a sample, set the number of iterations to 200, the batch size to 64, and the learning rate to 0.001, and save the results of each round of training and testing after the training starts; Repeat step S5.1 until the model test accuracy reaches above 98%, reaching the expected target, and save the model parameter weights at this test accuracy; The step S6 is specifically as follows: S6.1: Set the visual distraction duration threshold TD0, rear-end collision time threshold TTC0, and vehicle deviation time distance TLC0 based on driver distraction theory and safe driving theory; S6.2: Obtain the final HDLM model parameters from step S5.1; S6.3: The system starts working, and the data acquisition unit collects data in real time; the DMS system determines the driver's current T X Whether to start controlling the full touch screen at the moment, if yes, proceed to the next step; S6.4: Data acquisition unit collects T X to T X +S time period data, real-time construction of model input parameter matrix {X i , G i }; S6.5: Transform the parameter matrix {X i , G i } is sent to the HDLM algorithm module in the microprocessor to X +S+Δt time to predict the gaze area G; S6.6: In the microprocessor, calculate T X +S: The total time TD that the driver continuously stares at the entire touch area; Record T X to T X At time +S, the model HDLM predicts the number of times G=1. If the number is NG, the fixation time is NG / 30; S6.7: In the microprocessor, calculate T X + TTC of the vehicle and the preceding vehicle at time S; TTC = (vf - vs) / d, where vf and vs represent the speeds of the preceding vehicle and the ego vehicle, respectively, and d represents the distance between the preceding vehicle and the ego vehicle. S6.8: In the microprocessor, calculate T X +S time interval TLC of the vehicle; TLC=y / LV, y represents the lateral distance between the center of the vehicle and the lane line, and LV represents the lateral speed of the vehicle; S6.9: When T X + TD at time S > 2 s, and T X + TTC at time S < TTC0, and T X + TLC at time S < TLC0, and T X + When G = 1 at time S + Δt, the microprocessing unit controls the decision to give a safety reminder; S6.10: The microprocessor controls the vehicle-mounted computer assembly to display a pop-up prompt message in the upper center of the touchscreen interface. Simultaneously, the microprocessor controls the vehicle-mounted computer assembly to emit a warning sound from the audio unit. S6.11: After the DMS system determines that the driver is driving normally, the system stops working.

2. The driver gaze behavior safety reminder method applicable to the smart cockpit according to claim 1 is characterized in that: The step S2 is specifically as follows: S2.1: Embed visual feature extraction models into the microprocessor: the token-based head pose estimator TokenHPE, the gaze region recognition model iTracker, and the hand keypoint extraction model MediaPipe-Hands; S2.2: Extract N from the original data of step S1 T Interactive task process fragments; S2.3: Use the data acquisition unit to collect and store data, and send it to the microprocessor unit. In the microprocessor unit, head posture, gaze area, and hand key points are analyzed for each interaction sample; S2.4: In the microprocessor, a token-based head pose estimator (TokenHPE) is used to extract the driver's head pose time series. The head pose includes the head pitch angle Y, the head yaw angle P, and the head roll angle R. S2.5: In the microprocessor unit, a gaze region estimation model is used to extract a gaze region label G; the value of G is 0 and 1, where 0 represents no gaze on the full touch screen and 1 represents gaze on the full touch screen; S2.6: In the microprocessor unit, use the MediaPipe-Hands hand keypoint extraction model to extract the time series of the driver's hand keypoints; S2.7: In the microprocessor unit, extract hand motion parameters from the key points of the driver's hand, including hand movement speed Hms, hand lateral position Hx, hand longitudinal position Hy, and finger bending angle Fb.

3. The driver gaze behavior safety reminder method for a smart cockpit according to claim 2 is characterized in that: The step S3 is specifically as follows: S3.1: Construct an interaction sample parameter matrix. Each interaction sample is represented by a parameter matrix X = {G, Y, P, R, Hms, Hx, Hy, Fb, V, δ, Ll}, where V, δ, and Ll represent the vehicle speed, steering wheel angle, and vehicle lateral position extracted from the vehicle data acquisition unit, respectively. S3.2: Use the Savitzky-Golay algorithm to smooth the parameter matrix X = {G, Y, P, R, Hms, Hx, Hy, Fb, V, δ, Ll}; S3.3: Use the maximum and minimum method to normalize the parameter matrix X = {G, Y, P, R, Hms, Hx, Hy, Fb, V, δ, Ll}; S3.4: Generate feature matrix X for input model using sliding window method i and predicted region labels G i ; In the time interval Δt, the cumulative duration of the interactive task is represented by T, the start of the task is represented by t0, and the end of the task is represented by te, T = te- t0; The frequency of the time series data is 30 Hz, which means that there are 30 sets of different feature data within a time interval of one second. The sliding window step size S is set, and the prediction time interval Δt is set. The values ​​of S and Δt satisfy S≥2Δt.

4. The driver gaze behavior safety reminder method for a smart cockpit according to claim 3 is characterized in that: The step S4 is specifically as follows: S4.1: Extract the underlying features of the input multidimensional time series data based on the CNN convolutional neural network model; S4.2: Capturing long-term causal relationships based on the BiLSTM bidirectional long-short-term network model; S4.3: Combined with the ability of the Attention mechanism to assign weights, the model can adaptively assign training weights; S4.4: Use different BiLSTM modules to extract temporal information from the output of each convolutional layer in the CNN model; S4.5: Add separate attention mechanism modules to the forward and backward hidden layers of each parallel BiLSTM module to focus the relevant information within each BiLSTM module and reduce potential interference between key information. After calculations from S4.1 to S4.5, the final architecture HDLM of hybrid deep learning can be built. HDLM contains three layers of CNN 、 and , each layer of CNN is connected to an independent BiLSTM module 、 、 , and then each BiLSTM module is connected to the bidirectional attention module , , , and finally through a shallow neural network Output the gaze area prediction result G. The above calculation process is expressed as follows: ; ; ; ; Where: y 1 is the output result of the first layer attention module; y 2 is the output result of the second-layer attention module; y 3 is the output result of the third-layer attention module; G is the prediction result of the gaze area.

5. The driver gaze behavior safety reminder method applicable to the smart cockpit according to claim 4 is characterized in that: Specifically, step S5 further includes: S5.2: Build a hardware-in-the-loop test platform for model testing, including a driving simulator, a virtual simulation environment, a computing server, a six-degree-of-freedom vehicle platform, a vehicle data acquisition unit, a dual-source camera, a driver status monitoring system, a data acquisition unit, a microprocessor, and a logic judgment unit. Recruit Nx drivers to conduct performance verification. During the test, the driver sits on a six-degree-of-freedom vehicle platform and drives in a simulated environment, operating the central control screen while driving. The dual-source camera captures real-time images of the driver's face and hands, and the vehicle data collector collects vehicle operation and running data. The data mobile unit transmits the collected data to the microprocessor unit, which calculates the real-time gaze area prediction results. S5.3: Conduct Likert scale satisfaction surveys on Nx drivers. When the satisfaction rate reaches above 95%, the proposed method is deemed to meet the design standards and the algorithm is integrated into the microprocessor unit.

6. A device for implementing the driver gaze behavior safety reminder method applicable to a smart cockpit according to claim 1, characterized in that: include: Dual-source camera monitoring system: The dual-source camera monitoring system consists of two RGB cameras installed in different positions. One camera captures the driver's facial image, and the other captures the driver's hand image. During operation, the dual-source cameras are connected to the data acquisition unit. DMS system: It can obtain the driver's behavior status in real time, determine whether the driver starts to perform touch screen interaction tasks through the DMS system, connect to the data acquisition unit through the signal harness, and transmit data to the data acquisition unit; Vehicle data collector: A vehicle data collector is a hardware unit used to collect vehicle motion status and driver operation status; During the working process, it collects data in real time and transmits the data to the data acquisition unit through the signal harness; Data acquisition unit: The data acquisition unit is connected to the vehicle data acquisition unit and the dual-source camera monitoring system; it collects and stores the data matrix over a period of time, waiting for the microprocessor unit to call; Microprocessing unit: One end of the microprocessing unit is connected to the data acquisition unit through the input data interface, and the other end is connected to the safety prompt device through the output data interface; The microprocessor unit includes an algorithm module and a logic judgment module. The algorithm module integrates the HDLM model to predict the driver's gaze area. The logic judgment module outputs safety warning signals based on the driver's gaze area, the driver's continuous gaze time, and the driving risk index; Safety reminder device: includes a visual reminder module and a voice reminder module, both of which are integrated with the vehicle-mounted assembly; for the visual reminder module, when a safety reminder is required, the reminder information will appear in a small window on the full touch screen, and the content will be displayed in the form of a pop-up information interface; for the voice reminder module, when a safety reminder is required, it will issue a reminder sound in the voice module of the full touch screen.

Images