Method and device for predicting physical interaction feasibility of full touch screen of intelligent cabin
By building an integrated stacked model EPCAM, combining deep learning and machine learning technology, real-time prediction of driver distraction and behavioral risks during physical interaction of full-touch screens, solving the failure to predict and avoid driver distraction in the existing technology. The problem improves driving safety.
Patent Information
- Application Number
- CN202410990386.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-07-23
AI Technical Summary
The prior art fails to effectively predict and avoid the driver's degree of distraction and behavioral risks during physical interaction of full touch screens, resulting in the threat of driving safety.
Using the distraction degree label generation method based on multi-layer perceptron and dynamic peak clustering algorithm, the integrated stacking model EPCAM is constructed, combined with deep learning and machine learning technology, and the driver's facial and hand data, vehicle motion status is collected through the camera, and the vehicle is constructed. Distraction degree identification model is identified and integrated into microprocessing units to predict driver distraction degree and driving risks in real time, providing reminder information.
It realizes the prediction of the driver's degree of distraction and behavioral risks during physical interaction of the full touch screen, and provides guidance in advance, improving driving safety and reducing safety hazards caused by human-computer interaction.
Smart Images

Figure CN118940891A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a method and a device for predicting the feasibility of physical interaction of a full-touch screen in an intelligent cockpit, and belongs to the technical field of road traffic driving safety. Background Art
[0002] With the introduction of automobile intelligence and the concept of smart cockpit, more and more vehicles are beginning to use large-size full-touch screens for interaction. This interaction mode has become the main source of driver distraction. When the driver interacts physically with the full touch screen, the driver will bear a large visual and manual distraction load, which will seriously endanger driving safety. So far, there are a large number of methods for driver status monitoring, but these methods focus on real-time detection of whether the driver is distracted, and do not involve the problem of predicting the degree of driver distraction. There has also been no invention that focuses on whether the driver should perform human-computer interaction at a certain moment to avoid dangerous driving behavior in advance. If a device can be developed to predict the driver's distraction level and behavioral risks when the driver is about to interact physically with the full touch screen, and give guiding opinions on whether the driver can continue to interact, it will be of great significance to improve driving safety, promote the development of the smart cockpit industry, and improve human-computer relationships. Therefore, the present invention focuses on the above scenario and proposes a method and device suitable for predicting the feasibility of physical interaction of a full touch screen in a smart cockpit and a driver safety reminder, aiming to improve the safety of physical interaction of the driver on the full touch screen. Summary of the invention
[0003] Purpose of the invention: In view of the deficiencies in the prior art, the present invention provides a method for predicting the feasibility of physical interaction of a full-touch screen in a smart cockpit to solve the problems mentioned in the above background technology.
[0004] Technical solution: A method for predicting the feasibility of physical interaction of a full touch screen in an intelligent cockpit, characterized in that it includes the following steps:
[0005] S1: Construct a smart cockpit full touch screen distraction dataset;
[0006] S2: Construct a distraction degree label generation method based on multi-layer perceptron and dynamic peak clustering algorithm;
[0007] S3: Construct the distraction degree recognition model EPCAM based on the integrated stacking model;
[0008] S4: training a distraction prediction model;
[0009] S5: Distraction prediction model for real-car testing;
[0010] S6: Integrate the distraction degree recognition model EPCAM into the microprocessing unit to execute the specific implementation method of physical interaction feasibility prediction.
[0011] The S1 is specifically:
[0012] S1.1: Conduct market research and select N0 models with full touch screens for smart cockpits;
[0013] S1.2: Design a driver interaction test and select N1 interactive tasks based on the vehicle's interactive functions (using the vehicle's full touch screen to make calls, using the vehicle's full touch screen to adjust the air conditioning volume, using the vehicle's full touch screen to navigate, etc.);
[0014] S1.3: Recruit N2 drivers of different genders, driving experience and ages; each driver performs each interactive task N3 times; design N4 actual driving routes, each of which has N5 different vehicle speeds; the total number of interactive task samples finally obtained is N0×N1×N2×N3×N4×N5=N T ;
[0015] S1.4: Camera 1 and camera 2 are installed to collect the driver's facial image and hand image respectively; camera 1 is used to collect the driver's hand operation status and store it in the form of RGB image; camera 2 is used to collect facial vision and emotional state data and store it in the form of RGB image; the vehicle data collector is used to collect vehicle motion state parameters and save them as text data;
[0016] S1.5: After data collection is completed, a deep learning model is used to extract distraction features, driver status, vehicle motion status, and vehicle operating parameters;
[0017] MediaPipe-Hand, a hand feature point extraction model, is used to analyze the hand video of camera 1 to extract manual load parameters in a single interactive task, including the maximum time HLMAX of the hand leaving the steering wheel, the average time HLM of the hand leaving the steering wheel, and the total time HLT of the hand leaving the steering wheel. MediaPipe-Face, a hand feature point extraction model, and iTracker, a driver gaze area recognition model, are used to extract the driver's visual distraction features in a single interactive task, including the maximum time DGMAX of a single touch screen gaze, the average touch screen gaze time DGM, and the total touch screen gaze time DGS. The steering wheel turning rate index VSWTR, the steering wheel angular velocity entropy VSWS, and the vehicle lateral position standard deviation SDLP are extracted from the vehicle data acquisition instrument. In addition, the driver's tension index FL is extracted from the facial image through the DMS system. The parameter matrix reflecting the degree of driving distraction is finally expressed as DLL H ={HLMAX,HLM,HLT,DGMAX,DGM,DGS,VSWTR,VSWS,SDLP,FL}, the dimension of this matrix is 10×NT .
[0018] The S2 is specifically:
[0019] S2.1: Constructing a feature dimension reduction algorithm F based on multi-layer perceptron MLP , the original parameter matrix DLL H Dimensionality reduction to DLL L ;
[0020] First, use the encoder to encode the DLL H Dimensionality reduction to DLL L :
[0021] Encoder(DLL H ):w(DLL H )+b
[0022] Then use the decoder to decode the DLL L Refactoring to DLL H :
[0023] Decoder(DLL L ):w T (DLL L )+b′
[0024] By minimizing the reconstruction error, MLP Optimize the parameters:
[0025] min(F MLP :((w T (wDLL H +b)+b′),DLL H ))
[0026] S2.2: Construct a distraction degree discretization label generation algorithm G based on density peak L , Algorithm G L The concept of local density is introduced; the data point x i The local density is defined as ρ i , then its expression is
[0027]
[0028] Where, d c represents the cutoff distance, d ij Represents data point x i With data point x j The Euclidean distance between them, χ(·) is the logical judgment function, d ij The expressions of and χ(·) are as follows:
[0029] d ij=||x i -x j ||
[0030]
[0031] Data point x i The minimum distance δ to any other high local density point i It is expressed as:
[0032]
[0033] In determining the data point ρ i and δ i Then draw a decision diagram, where ρ i is the horizontal axis of the decision diagram, δ i is the vertical axis of the decision diagram; based on the decision diagram, ρ i and δ i The relatively high points (also called peaks) are marked as cluster centers, and ρ i Lower but δ i The higher points are marked as noise. Based on the determined cluster centers, the remaining data points are assigned to the cluster centers where the data points with the closest distance and greater density are located;
[0034] According to the algorithm G L The load characterization parameter DLL after dimensionality reduction L Clustered into discrete load level labels Y = {DLL1, DLL2, DLL3}, where DLL1 represents a low level of distraction, DLL2 represents a medium level of distraction, and DLL3 represents a high level of distraction.
[0035] The S3 is specifically:
[0036] S3.1: construct a distraction degree recognition model EPCAM based on an integrated stacking model, and select base classifier 1, base classifier 2, base classifier 3, base classifier 4, and base classifier 5 from a machine learning model library;
[0037] S3.2: Select a meta-classifier from the machine learning model library, including LR: logistic regression model; DNN: deep neural network model; KNN: K nearest neighbor model;
[0038] S3.3: Select the base classifier and meta-classifier integration strategy from the knowledge base, integrate the model, use public data to verify the model, and proceed to the next step after judging that the classification performance reaches more than 98%;
[0039] S3.4: Improve the defects of the integrated model and further optimize the model's anti-overfitting performance; integrate the principal component analysis algorithm into step S3.3, use the principal component analysis algorithm to reduce the dimension of the original features, and use the principal components after dimensionality reduction, that is, the principal component features with a contribution rate of more than 80%, as the supplementary features of the meta-classifier.
[0040] The S4 is specifically:
[0041] S4.1: Take distraction level Y = {DLL1, DLL2, DLL3} as the model EPCAM output label;
[0042] S4.2: Taking the start time t1 of the subtask as the reference point, collect time series data of time length T in reverse with time interval t0, where T is in seconds and collect 30 frames of data per second;
[0043] S4.3: Each frame of data in S4.2 includes the following parameters: driver age A, gender G, driver operation intention DT, and emotional parameter FL are extracted from the DMS system; road conditions LE, traffic flow complexity TF, and weather parameters TS are extracted from the autonomous driving environment perception system; vehicle speed VS, driving task type DT, and steering wheel angle SC are extracted from the vehicle data collector; and full touch screen parameter vector PHMI and human-computer interaction geometric relationship parameter vector HMGP are extracted from the vehicle built-in parameter table;
[0044] S4.4: Take the data unit X={A, FL, G, DT, LE, TF, TS, VS, DT, SC, PHMI, HMGP} of time length T as the input of the model EPCAM;
[0045] S4.5: Build the EPCAM model training platform and test software platform based on Python language, and set the model parameter θ optimization process:
[0046] The prior probability distribution of parameter θ is P(θ), the probability distribution of output Y under parameter θ and input X is P(Y|X,θ), parameter θ i The probability distribution under the input X is P(Y|X,θ i ), the posterior probability distribution of the parameter θ under input X and output Y is P(θ|X,Y). According to Bayes' theorem, the posterior probability distribution formula is The Gaussian process model is used to describe the posterior probability distribution, and the kernel function selected is k(x,x′)=φ(x) T φ(x′), φ(x′) is the eigenvector of x, and the kernel matrix is further expressed as By calculating the inverse of the kernel matrix, we can get the posterior probability distribution of the hyperparameters Finally, the optimal parameters are selected using the posterior probability distribution;
[0047] S4.6: Start training the EPCAM model, determine whether the verification result of the EPCAM model reaches the optimal value, and the test accuracy of the EPCAM model on the verification set reaches more than 98%, and then save the parameter weights. The specific training process includes:
[0048] Input training data set D is randomly divided into five subsets D = {D1, D2, D3, D4};
[0049] Train the base classifier using data D k Training c ,hc∈{h RF (x i ),h GBDT (x i ),h XGBoost (x i ),h LightGBM (x i )}.
[0050] Construct a new feature matrix {x i ′,y i}(x i ′={h RF (x i ),h GBDT (x i ),h XGBoost (x i ),h LightGBM (x i ),P},P is the low-dimensional feature obtained by principal component analysis;
[0051] Train a meta-classifier using the new features (x i ′,y i ) Train h′, ensemble classifier E(x) = h′;
[0052] Cross-training and validation are performed based on the data subset D = {D1, D2, D3, D4}. The validation set accuracy reaches more than 98%, and the E(x) parameter weights are saved.
[0053] The S5 is specifically:
[0054] S5.1: Build a real vehicle test hardware platform, including the vehicle-mounted DMS system, autonomous driving environment perception system, data collection unit, microprocessor unit and early warning device;
[0055] S5.2: Optimize model EPCAM based on test feedback;
[0056] S5.3: Recruit drivers for real vehicle testing again, fill in the model prediction result test feedback questionnaire, determine the effectiveness of the method through subjective and objective judgment, and use the Likert subjective questionnaire to collect driver satisfaction. When the overall satisfaction is higher than 95%, it is judged that θ is valid. If the overall satisfaction is lower than 95%, repeat steps 2, 3, and 4, and optimize the main parameters of EPCAM according to the method in S4.5 until the conditions in step 5 are met.
[0057] The S6 is specifically:
[0058] S6.1: Setting the distraction level threshold DLL0 and the driving risk field index level threshold FX0;
[0059] S6.2: Obtain the final EPCAM model parameters from step S5.1;
[0060] S6.3: The system starts working, and the data collection unit collects and stores real-time data;
[0061] S6.4: The DMS system determines whether the driver wants to start operating the full touch screen and records the time T0 when the driver starts operating the full touch screen;
[0062] S6.5: call the data units {A, FL, G, DT, LE, TF, TS, VS, DT, SC, PHMI, HMGP} in the time period T0 before time T0;
[0063] S6.6: Send the data unit to the EPCAM algorithm module in the microprocessor to predict the distraction level DLL generated by the driver continuing to perform physical interaction;
[0064] S6.7: Calculate the driving risk field index FX at time T0 in the microprocessor unit;
[0065] S6.8: In the microprocessor unit, judging whether it is necessary to provide prompt information to the driver according to the distraction degree prediction level DLL and the driving risk field index FX; when DLL>DLL0 and FX>FX0, proceeding to the next step;
[0066] S6.9: To remind the driver, the microprocessor controls the vehicle signal lights and steering wheel vibration device to send out warning signals of corresponding levels. At the same time, the voice device sends out a prompt tone of "it is recommended to use voice for interaction".
[0067] A smart cockpit full touch screen physical interaction feasibility prediction device, including a vehicle-mounted DMS system, an automatic driving environment perception system, a data collection unit, a microprocessor unit and an early warning device; wherein,
[0068] The vehicle-mounted DMS system acquires the driver’s emotions, fatigue and attention in real time, and is connected to the data collection unit via a signal transmission harness;
[0069] The autonomous driving environment perception system is used to output driving environment characteristic parameters, including road conditions, traffic flow conditions and weather conditions, and is connected to the data collection unit through a signal transmission harness;
[0070] The data collection unit is used to collect the relevant data of the driver's operation of the touch screen and store it, waiting for the call of the microprocessor unit;
[0071] One end of the microprocessor unit is connected to the data collection unit through an input data interface, and the other end is connected to the early warning device through an output data interface; the early warning logic judgment module is used to output early warning signals according to the degree of distraction and risk field parameters;
[0072] Early warning device: The early warning device includes a vehicle-mounted signal light, a steering wheel vibration device and a voice device; the vehicle-mounted signal light can flash at different frequencies, the steering wheel vibration device can vibrate the steering wheel at different frequencies, and the voice device can broadcast prompt information.
[0073] Beneficial effects: The present invention constructs a distraction degree recognition model EPCAM based on an integrated stacking model, takes the cockpit touch screen spatial layout geometric parameters, touch screen physical response parameters, driver body parameters, driver attribute parameters, vehicle motion parameters, and road environment parameters as model input parameters, and takes the machine learning clustering results of visual parameters, manual parameters, and physiological indicators as model outputs, and finally realizes distraction degree prediction through model training, testing and integration; then, the feasibility of physical interaction is evaluated according to the driver's distraction degree prediction value and the vehicle operation risk characterization parameter, when the interaction conditions are met, no prompt information is sent, and when the interaction conditions are not met, the microprocessing unit controls the prompt device to send a prompt information; the method is aimed at the full touch screen interaction of the smart cockpit, can predict the driver's distraction degree, and evaluate the risk of the driver continuing the physical interaction with the vehicle in combination with the vehicle operation risk, and remind the driver through active interaction when the conditions are met, so as to avoid dangerous driving behavior in advance. BRIEF DESCRIPTION OF THE DRAWINGS
[0074] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0075] Figure 1 The figure is a flow chart of the prediction method of the present invention.
[0076] Figure 2 This is a flow chart of algorithm training and testing of the present invention.
[0077] Figure 3 This is a logical framework diagram of the algorithm of the present invention.
[0078] Figure 4 Schematic diagram of the prediction device of the present invention.
[0079] RF: Random Forest; GBDT: Gradient Boosted Decision Tree; XGBoost: Extreme Gradient Boosted Tree; LightGBM: Lightweight Gradient Boosting Machine Algorithm; PAC: Principal Component Analysis; PC1: Principal Component 1, PC2: Principal Component 2; PC3: Principal Component 3; learn: learning; predict: prediction; predictions: predicted value; New feature matrix: new feature matrix; LR: logistic regression model; DNN: deep neural network model; KNN: K nearest neighbor model. DETAILED DESCRIPTION
[0080] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0081] In the description of the present invention, it is necessary to understand that the terms "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.
[0082] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may include that the first and second features are in direct contact, or may include that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, a first feature being "above", "above" and "above" a second feature includes that the first feature is directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. A first feature being "below", "below" and "below" a second feature includes that the first feature is directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature.
[0083] like Figures 1 to 4As shown, a method for predicting the feasibility of physical interaction of a full touch screen in an intelligent cockpit is characterized by comprising the following steps:
[0084] S1: Construct a smart cockpit full touch screen distraction dataset;
[0085] S1.1: Conduct market research and select N0 models with full touch screens for smart cockpits;
[0086] S1.2: Design a driver interaction test and select N1 interactive tasks based on the vehicle's interactive functions (using the vehicle's full touch screen to make calls, using the vehicle's full touch screen to adjust the air conditioning volume, using the vehicle's full touch screen to navigate, etc.);
[0087] S1.3: Recruit N2 drivers of different genders, driving experience and ages; each driver performs each interactive task N3 times; design N4 actual driving routes, each of which has N5 different vehicle speeds; the total number of interactive task samples finally obtained is N0×N1×N2×N3×N4×N5=N T ;
[0088] S1.4: Camera 1 and camera 2 are installed to collect the driver's facial image and hand image respectively; camera 1 is used to collect the driver's hand operation status and store it in the form of RGB image; camera 2 is used to collect facial vision and emotional state data and store it in the form of RGB image; the vehicle data collector is used to collect vehicle motion state parameters and save them as text data;
[0089] S1.5: After data collection is completed, a deep learning model is used to extract distraction features, driver status, vehicle motion status, and vehicle operating parameters;
[0090] MediaPipe-Hand, a hand feature point extraction model, is used to analyze the hand video of camera 1 to extract manual load parameters in a single interactive task, including the maximum time HLMAX of the hand leaving the steering wheel, the average time HLM of the hand leaving the steering wheel, and the total time HLT of the hand leaving the steering wheel. MediaPipe-Face, a hand feature point extraction model, and iTracker, a driver gaze area recognition model, are used to extract the driver's visual distraction features in a single interactive task, including the maximum time DGMAX of a single touch screen gaze, the average touch screen gaze time DGM, and the total touch screen gaze time DGS. The steering wheel turning rate index VSWTR, the steering wheel angular velocity entropy VSWS, and the vehicle lateral position standard deviation SDLP are extracted from the vehicle data acquisition instrument. In addition, the driver's tension index FL is extracted from the facial image through the DMS system. The parameter matrix reflecting the degree of driving distraction is finally expressed as DLL H={HLMAX,HLM,HLT,DGMAX,DGM,DGS,VSWTR,VSWS,SDLP,FL}, the dimension of this matrix is 10×N T .
[0091] S2: Construct a distraction degree label generation method based on multi-layer perceptron and dynamic peak clustering algorithm;
[0092] S2.1: Constructing a feature dimension reduction algorithm F based on multi-layer perceptron MLP , the original parameter matrix DLL H Dimensionality reduction to DLL L ;
[0093] First, use the encoder to encode the DLL H Dimensionality reduction to DLL L :
[0094] Encoder(DLL H ):w(DLL H )+b
[0095] Then use the decoder to decode the DLL L Refactoring to DLL H :
[0096] Decoder(DLL L ):w T (DLL L )+b′
[0097] By minimizing the reconstruction error, MLP Optimize the parameters:
[0098] min(F MLP :((w T (wDLL H +b)+b′),DLL H ))
[0099] S2.2: Construct a distraction degree discretization label generation algorithm G based on density peak L , Algorithm G L The concept of local density is introduced; the data point x i The local density is defined as ρ i , then its expression is
[0100]
[0101] Where, d c represents the cutoff distance, d ij Represents data point x i With data point x jThe Euclidean distance between them, χ(·) is the logical judgment function, d ij The expressions of and χ(·) are as follows:
[0102] d ij =||x i -x j ||
[0103]
[0104] Data point x i The minimum distance δ to any other high local density point i It is expressed as:
[0105]
[0106] In determining the data point ρ i and δ i Then draw a decision diagram, where ρ i is the horizontal axis of the decision diagram, δ i is the vertical axis of the decision diagram; based on the decision diagram, ρ i and δ i The relatively high points (also called peaks) are marked as cluster centers, and ρ i Lower but δ i The higher points are marked as noise. Based on the determined cluster centers, the remaining data points are assigned to the cluster centers where the data points with the closest distance and greater density are located;
[0107] According to the algorithm G L The load characterization parameter DLL after dimensionality reduction L Clustered into discrete load level labels Y = {DLL1, DLL2, DLL3}, where DLL1 represents a low level of distraction, DLL2 represents a medium level of distraction, and DLL3 represents a high level of distraction.
[0108] S3: Construct the distraction degree recognition model EPCAM based on the integrated stacking model;
[0109] S3.1: construct a distraction degree recognition model EPCAM based on an integrated stacking model, and select base classifier 1, base classifier 2, base classifier 3, base classifier 4, and base classifier 5 from a machine learning model library;
[0110] S3.2: Select a meta-classifier from the machine learning model library;
[0111] S3.3: Select the base classifier and meta-classifier integration strategy from the knowledge base, integrate the model, use public data to verify the model, and proceed to the next step after judging that the classification performance reaches more than 98%;
[0112] S3.4: Improve the defects of the integrated model and further optimize the model's anti-overfitting performance; integrate the principal component analysis algorithm into step S3.3, use the principal component analysis algorithm to reduce the dimension of the original features, and use the principal components after dimensionality reduction, that is, the principal component features with a contribution rate of more than 80%, as the supplementary features of the meta-classifier.
[0113] S4: training a distraction prediction model;
[0114] S4.1: Take distraction level Y = {DLL1, DLL2, DLL3} as the model EPCAM output label;
[0115] S4.2: Taking the start time t1 of the subtask as the reference point, collect time series data of time length T in reverse with time interval t0, where T is in seconds and collect 30 frames of data per second;
[0116] S4.3: Each frame of data in S4.2 includes the following parameters: driver age A, gender G, driver operation intention DT, and emotional parameter FL are extracted from the DMS system; road conditions LE, traffic flow complexity TF, and weather parameters TS are extracted from the autonomous driving environment perception system; vehicle speed VS, driving task type DT, and steering wheel angle SC are extracted from the vehicle data collector; and full touch screen parameter vector PHMI and human-computer interaction geometric relationship parameter vector HMGP are extracted from the vehicle built-in parameter table;
[0117] S4.4: Take the data unit X={A, FL, G, DT, LE, TF, TS, VS, DT, SC, PHMI, HMGP} of time length T as the input of the model EPCAM;
[0118] S4.5: Build the EPCAM model training platform and test software platform based on Python language, and set the model parameter θ optimization process:
[0119] The prior probability distribution of parameter θ is P(θ), the probability distribution of output Y under parameter θ and input X is P(Y|X,θ), parameter θ i The probability distribution under the input X is P(Y|X,θ i ), the posterior probability distribution of the parameter θ under input X and output Y is P(θ|X,Y). According to Bayes' theorem, the posterior probability distribution formula is The Gaussian process model is used to describe the posterior probability distribution, and the kernel function selected is k(x,x′)=φ(x) T φ(x′), φ(x′) is the eigenvector of x, and the kernel matrix is further expressed as By calculating the inverse of the kernel matrix, we can get the posterior probability distribution of the hyperparameters Finally, the optimal parameters are selected using the posterior probability distribution;
[0120] S4.6: Start training the EPCAM model, determine whether the verification result of the EPCAM model reaches the optimal value, and the test accuracy of the EPCAM model on the verification set reaches more than 98%, and then save the parameter weights. The specific training process includes:
[0121] Input training data set D is randomly divided into five subsets D = {D1, D2, D3, D4};
[0122] Train the base classifier using data D k Training c ,hc∈{h RF (x i ),h GBDT (x i ),h XGBoost (x i ),h LightGBM (x i )}.
[0123] Construct a new feature matrix {x i ′,y i}(x i ′={h RF (x i ),h GBDT (x i ),h XGBoost (x i ),h LightGBM (x i ),P},P is the low-dimensional feature obtained by principal component analysis;
[0124] Train a meta-classifier using the new features (x i ′,y i ) Train h′, ensemble classifier E(x) = h′;
[0125] Cross-training and validation are performed based on the data subset D = {D1, D2, D3, D4}. The validation set accuracy reaches more than 98%, and the E(x) parameter weights are saved.
[0126] S5: Distraction prediction model for real-car testing;
[0127] S5.1: Build a real vehicle test hardware platform, including the vehicle-mounted DMS system, autonomous driving environment perception system, data collection unit, microprocessor unit and early warning device;
[0128] S5.2: Optimize model EPCAM based on test feedback;
[0129] S5.3: Recruit drivers for real vehicle testing again, fill in the model prediction result test feedback questionnaire, determine the effectiveness of the method through subjective and objective judgment, and use the Likert subjective questionnaire to collect driver satisfaction. When the overall satisfaction is higher than 95%, it is judged that θ is valid. If the overall satisfaction is lower than 95%, repeat steps 2, 3, and 4, and optimize the main parameters of EPCAM according to the method in S4.5 until the conditions in step 5 are met.
[0130] S6: Integrate the algorithm into a microprocessing unit to perform a specific implementation method of physical interaction feasibility prediction;
[0131] S6.1: Setting the distraction level threshold DLL0 and the driving risk field index level threshold FX0;
[0132] S6.2: Obtain the final EPCAM model parameters from step S5.1;
[0133] S6.3: The system starts working, and the data collection unit collects and stores real-time data;
[0134] S6.4: The DMS system determines whether the driver wants to start operating the full touch screen and records the time T0 when the driver starts operating the full touch screen;
[0135] S6.5: call the data units {A, FL, G, DT, LE, TF, TS, VS, DT, SC, PHMI, HMGP} in the time period T0 before time T0;
[0136] S6.6: Send the data unit to the EPCAM algorithm module in the microprocessor to predict the distraction level DLL generated by the driver continuing to perform physical interaction;
[0137] S6.7: Calculate the driving risk field index FX at time T0 in the microprocessor unit;
[0138] S6.8: In the microprocessor unit, judging whether it is necessary to provide prompt information to the driver according to the distraction degree prediction level DLL and the driving risk field index FX; when DLL>DLL0 and FX>FX0, proceeding to the next step;
[0139] S6.9: To remind the driver, the microprocessor controls the vehicle signal lights and steering wheel vibration device to send out warning signals of corresponding levels. At the same time, the voice device sends out a prompt tone of "it is recommended to use voice for interaction".
[0140] A smart cockpit full touch screen physical interaction feasibility prediction device, including a vehicle-mounted DMS system, an automatic driving environment perception system, a data collection unit, a microprocessor unit and an early warning device; wherein,
[0141] The vehicle-mounted DMS system acquires the driver’s emotions, fatigue and attention in real time, and is connected to the data collection unit via a signal transmission harness;
[0142] The autonomous driving environment perception system is used to output driving environment characteristic parameters, including road conditions, traffic flow conditions and weather conditions, and is connected to the data collection unit through a signal transmission harness;
[0143] The data collection unit is used to collect the relevant data of the driver's operation of the touch screen and store it, waiting for the call of the microprocessor unit;
[0144] One end of the microprocessor unit is connected to the data collection unit through an input data interface, and the other end is connected to the early warning device through an output data interface; the early warning logic judgment module is used to output early warning signals according to the degree of distraction and risk field parameters;
[0145] Early warning device: The early warning device includes a vehicle-mounted signal light, a steering wheel vibration device and a voice device; the vehicle-mounted signal light can flash at different frequencies, the steering wheel vibration device can vibrate the steering wheel at different frequencies, and the voice device can broadcast prompt information.
[0146] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0147] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for predicting the feasibility of physical interaction of a full touch screen in an intelligent cockpit, characterized in that: The following steps are involved: S1: Construct a smart cockpit full touch screen distraction dataset; S2: Construct a distraction degree label generation method based on multi-layer perceptron and dynamic peak clustering algorithm; S3: Construct the distraction degree recognition model EPCAM based on the integrated stacking model; S4: training a distraction prediction model; S5: Distraction prediction model for real-car testing; S6: Integrate the distraction degree recognition model into the microprocessing unit and perform a specific implementation method for physical interaction feasibility prediction.
2. The method for predicting the feasibility of physical interaction of a full touch screen in a smart cockpit according to claim 1 is characterized in that: The S1 is specifically: S1.1: Conduct market research and select N0 models with full touch screens for smart cockpits; S1.2: Design a driver interaction test and select N1 interactive tasks based on the vehicle's interactive functions (using the vehicle's full touch screen to make calls, using the vehicle's full touch screen to adjust the air conditioning volume, using the vehicle's full touch screen to navigate, etc.); S1.3: Recruit N2 drivers of different genders, driving experience and ages; each driver performs each interactive task N3 times; design N4 actual driving routes, each of which has N5 different vehicle speeds; the total number of interactive task samples finally obtained is N0×N1×N2×N3×N4×N5=N T ; S1.4: Install camera 1 and camera 2 to collect the driver's facial image and hand image respectively; Camera 1 is used to collect the driver's hand operation status and store it in RGB image format; Camera 2 is used to collect facial vision and emotional state data and store it in RGB image format; The vehicle data collector is used to collect vehicle motion state parameters and save them as text data; S1.5: After data collection is completed, a deep learning model is used to extract distraction features, driver status, vehicle motion status, and vehicle operating parameters; MediaPipe-Hand, a hand feature point extraction model, is used to analyze the hand video of camera 1 to extract the manual load parameters in a single interactive task, including the maximum time HLMAX of the hand leaving the steering wheel, the average time HLM of the hand leaving the steering wheel, and the total time HLT of the hand leaving the steering wheel. MediaPipe-Face, a hand feature point extraction model, and iTracker, a driver gaze area recognition model, are used to extract the driver's visual distraction features in a single interactive task, including the maximum time DGMAX of a single touch screen gaze, the average touch screen gaze time DGM, and the total touch screen gaze time DGS. The steering wheel turning rate index VSWTR, steering wheel angular velocity entropy VSWS and vehicle lateral position standard deviation SDLP are extracted from the vehicle data acquisition instrument; in addition, the driver's nervousness index FL is extracted from the facial image through the DMS system; the parameter matrix reflecting the degree of driving distraction is finally expressed as DLL H ={HLMAX,HLM,HLT,DGMAX,DGM,DGS,VSWTR,VSWS,SDLP,FL}, the dimension of this matrix is 10×N T .
3. The method for predicting the feasibility of physical interaction of a full touch screen in a smart cockpit according to claim 2 is characterized in that: The S2 is specifically: S2.1: Constructing a feature dimension reduction algorithm F based on multi-layer perceptron MLP , the original parameter matrix DLL H Dimensionality reduction to DLL L ; First, use the encoder to encode the DLL H Dimensionality reduction to DLL L : Encoder(DLL H ):w(DLL H )+b Then use the decoder to decode the DLL L Refactoring to DLL H : Decoder(DLL L ):w T (DLL L )+b′ By minimizing the reconstruction error, MLP Optimize the parameters: min(F MLP :((w T (wDLL H +b)+b′),DLL H )) S2.2: Construct a distraction degree discretization label generation algorithm G based on density peak L , Algorithm G L The concept of local density is introduced; the data point x i The local density is defined as ρ i , then its expression is Where, d c represents the cutoff distance, d ij Represents data point x i With data point x j The Euclidean distance between them, c(·) is the logical judgment function, d ij The expressions of and c(·) are as follows: d ij =||x i -x j || Data point x i The minimum distance δ to any other high local density point i It is expressed as: In determining the data point ρ i and δ i Then draw a decision diagram, where ρ i is the horizontal axis of the decision diagram, δ i is the vertical axis of the decision diagram; based on the decision diagram, ρ i and δ i The relatively high points, also called peaks, are marked as cluster centers. i Lower but δ i The higher points are marked as noise; based on the determined cluster center, the remaining data points are assigned to the cluster center where the data point with the closest distance and greater density is located; According to the algorithm G L The load characterization parameter DLL after dimensionality reduction L Clustered into discrete load level labels Y = {DLL1, DLL2, DLL3}, where DLL1 represents a low level of distraction, DLL2 represents a medium level of distraction, and DLL3 represents a high level of distraction.
4. The method for predicting the feasibility of physical interaction of a full touch screen in a smart cockpit according to claim 3 is characterized in that: The S3 is specifically: S3.1: construct a distraction degree recognition model EPCAM based on an integrated stacking model, and select base classifier 1, base classifier 2, base classifier 3, base classifier 4, and base classifier 5 from a machine learning model library; S3.2: Select a meta-classifier from the machine learning model library; S3.3: Select the base classifier and meta-classifier integration strategy from the knowledge base, integrate the model, use public data to verify the model, and proceed to the next step after judging that the classification performance reaches more than 98%; S3.4: Improve the defects of the integrated model and further optimize the model's anti-overfitting performance; integrate the principal component analysis algorithm into step S3.3, use the principal component analysis algorithm to reduce the dimension of the original features, and use the principal components after dimensionality reduction, that is, the principal component features with a contribution rate of more than 80%, as the supplementary features of the meta-classifier.
5. The method for predicting the feasibility of physical interaction of a full touch screen in a smart cockpit according to claim 4 is characterized in that: The S4 is specifically: S4.1: Take distraction level Y = {DLL1, DLL2, DLL3} as the model EPCAM output label; S4.2: Taking the start time t1 of the subtask as the reference point, collect time series data of time length T in reverse with time interval t0, where T is in seconds and collect 30 frames of data per second; S4.3: Each frame of data in S4.2 includes the following parameters: driver age A, gender G, driver operation intention DT, and emotional parameter FL are extracted from the DMS system; road conditions LE, traffic flow complexity TF, and weather parameters TS are extracted from the autonomous driving environment perception system; vehicle speed VS, driving task type DT, and steering wheel angle SC are extracted from the vehicle data collector; and full touch screen parameter vector PHMI and human-computer interaction geometric relationship parameter vector HMGP are extracted from the vehicle built-in parameter table; S4.4: Take the data unit X={A, FL, G, DT, LE, TF, TS, VS, DT, SC, PHMI, HMGP} of time length T as the input of the model EPCAM; S4.5: Build the EPCAM model training platform and test software platform based on Python language, and set the model parameter θ optimization process: The prior probability distribution of parameter θ is P(θ), the probability distribution of output Y under parameter θ and input X is P(Y|X,θ), parameter θ i The probability distribution under the input X is P(Y|X,θ i ), the posterior probability distribution of the parameter θ under input X and output Y is P(θ|X,Y). According to Bayes' theorem, the posterior probability distribution formula is The Gaussian process model is used to describe the posterior probability distribution, and the kernel function selected is k(x,x′)=φ(x) T φ(x′), φ(x′) is the eigenvector of x, and the kernel matrix is further expressed as By calculating the inverse of the kernel matrix, we can get the posterior probability distribution of the hyperparameters Finally, the optimal parameters are selected using the posterior probability distribution; S4.6: Start training the model EPCAM, determine that the verification result of the model EPCAM reaches the optimal value, the test accuracy of the model EPCAM on the verification set reaches more than 98%, and then save the parameter weight; The specific training process includes: Input training data set D is randomly divided into five subsets D = {D1, D2, D3, D4}; Train the base classifier using data D k Training c ,hc∈{h RF (x i ),h GBDT (x i ),h XGBoost (x i ),h LightGBM (x i )}; Construct a new feature matrix {x i ′,y i }(x i ′={h RF (x i ),h GBDT (x i ),h XGBoost (x i ),h LightGBM (x i ),P},P is the low-dimensional feature obtained by principal component analysis; Train a meta-classifier using the new features (x i ′,y i ) Train h′, ensemble classifier E(x) = h′; Cross-training and validation are performed based on the data subset D = {D1, D2, D3, D4}. The validation set accuracy reaches more than 98%, and the E(x) parameter weights are saved.
6. The method for predicting the feasibility of physical interaction of a full touch screen in a smart cockpit according to claim 5 is characterized in that: The S5 is specifically: S5.1: Build a real vehicle test hardware platform, including the vehicle-mounted DMS system, autonomous driving environment perception system, data collection unit, microprocessor unit and early warning device; S5.2: Optimize model EPCAM based on test feedback; S5.3: Recruit drivers for real vehicle testing again, and use the Likert subjective questionnaire to collect driver satisfaction. When the overall satisfaction is higher than 95%, it is judged that θ is valid. If the overall satisfaction is lower than 95%, repeat steps 2, 3, and 4, and optimize the main parameters of EPCAM again according to the method in S4.5 until S5 meets the conditions.
7. The method for predicting the feasibility of physical interaction of a full touch screen in a smart cockpit according to claim 6 is characterized in that: The S6 is specifically: S6.1: Setting the distraction level threshold DLL0 and the driving risk field index level threshold FX0; S6.2: Obtain the final EPCAM model parameters from step S5.1; S6.3: The system starts working, and the data collection unit collects and stores real-time data; S6.4: The DMS system determines whether the driver wants to start operating the full touch screen and records the time T0 when the driver starts operating the full touch screen; S6.5: call the data units {A, FL, G, DT, LE, TF, TS, VS, DT, SC, PHMI, HMGP} in the time period T0 before time T0; S6.6: Send the data unit to the EPCAM algorithm module in the microprocessor to predict the distraction level DLL generated by the driver continuing to perform physical interaction; S6.7: Calculate the driving risk field index FX at time T0 in the microprocessor unit; S6.8: In the microprocessor unit, judging whether it is necessary to provide prompt information to the driver according to the distraction degree prediction level DLL and the driving risk field index FX; when DLL>DLL0 and FX>FX0, proceeding to the next step; S6.9: To remind the driver, the microprocessor controls the vehicle signal lights and steering wheel vibration device to send out warning signals of corresponding levels. At the same time, the voice device sends out a prompt tone of "it is recommended to use voice for interaction".
8. A device for predicting the feasibility of physical interaction of a full touch screen in an intelligent cockpit, characterized in that: It includes the vehicle-mounted DMS system, the autonomous driving environment perception system, the data collection unit, the microprocessing unit and the early warning device; among them, The vehicle-mounted DMS system acquires the driver’s emotions, fatigue and attention in real time, and is connected to the data collection unit via a signal transmission harness; The autonomous driving environment perception system is used to output driving environment characteristic parameters, including road conditions, traffic flow conditions and weather conditions, and is connected to the data collection unit through a signal transmission harness; The data collection unit is used to collect the relevant data of the driver's operation of the touch screen and store it, waiting for the call of the microprocessor unit; One end of the microprocessing unit is connected to the data collection unit via an input data interface, and the other end is connected to the early warning device via an output data interface; Early warning device: The early warning device includes a vehicle-mounted signal light, a steering wheel vibration device and a voice device; the vehicle-mounted signal light can flash at different frequencies, the steering wheel vibration device can vibrate the steering wheel at different frequencies, and the voice device can broadcast prompt information.
Citation Information
Patent Citations
Cognitive distraction assessing method and system
CN107233104A
Monitoring and intervention device for self-adaption distraction driving behaviors
CN108248610A
Hierarchical driver cognitive distraction detection system
CN113171095A
Distraction driving behavior recognition method and device and storage medium
CN114529889A
Online car-hailing distracted driving behavior identification method based on multi-source data
CN117576605A
