An intelligent cockpit full-touch screen physical interaction feasibility prediction method and device

Abstract

The application provides a kind of intelligent cockpit full touch screen physical interaction feasibility prediction method and device, it is characterized in that: including the following steps: S1: constructs intelligent cockpit full touch screen distraction data set;S2: constructs distraction degree label generation method based on multilayer perception machine and dynamic peak clustering algorithm;S3: constructs distraction degree identification model EPCAM based on integrated stack model;S4: distraction degree prediction model is trained;S5: distraction degree prediction model is tested in real vehicle;S6: distraction degree identification model EPCAM is integrated into micro processing unit, executes physical interaction feasibility prediction specific implementation method.Affirmative effect: the method is aimed at intelligent cockpit full touch screen interaction, can predict the distraction degree of driver, evaluates the risk of driver continuing physical interaction between vehicle and vehicle in combination with vehicle operation risk, and in the case where conditions are met, reminds driver through active interaction, to avoid dangerous driving behavior in advance.

Description

TECHNICAL FIELD

[0001] The application relates to an intelligent cabin full-touch screen physical interaction feasibility prediction method and device, and belongs to the technical field of road traffic driving safety. BACKGROUND

[0002] With the proposal of the concepts of automobile intelligence and intelligent cabin, more and more vehicles begin to use large-size full-touch screens for interaction. This interaction mode has become a main source of distraction for drivers. When a driver physically interacts with a full-touch screen, the driver will bear a large visual and manual distraction load, which will seriously endanger driving safety. So far, a large number of methods for monitoring the state of a driver have been proposed, but these methods focus on real-time detection of whether a driver is distracted and do not involve the prediction of the degree of distraction of a driver. No invention has focused on whether a driver should interact with a machine at a certain moment to avoid dangerous driving behavior in advance. If a device can be developed to predict the degree of distraction and behavior risk of a driver when the driver is about to physically interact with a full-touch screen and give the driver guidance on whether the driver can continue to interact, it will be of great significance to improve driving safety, promote the development of the intelligent cabin industry and improve the relationship between man and machine. Therefore, the application focuses on the above-mentioned scenario and proposes a method and device suitable for predicting the feasibility of physical interaction of an intelligent cabin full-touch screen and reminding a driver of safety, aiming to improve the safety of physical interaction of a driver on a full-touch screen. SUMMARY

[0003] The application provides an intelligent cabin full-touch screen physical interaction feasibility prediction method to solve the problems mentioned in the background.

[0004] Technical scheme: An intelligent cabin full-touch screen physical interaction feasibility prediction method, characterized by comprising the following steps:

[0005] S1: Constructing an intelligent cabin full-touch screen distraction data set;

[0006] S2: Constructing a distraction degree label generation method based on a multi-layer perception machine and a dynamic peak clustering algorithm;

[0007] S3: Constructing a distraction degree recognition model EPCAM based on an integrated stacking model;

[0008] S4: Training a distraction degree prediction model;

[0009] S5: Testing the distraction degree prediction model in a real vehicle;

[0010] S6: Integrating the distraction degree recognition model EPCAM into a micro processing unit to execute a physical interaction feasibility prediction specific implementation method.

[0011] The S1 is specifically:

[0012] S1.1: Conduct market research and select N0 vehicle models with intelligent cockpit full-touch screens;

[0013] S1.2: Design a driver interaction test, and select N1 interaction tasks (making a call using the vehicle-mounted full-touch screen, adjusting the air conditioner air volume using the vehicle-mounted full-touch screen, navigating using the vehicle-mounted full-touch screen, etc.) according to the vehicle's interactive functions;

[0014] S1.3: Recruit N2 drivers with different genders, driving ages, and ages; each driver performs N3 times for each interaction task; design N4 actual driving routes, and perform N5 vehicle speeds for each actual driving route; the total number of interaction task samples obtained is N0×N1×N2×N3×N4×N5=N T ;

[0015] S1.4: Install camera 1 and camera 2 to collect the driver's facial image and hand image, respectively; camera 1 is used to collect the driver's hand operation state and store it as an RGB image; camera 2 is used to collect the facial visual and emotional state data and store it in the form of an RGB image; a vehicle data acquisition instrument is used to collect vehicle motion state parameters and save them as text data;

[0016] S1.5: After data collection, a deep learning model is used to extract distraction features, driver states, vehicle motion states, and vehicle operation parameters;

[0017] A MediaPipe-Hand hand feature point extraction model is used to analyze the hand video of camera 1 and extract hand load parameters in a single interaction task, including the maximum value of hand single departure from the steering wheel time HLMAX, the average hand single departure time HLM, and the total hand departure time HLT; a MediaPipe-Face hand feature point extraction model and an iTracker driver gaze area recognition model are used to extract driver visual distraction features in a single interaction task, including the maximum gaze touch screen time DGMAX, the average gaze touch screen time DGM, and the total gaze touch screen time DGS; the steering wheel rotation rate index VSWTR, the steering wheel angular velocity entropy VSWS, and the vehicle lateral position standard deviation SDLP are extracted from the vehicle data acquisition instrument; in addition, the driver's tension emotion index FL is extracted from the facial image through the DMS system; the parameter matrix representing the degree of driving distraction is finally expressed as DLL H ={HLMAX, HLM, HLT, DGMAX, DGM, DGS, VSWTR, VSWS, SDLP, FL}, and the dimension of the matrix is 10×N.T .

[0018] The S2 is specifically:

[0019] S2.1: Constructing a feature dimension reduction algorithm F based on multi-layer perception MLP , the original parameter matrix DLL H is reduced to DLL L ;

[0020] First, through the encoder Encoder, DLL H is reduced to DLL L :

[0021] Encoder(DLL H ):w(DLL H )+b

[0022] Then through the decoder Decoder, DLL L is reconstructed to DLL H :

[0023] Decoder(DLL L ):w T (DLL L )+b′

[0024] By minimizing the reconstruction error, the parameters of F MLP are optimized:

[0025] min(F MLP :((w T (wDLL H +b)+b′),DLL H ))

[0026] S2.2: Constructing a distraction degree discretization label generation algorithm G based on density peak L , the algorithm G L introduces the concept of local density; the local density of data point x i is defined as ρ i , and its expression is

[0027]

[0028] In the formula, d c represents the truncation distance, d ij represents the Euclidean distance between data point x i and data point x j , χ(·) is a logical judgment function, and the expressions of d ij and χ(·) are as follows:

[0029] d ij=||x i -x j ||

[0030]

[0031] data points x i minimum distance δ of any one of the other high local density points i is expressed as:

[0032]

[0033] After determining the ρ of the data points i and δ i , a decision graph is drawn, where ρ i is the horizontal axis of the decision graph and δ i is the vertical axis of the decision graph; based on the decision graph, points with both ρ i and δ i relatively high (also referred to as peaks) are marked as cluster centers, and points with ρ i relatively low but δ i relatively high are marked as noise. Based on the determined cluster centers, the remaining data points are assigned to the cluster center where the data point is closest to and has a greater density than the data point;

[0034] According to the algorithm G L , the reduced load characterization parameters DLL L are clustered into discrete load degree labels Y = {DLL1, DLL2, DLL3}, where DLL1 represents a low distraction degree, DLL2 represents a moderate distraction degree, and DLL3 represents a high distraction degree.

[0035] The S3 is specifically:

[0036] S3.1: Construct a distraction degree recognition model EPCAM based on an integrated stacking model, select a base classifier 1, a base classifier 2, a base classifier 3, a base classifier 4, and a base classifier 5 from a machine learning model library;

[0037] S3.2: Select a meta-classifier from a machine learning model library, including LR: a logistic regression model; DNN: a deep neural network model; KNN: a K nearest neighbor model;

[0038] S3.3: Select a base classifier and a meta-classifier integration strategy from a knowledge base, integrate the model, and use public data to verify the model to determine whether the classification performance is above 98% before proceeding to the next step;

[0039] S3.4: Improve the integrated model defects, further optimize the model anti-overfitting performance; incorporate principal component analysis algorithm into step S3.3, use principal component analysis algorithm to reduce dimensionality of original features, and use the principal components after dimensionality reduction, i.e. the principal component features with contribution rate exceeding 80%, as supplementary features of the meta-classifier.

[0040] The S4 is specifically:

[0041] S4.1: The distraction degree Y = {DLL1, DLL2, DLL3} is taken as the model EPCAM output label.

[0042] S4.2: Taking the time t1 when the subtask starts as the reference point, the time series data of T time length is collected in reverse with t0 as the time interval, T is in seconds, and 30 frames of data are collected per second.

[0043] S4.3: Each frame of data in S4.2 includes the following parameters: driver age A, gender G, driver operation intention DT, and emotion parameter FL are extracted from the DMS system; road conditions LE, traffic flow complexity TF, and weather parameters TS are extracted from the automatic driving environment perception system; vehicle speed VS, driving task type DT, and steering wheel angle SC are extracted from the vehicle data acquisition instrument; and full-touch screen parameter vector PHMI and human-machine interaction geometric relationship parameter vector HMGP are extracted from the vehicle built-in parameter table.

[0044] S4.4: The data unit X = {A, FL, G, DT, LE, TF, TS, VS, DT, SC, PHMI, HMGP} of T time length is taken as the input of the model EPCAM.

[0045] S4.5: Build an EPCAM model training platform and test software platform based on python language, and set the model parameter θ optimization process:

[0046] The prior probability distribution of parameter θ is P(θ), the probability distribution of output Y under parameter θ and input X is P(Y|X, θ), and the posterior probability distribution of parameter θ under input X and output Y is P(θ|X, Y). Through Bayes' theorem, the posterior probability distribution formula is i i The posterior probability distribution of parameter θ under input X and output Y is P(θ|X, Y). Through Bayes' theorem, the posterior probability distribution formula is Use Gaussian process model to describe the posterior probability distribution, and select the kernel function as k(x, x') = φ(x) T φ(x'), φ(x') is the feature vector of x, and the kernel matrix is further represented as By calculating the inverse of the kernel matrix, the posterior probability distribution of the hyperparameter can be obtained Finally, the optimal parameter is selected using the posterior probability distribution. ​

[0047] S4.6: Start training model EPCAM, judge whether the model EPCAM verification result reaches the optimal value, the test accuracy of the model EPCAM on the verification set reaches 98% or more, and then save the parameter weight. The specific training process includes:

[0048] Input training data set D is randomly divided into five subsets D={D1, D2, D3, D4};

[0049] Train base classifier, use data D k Train h c ,hc∈{h RF (x i ),h GBDT (x i ),h XGBoost (x i ),h LightGBM (x i )}.

[0050] Construct a new feature matrix {x i ′,y i}(x i ′={h RF (x i ),h GBDT (x i ),h XGBoost (x i ),h LightGBM (x i ),P}, P is the low-dimensional feature obtained by principal component analysis;

[0051] Train meta-classifier, train h' using new features (x i ′,y i ), and integrate the classifier E(x)=h';

[0052] Based on data subsets D={D1, D2, D3, D4}, cross-training and verification are performed, the verification set accuracy reaches 98% or more, and the E(x) parameter weight is saved.

[0053] The S5 is specifically:

[0054] S5.1: Build a real vehicle test hardware platform, including a vehicle-mounted DMS system, an automatic driving environment perception system, a data collection unit, a micro processing unit and a warning device;

[0055] S5.2: Optimize the model EPCAM according to the test feedback;

[0056] S5.3: Recruit drivers again for real vehicle testing, fill in the model prediction result test feedback questionnaire, judge the effectiveness of the subjective and objective judgment method, use the Richter subjective questionnaire to collect the satisfaction of the driver, when the overall satisfaction is higher than 95%, judge that the theta is effective, if the overall satisfaction is lower than 95%, repeat steps 2, 3, 4, and optimize the main parameters of EPCAM according to the method in S4.5, until step 5 meets the conditions.

[0057] The S6 is specifically:

[0058] S6.1: Set the distraction level threshold DLL0 and the driving risk field index level threshold FX0;

[0059] S6.2: Obtain the final EPCAM model parameters from step S5.1;

[0060] S6.3: The system starts to work, and the data collection unit performs real-time data collection and storage;

[0061] S6.4: The DMS system judges whether the driver wants to start operating the full-touch screen, and records the time T0 of starting to operate the full-touch screen;

[0062] S6.5: Call the data unit {A, FL, G, DT, LE, TF, TS, VS, DT, SC, PHMI, HMGP} in the T0 time period before T0 moment;

[0063] S6.6: Send the data unit into the EPCAM algorithm module in the microprocessor to predict the distraction level DLL generated by the driver's continuous physical interaction;

[0064] S6.7: Calculate the driving risk field index FX at T0 moment in the microprocessing unit;

[0065] S6.8: In the microprocessing unit, judge whether the driver needs to be prompted according to the distraction level prediction grade DLL and the driving risk field index FX; when DLL>DLL0 and FX>FX0, go to the next step;

[0066] S6.9: Remind the driver, and the microprocessor controls the vehicle-mounted signal light and the steering wheel vibration device to issue a corresponding level of early warning signal, and at the same time, the voice device issues a prompt sound of "suggestion to use voice for interaction".

[0067] An intelligent cockpit full-touch screen physical interaction feasibility prediction device, comprising a vehicle-mounted DMS system, an automatic driving environment perception system, a data collection unit, a microprocessing unit and a warning device; wherein,

[0068] The vehicle-mounted DMS system acquires the driver's emotion, fatigue and attention in real time, and is connected to the data collection unit through a signal transmission wire harness;

[0069] The automatic driving environment perception system is used to output driving environment characteristic parameters, including road conditions, traffic flow conditions and weather conditions, and is connected to the data collection unit through a signal transmission wire harness;

[0070] The data collection unit is used to collect relevant data of the driver operating the touch screen and store it for the calling of the micro-processing unit;

[0071] The micro-processing unit is connected to the data collection unit through an input data interface at one end and connected to the early warning device through an output data interface at the other end; the early warning logic judgment module is used to output an early warning signal according to the distraction degree and risk field parameters;

[0072] The early warning device includes a vehicle-mounted signal lamp, a steering wheel vibration device and a voice device; the vehicle-mounted signal lamp can flash at different frequencies, the steering wheel vibration device can make the steering wheel vibrate at different frequencies, and the voice device can broadcast prompt information.

[0073] Beneficial effects: the distraction degree recognition model EPCAM based on the integrated stacking model is constructed, the cabin touch screen space arrangement geometric parameters, touch screen physical response parameters, driver body parameters, driver attribute parameters, vehicle motion parameters and road environment parameters are taken as model input parameters, the machine learning clustering results of visual parameters, manual parameters and physiological indexes are taken as model outputs, and finally the distraction degree prediction is realized through model training, testing and integration; then, the feasibility of physical interaction is evaluated according to the driver distraction degree prediction value and the vehicle operation risk representation parameters, when the interaction condition is met, no prompt information is sent, when the interaction condition is not met, the micro-processing unit controls the prompt device to send prompt information; the method can predict the distraction degree of the driver for the full touch screen interaction of the intelligent cabin, evaluate the risk of the driver continuing the physical interaction between the driver and the vehicle in combination with the vehicle operation risk, and remind the driver through active interaction when the condition is met, so as to avoid dangerous driving behaviors in advance. BRIEF DESCRIPTION OF DRAWINGS

[0074] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the provided drawings.

[0075] Fig. 1 The flowchart of the prediction method of the present application.

[0076] Fig. 2 Flow chart for training and testing process of the algorithm of the present application.

[0077] Fig. 3 Logical framework chart for the algorithm of the present application.

[0078] Fig. 4 Schematic diagram of the prediction device of the present application.

[0079] RF: Random Forest; GBDT: Gradient Boosting Decision Tree; XGBoost: eXtreme Gradient Boosting; LightGBM: Light Gradient Boosting Machine algorithm; PAC: Principal Component Analysis; PC1: Principal Component 1, PC2: Principal Component 2; PC3: Principal Component 3; learn: learning; predict: prediction; predictions: predicted values; New feature matrix: new feature matrix; LR: Logistic Regression model; DNN: Deep Neural Network model; KNN: K-Nearest Neighbors model. DETAILED DESCRIPTION

[0080] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0081] In the description of the present application, it should be understood that the terms "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the purpose of facilitating the description of the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application.

[0082] In the present application, unless otherwise explicitly specified and limited, "on" or "under" of a first feature to a second feature can include that the first and second features are in direct contact, or that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, "on", "above" and "above" of the first feature to the second feature include that the first feature is directly above and obliquely above the second feature, or only indicates that the horizontal height of the first feature is higher than that of the second feature. "Below", "below" and "below" of the first feature to the second feature include that the first feature is directly below and obliquely below the second feature, or only indicates that the horizontal height of the first feature is less than that of the second feature.

[0083] As Figs. 1 to 4As shown, a smart cockpit full-touch screen physical interaction feasibility prediction method is characterized by the following steps:

[0084] S1: Construct a smart cockpit full-touch screen distraction data set;

[0085] S1.1: Conduct market research and select N0 vehicle models with smart cockpit full-touch screens;

[0086] S1.2: Design a driver interaction test, and select N1 interaction tasks according to the vehicle's interactive functions (make a call using the vehicle's full-touch screen, adjust the air conditioner's air volume using the vehicle's full-touch screen, navigate using the vehicle's full-touch screen, etc.);

[0087] S1.3: Recruit N2 drivers with different genders, driving ages, and ages; each driver performs N3 times for each interaction task; design N4 actual driving routes, and perform N5 vehicle speeds for each actual driving route; the total number of interaction task samples is N0×N1×N2×N3×N4×N5=N T ;

[0088] S1.4: Install camera 1 and camera 2 to capture the driver's facial image and hand image; camera 1 is used to capture the driver's hand operation state and store it as an RGB image; camera 2 is used to capture the facial visual and emotional state data and store it in the format of an RGB image; the vehicle data acquisition instrument is used to collect vehicle motion state parameters and save them as text data;

[0089] S1.5: After data collection, use a deep learning model to extract distraction features, driver state, vehicle motion state, and vehicle operation parameters;

[0090] Use a MediaPipe-Hand hand feature point extraction model to analyze the hand video of camera 1 and extract hand load parameters in a single interaction task, including the maximum value of hand single departure from the steering wheel time HLMAX, the average hand single departure time HLM, and the total hand departure time HLT; use a MediaPipe-Face hand feature point extraction model and an iTracker driver gaze area recognition model to extract driver visual distraction features in a single interaction task, including the maximum time of single gaze on the touch screen DGMAX, the average gaze on the touch screen time DGM, and the total gaze on the touch screen time DGS; extract the steering wheel rotation rate index VSWTR, the steering wheel angular velocity entropy VSWS, and the vehicle lateral position standard deviation SDLP from the vehicle data acquisition instrument; in addition, extract the driver's tension emotion index FL from the facial image through the DMS system; the parameter matrix representing the degree of driving distraction is finally represented as DLL H= {HLMAX, HLM, HLT, DGMAX, DGM, DGS, VSWTR, VSWS, SDLP, FL}, the dimension of the matrix is 10xN T .

[0091] S2: Construct a distraction degree label generation method based on multi-layer perception and dynamic peak clustering algorithm;

[0092] S2.1: Construct a feature dimension reduction algorithm F based on multi-layer perception MLP , the original parameter matrix DLL H is reduced to DLL L ;

[0093] First, the encoder Encoder reduces DLL H to DLL L :

[0094] Encoder(DLL H ):w(DLL H )+b

[0095] Then, the decoder Decoder reconstructs DLL L to DLL H :

[0096] Decoder(DLL L ):w T (DLL L )+b′

[0097] By minimizing the reconstruction error, the parameters of F MLP are optimized:

[0098] min(F MLP :((w T (wDLL H +b)+b′),DLL H ))

[0099] S2.2: Construct a distraction degree discretization label generation algorithm G based on density peak L , algorithm G L introduces the concept of local density; the local density of data point x i is defined as ρ i , and its expression is

[0100]

[0101] In the formula, d c represents the truncation distance, d ij represents the distance between data point x i and data point x jEuclidean distance between them, χ(·) is a logical judgment function, d ij and χ(·) are expressed as follows:

[0102] d ij =||x i -x j ||

[0103]

[0104] Data points x i and the minimum distance δ i of any other high local density point are expressed as:

[0105]

[0106] After determining the ρ i and δ i of the data points, a decision graph is drawn, where ρ i is the horizontal axis of the decision graph and δ i is the vertical axis of the decision graph; based on the decision graph, points with both relatively high ρ i and δ i (also known as peaks) are marked as cluster centers, and points with relatively low ρ i but high δ i are marked as noise. Based on the determined cluster centers, the remaining data points are assigned to the cluster center where the data point is closest to and has a greater density than it;

[0107] According to the algorithm G L , the reduced load characterization parameters DLL L are clustered into discrete load level labels Y = {DLL1, DLL2, DLL3}, where DLL1 represents low distraction level, DLL2 represents moderate distraction level, and DLL3 represents high distraction level.

[0108] S3: Construct a distraction level recognition model EPCAM based on an integrated stacking model;

[0109] S3.1: Construct a distraction level recognition model EPCAM based on an integrated stacking model, select base classifier 1, base classifier 2, base classifier 3, base classifier 4, and base classifier 5 from the machine learning model library;

[0110] S3.2: Select a meta-classifier from the machine learning model library;

[0111] S3.3: Select the base classifier and meta-classifier integration strategy from the knowledge base, integrate the model, and use public data to verify the model. If the classification performance is above 98%, proceed to the next step;

[0112] S3.4: Improve the integrated model defects, further optimize the model anti-overfitting performance; incorporate principal component analysis algorithm into step S3.3, use principal component analysis algorithm to reduce dimensionality of original features, and use the principal components after dimensionality reduction, i.e. the principal component features with contribution rate exceeding 80%, as supplementary features of meta-classifier.

[0113] S4: Train the distraction degree prediction model;

[0114] S4.1: Take the distraction degree Y = {DLL1, DLL2, DLL3} as the output label of model EPCAM;

[0115] S4.2: Take the time t1 when the secondary task starts as the reference point, and collect time series data of T length in t0 time interval, T is in s unit, 30 frames of data are collected per second;

[0116] S4.3: Each frame of data in S4.2 includes the following parameters: driver's age A, gender G, driver's operation intention DT, emotion parameter FL are extracted from DMS system, road condition LE, traffic flow complexity TF and weather parameter TS are extracted from automatic driving environment perception system, vehicle speed VS, driving task type DT and steering wheel angle SC are extracted from vehicle data acquisition instrument, and full-touch screen parameter vector PHMI and human-machine interaction geometric relationship parameter vector HMGP are extracted from vehicle built-in parameter table;

[0117] S4.4: Take the data unit X = {A, FL, G, DT, LE, TF, TS, VS, DT, SC, PHMI, HMGP} of T length as the input of model EPCAM;

[0118] S4.5: Build EPCAM model training platform and test software platform based on python language, set model parameter θ optimization process:

[0119] The prior probability distribution of parameter θ is P(θ), the probability distribution of output Y under parameter θ and input X is P(Y|X, θ), the posterior probability distribution of parameter θ under input X and output Y is P(θ|X, Y), through Bayes theorem, the posterior probability distribution formula is i i The posterior probability distribution of parameter θ under input X and output Y is P(θ|X, Y), through Bayes theorem, the posterior probability distribution formula is Use Gaussian process model to describe the posterior probability distribution, and select the kernel function as k(x, x') = φ(x) T φ(x'), φ(x') is the feature vector of x, and the kernel matrix is further represented as By calculating the inverse of the kernel matrix, the posterior probability distribution of the hyperparameter can be obtained Finally, the optimal parameter is selected using the posterior probability distribution; ​

[0120] S4.6: Start training model EPCAM, judge whether the model EPCAM verification result reaches the optimal value, the test accuracy of the model EPCAM on the verification set reaches 98% or more, and then save the parameter weight. The specific training process includes:

[0121] Input training data set D is randomly divided into five subsets D={D1, D2, D3, D4};

[0122] Train base classifier, use data D k Train h c , hc∈{h RF (x i ),h GBDT (x i ),h XGBoost (x i ),h LightGBM (x i )}.

[0123] Construct a new feature matrix {x i ′,y i}(x i ′={h RF (x i ),h GBDT (x i ),h XGBoost (x i ),h LightGBM (x i ),P}, P is the low-dimensional feature obtained by principal component analysis;

[0124] Train meta-classifier, train h′ using new features (x i ′,y i ), and integrate classifier E(x)=h′;

[0125] Based on data subsets D={D1, D2, D3, D4}, perform cross-training and verification, and the verification set accuracy reaches 98% or more. Save E(x) parameter weight.

[0126] S5: Real vehicle test distraction degree prediction model;

[0127] S5.1: Build a real vehicle test hardware platform, including a vehicle-mounted DMS system, an automatic driving environment perception system, a data collection unit, a micro processing unit and a warning device;

[0128] S5.2: Optimize model EPCAM according to test feedback;

[0129] S5.3: Recruit drivers again for real vehicle testing, fill in the model prediction result test feedback questionnaire, judge the effectiveness of the subjective and objective judgment method, use the Richter subjective questionnaire to collect the satisfaction of the driver, when the overall satisfaction is higher than 95%, judge that the theta is effective, if the overall satisfaction is lower than 95%, repeat steps 2, 3, 4, and optimize the main parameters of EPCAM according to the method in S4.5, until step 5 meets the conditions.

[0130] S6: Integrate the algorithm into the microprocessor unit and execute the physical interaction feasibility prediction implementation method;

[0131] S6.1: Set the distraction level threshold DLL0 and the driving risk field index level threshold FX0;

[0132] S6.2: Obtain the final EPCAM model parameters from step S5.1;

[0133] S6.3: The system starts working, and the data collection unit performs real-time data acquisition and storage;

[0134] S6.4: The DMS system judges whether the driver wants to start operating the full-touch screen, and records the time T0 of starting to operate the full-touch screen;

[0135] S6.5: Call the data unit {A, FL, G, DT, LE, TF, TS, VS, DT, SC, PHMI, HMGP} of the T0 time period before T0 moment;

[0136] S6.6: Send the data unit to the EPCAM algorithm module in the microprocessor to predict the distraction level DLL generated by the driver's continued physical interaction;

[0137] S6.7: Calculate the driving risk field index FX at T0 moment in the microprocessor unit;

[0138] S6.8: In the microprocessor unit, judge whether a prompt information needs to be provided to the driver according to the distraction level prediction grade DLL and the driving risk field index FX; when DLL>DLL0 and FX>FX0, go to the next step;

[0139] S6.9: Remind the driver, and the microprocessor controls the vehicle-mounted signal light and the steering wheel vibration device to issue a corresponding level of warning signal, at the same time, the voice device issues a prompt sound of "suggestion to use voice for interaction".

[0140] An intelligent cockpit full-touch screen physical interaction feasibility prediction device, comprising a vehicle-mounted DMS system, an automatic driving environment perception system, a data collection unit, a microprocessor unit and a warning device; wherein,

[0141] The vehicle-mounted DMS system acquires the driver's emotion, fatigue and attention in real time, and is connected to the data collection unit through a signal transmission wire harness;

[0142] The automatic driving environment perception system is used to output driving environment characteristic parameters, including road conditions, traffic flow conditions and weather conditions, and is connected to the data collection unit through a signal transmission wire harness;

[0143] The data collection unit is used to collect relevant data of the driver operating the touch screen and store it, waiting for the call of the micro-processing unit.

[0144] The micro-processing unit is connected to the data collection unit through an input data interface at one end and connected to the early warning device through an output data interface at the other end; the early warning logic judgment module is used to output an early warning signal according to the distraction degree and risk field parameters.

[0145] The early warning device includes a vehicle-mounted signal lamp, a steering wheel vibration device and a voice device; the vehicle-mounted signal lamp can flash at different frequencies, the steering wheel vibration device can make the steering wheel vibrate at different frequencies, and the voice device can broadcast prompt information.

[0146] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0147] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for predicting the feasibility of physical interaction with a full touch screen in an intelligent cockpit, characterized by: The following steps are involved: S1: Constructing a smart cockpit full touch screen distraction dataset; S2: Construct a distraction degree label generation method based on multi-layer perceptron and dynamic peak clustering algorithm; S3: Constructing the distraction degree recognition model EPCAM based on the integrated stacking model; S4: training a distraction prediction model; S5: Distraction prediction model for real-car testing; S6: Integrate the distraction level recognition model into the microprocessor unit to perform physical interaction feasibility prediction. The S1 is specifically: S1.1: Conduct market research and select N0 models with full-touch screen intelligent cockpits; S1.2: Design a driver interaction test. Select N1 interactive tasks based on the vehicle's interactive functions, including using the vehicle's full touchscreen to make a phone call, using the vehicle's full touchscreen to adjust the air conditioning volume, and using the vehicle's full touchscreen to navigate. S1.3: Recruit N2 drivers of different genders, driving experience, and ages; each driver performs each interactive task N3 times; design N4 actual driving routes, each of which has N5 different speeds; the total number of interactive task samples finally obtained is N0×N1×N2×N3×N4×N5=N T ; S1.4: Install cameras 1 and 2 to capture facial and hand images of the driver, respectively. Camera 1 is used to collect the driver's hand operation status and store it in RGB image format; Camera 2 is used to collect facial visual and emotional state data and store it in RGB image format; the vehicle data collector is used to collect vehicle motion state parameters and save them as text data; S1.5: After data collection is complete, a deep learning model is used to extract distraction characteristics, driver status, vehicle motion status, and vehicle operating parameters; The MediaPipe-Hand hand feature point extraction model is used to analyze the hand video from camera 1 to extract manual load parameters during a single interaction task, including the maximum time the hand leaves the steering wheel (HLMAX), the average time the hand leaves the steering wheel (HLM), and the total time the hand leaves the steering wheel (HLT). The MediaPipe-Face hand feature point extraction model and the iTracker driver gaze area recognition model are used to extract the driver's visual distraction characteristics during a single interaction task, including the maximum time the driver gazes at the touch screen (DGMAX), the average time the driver gazes at the touch screen (DGM), and the total time the driver gazes at the touch screen (DGS). The steering wheel turning rate index VSWTR, steering wheel angular velocity entropy VSWS, and vehicle lateral position standard deviation SDLP are extracted from the vehicle data acquisition instrument. In addition, the driver's tension index FL is extracted from the facial image through the DMS system. The parameter matrix reflecting the degree of driving distraction is expressed as DLL H ={HLMAX, HLM, HLT, DGMAX, DGM, DGS, VSWTR, VSWS, SDLP, FL}, the dimension of this matrix is ​​10×N T ; The S3 is specifically: S3.1: Construct a distraction level recognition model (EPCAM) based on an ensemble stacking model, and select base classifiers 1, 2, 3, 4, and 5 from the machine learning model library. S3.2: Select a meta-classifier from a library of machine learning models; S3.3: Select the base classifier and meta-classifier integration strategy from the knowledge base, integrate the model, and use public data to verify the model. Once the classification performance reaches above 98%, proceed to the next step. S3.4: Improve the defects of the integrated model and further optimize the model's anti-overfitting performance; integrate the principal component analysis algorithm into step S3.3, use the principal component analysis algorithm to reduce the dimensionality of the original features, and use the principal components after dimensionality reduction, that is, the principal component features with a contribution rate of more than 80%, as supplementary features of the meta-classifier.

2. The method for predicting the feasibility of physical interaction of a full-touch screen in a smart cockpit according to claim 1 is characterized in that: The S2 is specifically: S2.1: Construct a feature dimensionality reduction algorithm F based on a multi-layer perceptron MLP , the original parameter matrix DLL H Dimensionality reduction to DLL L ; First, use the encoder to encode the DLL H Dimensionality reduction to DLL L : ; Then use the decoder to decode the DLL L Refactoring to DLL H : ; By minimizing the reconstruction error, MLP Optimize the parameters: ; S2.2: Construct a distraction degree discretization label generation algorithm G based on density peak L , Algorithm G L The concept of local density is introduced; the data points The local density is defined as , then its expression is: ; Where, represents the cutoff distance, Represents data points With data points The Euclidean distance between is a logical judgment function, and The expression is as follows: ; ; Data Points The minimum distance to any other high local density point Expressed as: ; In determining the data point and Then draw a decision diagram, where is the horizontal axis of the decision diagram, Is the vertical axis of the decision diagram; Based on the decision diagram, and The relatively high points, also called peaks, are marked as cluster centers. Lower but The higher points are marked as noise; based on the determined cluster center, the remaining data points are assigned to the cluster center where the data point with the closest distance and greater density is located; According to the algorithm G L The load representation parameter DLL after dimensionality reduction L Clustering is done into discrete load level labels Y={DLL1, DLL2, DLL3}, where DLL1 represents low distraction level, DLL2 represents medium distraction level, and DLL3 represents high distraction level.

3. The method for predicting the feasibility of physical interaction of a full-touch screen in a smart cockpit according to claim 2 is characterized in that: The S4 is specifically: S4.1: Use distraction level Y = {DLL1, DLL2, DLL3} as the model EPCAM output label; S4.2: Using the start time t1 of the subtask as the reference point, reversely collect time series data for a time period of T at intervals of t0, where T is in seconds and 30 frames of data are collected per second. S4.3: Each frame of data in S4.2 includes the following parameters: driver age A, gender G, driver operation intention DT, and emotion parameter FL extracted from the DMS system; road condition LE, traffic flow complexity TF, and weather parameter TS extracted from the autonomous driving environment perception system; vehicle speed VS, driving task type DT, and steering wheel angle SC extracted from the vehicle data acquisition instrument; full touch screen parameter vector PHMI and human-machine interaction geometric relationship parameter vector HMGP extracted from the vehicle built-in parameter table; S4.4: Take the data unit X = {A, FL, G, DT, LE, TF, TS, VS, DT, SC, PHMI, HMGP} of time length T as the input of the model EPCAM; S4.5: Build a Python-based EPCAM model training platform and testing software platform, and set up the model parameter θ optimization process; S4.6: Start training the EPCAM model, determine whether the EPCAM model verification result reaches the optimal value, and the test accuracy of the EPCAM model on the verification set reaches more than 98%, and then save the parameter weights.

4. The method for predicting the feasibility of physical interaction of a full-touch screen in a smart cockpit according to claim 3 is characterized in that: The S5 is specifically: S5.1: Build a real vehicle testing hardware platform, including the vehicle DMS system, autonomous driving environment perception system, data collection unit, microprocessor unit, and warning device; S5.2: Optimize model EPCAM based on test feedback; S5.3: Recruit drivers for real-vehicle testing again and use the Likert subjective questionnaire to collect driver satisfaction. If the overall satisfaction is higher than 95%, θ is considered valid. If the overall satisfaction is lower than 95%, repeat steps 2, 3, and 4 and optimize the main EPCAM parameters according to the method in S4.5 until S5 meets the conditions.

5. The method for predicting the feasibility of physical interaction of a full-touch screen in a smart cockpit according to claim 4 is characterized in that: The S6 is specifically: S6.1: Set the distraction level threshold DLL0 and the driving risk index level threshold FX0; S6.2: Obtain the final EPCAM model parameters from step S5.1; S6.3: The system starts working and the data collection unit collects and stores real-time data; S6.4: The DMS determines whether the driver begins to operate the touchscreen and records the time T0 at which the driver begins operating the touchscreen. S6.5: Call the data units {A, FL, G, DT, LE, TF, TS, VS, DT, SC, PHMI, HMGP} in the time period T0 before time T0; S6.6: Send the data unit to the EPCAM algorithm module in the microprocessor to predict the distraction level DLL generated by the driver continuing the physical interaction; S6.7: Calculate the driving risk field index FX at time T0 in the microprocessor unit; S6.8: The microprocessor unit determines whether to provide prompt information to the driver based on the predicted distraction level DLL and the driving risk index FX. If DLL>DLL0 and FX>FX0, proceed to the next step. S6.9: To remind the driver, the microprocessor controls the vehicle's signal lights and steering wheel vibration device to issue a warning signal of the corresponding level. At the same time, the voice device issues a prompt tone indicating that "voice interaction is recommended." 6. A device for implementing the method for predicting the feasibility of physical interaction of a full-touch screen in a smart cockpit as described in claim 1, characterized in that: It includes the vehicle-mounted DMS system, the autonomous driving environment perception system, the data collection unit, the microprocessor unit and the early warning device; among them, The on-board DMS system acquires the driver's emotions, fatigue, and attention in real time and is connected to the data collection unit via a signal transmission harness; The autonomous driving environment perception system is used to output characteristic parameters of the driving environment, including road conditions, traffic flow conditions, and weather conditions, and is connected to the data collection unit via a signal transmission harness; The data collection unit is used to collect the relevant data of the driver's operation of the touch screen and store it, waiting for the call of the microprocessor unit; One end of the microprocessor unit is connected to the data collection unit via an input data interface, and the other end is connected to the early warning device via an output data interface; Early warning device: The early warning device includes a vehicle-mounted signal light, a steering wheel vibration device and a voice device; the vehicle-mounted signal light can flash at different frequencies, the steering wheel vibration device can vibrate the steering wheel at different frequencies, and the voice device can broadcast prompt information.

Images