A DMS driver distraction recognition system

Abstract

The invention belongs to the technical field of automobile driving research, and discloses a DMS driver distraction recognition system; the system comprises: collecting three-dimensional head parameters of a driver, establishing a three-dimensional head model of the driver according to the three-dimensional head parameters of the driver; using a ray tracing algorithm according to the three-dimensional head model of the driver to calculate and obtain the driver's front visual field range; building a driver's line of sight direction prediction model for predicting the driver's line of sight direction according to a deep learning algorithm; collecting three-dimensional head posture data of the driver; inputting the three-dimensional head posture data of the driver into the driver's line of sight direction prediction model; when the driver's line of sight direction predicted by the driver's line of sight direction prediction model is not within the front visual field range, the driver is distracted; and performing multi-modal warning feedback; effectively improving driving safety; and effectively coping with driving safety hazards.

Description

Technical Field

[0001] The present invention relates to the technical field of automobile driving research, and more particularly to a DMS driver distraction recognition system. Background Art

[0002] The patent with the authorization announcement number CN114943956A discloses a method, system and vehicle for identifying distracted driving in multiple scenarios. The method includes: constructing an initial distracted driving identification model, obtaining a sample set of distracted driving data in multiple scenarios, training the initial distracted driving identification model using the sample set of distracted driving data in multiple scenarios, and obtaining a target distracted driving identification model; obtaining vehicle parameters, surrounding environment information and driver head data in real time, determining driving scene information according to vehicle parameters and surrounding environment information, and inputting the vehicle parameters, driver head data and driving scene information into the target distracted driving identification model to obtain the driver's driving state; and determining warning information according to the driver's driving state. The present invention achieves accurate identification of distracted driving behaviors in various driving scenarios and issues alarms for distracted driving behaviors by establishing a distracted driving identification model in multiple scenarios, thereby improving driving safety.

[0003] At present, with the continuous complication of road traffic environment, driver distraction has become one of the main factors leading to traffic accidents; however, existing technologies have many deficiencies in detecting and warning drivers of distracted behaviors; on the one hand, there is a lack of effective ways to accurately calculate the driver's real-time field of view, and it is impossible to provide an accurate benchmark field of view for distracted behavior judgment, resulting in low accuracy of distraction detection; on the other hand, existing technologies cannot accurately obtain the driver's line of sight direction, and it is difficult to effectively compare the driver's field of view with the field of view facing forward, so it is impossible to efficiently detect whether the driver is distracted by line of sight deviation and distracted attention; in addition, existing distraction detection algorithms lack the ability to integrate multiple advanced models, and the detection accuracy and robustness are limited. They are easily affected by complex driving environments such as road conditions and lead to inaccurate detection; existing technologies have obvious deficiencies in accurately obtaining driving field of view, real-time detection of line of sight deviation, accuracy and robustness of detection algorithms, etc., which restricts the efficient detection and warning of driver distracted behaviors and cannot prevent potential driving safety hazards in a timely manner.

[0004] In view of this, the present invention proposes a DMS driver distraction recognition system to solve the above problem. Summary of the invention

[0005] In order to overcome the above-mentioned defects of the prior art and to achieve the above-mentioned purpose, the present invention provides the following technical solutions: A DMS driver distraction recognition system, comprising: a model parameter establishment module, for collecting the driver's three-dimensional head parameters, and establishing the driver's three-dimensional head model according to the driver's three-dimensional head parameters;

[0006] A visual field calculation module, which is used to calculate and obtain the driver's visual field range facing forward by using a ray tracing algorithm based on the driver's three-dimensional head model;

[0007] A model building module, for building a driver's sight direction prediction model for predicting the driver's sight direction according to a deep learning algorithm;

[0008] The distraction judgment module is used to collect the driver's three-dimensional head posture data; input the driver's three-dimensional head posture data into the driver's line of sight direction prediction model; when the driver's line of sight direction predicted by the driver's line of sight direction prediction model is not within the field of vision directly ahead, the driver is distracted; and multi-modal warning feedback is performed; each module is connected by wire and / or wireless means to realize data transmission between modules.

[0009] Furthermore, the method of collecting the driver's three-dimensional head parameters includes:

[0010] Single IR infrared cameras are installed on the left and right sides of the vehicle terminal respectively. The single IR infrared cameras are aimed at the driver's head to collect basic depth images. The basic depth images contain a three-dimensional point cloud data set of depth information.

[0011] Through the radius search algorithm, feature points are found in the 3D point cloud data set, and the 3D point cloud feature set of the driver's head area is extracted;

[0012] The method of extracting the three-dimensional point cloud feature set includes:

[0013] A radius search algorithm based on KD tree is constructed; a KD tree K of a 3D point cloud dataset is constructed; for a query point p; a backtracking search algorithm is used to search for all points within a radius r of p on K;

[0014] The process of the backtracking search algorithm includes:

[0015] Initialize the priority queue Q; insert the root node of K into Q; if Q is not empty, pop up the first node m of the priority queue; if ||mp|| is less than or equal to r, then m is a point within the radius r of p; if ||mp|| is greater than r, insert the child node of m into Q; iterate to obtain the initial point cloud feature set;

[0016] Define a 3D point cloud continuous frame registration algorithm based on iterative closest point; use the 3D point cloud continuous frame registration algorithm to process the initial point cloud feature set to obtain a 3D point cloud feature set;

[0017] A three-dimensional convolutional neural network is constructed, and a three-dimensional point cloud feature set is used as the input of the three-dimensional convolutional neural network, and the three-dimensional convolutional neural network outputs the driver's three-dimensional head parameters; the driver's three-dimensional head parameters include but are not limited to the coordinates of key head points, facial mesh parameters and skull structure parameters.

[0018] Furthermore, the method of using the 3D point cloud continuous frame registration algorithm to process the initial point cloud feature set to obtain the 3D point cloud feature set includes:

[0019] Define the objective function of the 3D point cloud continuous frame registration algorithm ;

[0020] ;in, The rotation matrix representing the rotation transformation; The translation vector representing the translation transformation; Source point cloud The points; Target point cloud The points; is the Euclidean norm;

[0021] Divide the points in the initial point cloud feature set into source point cloud and target point cloud, and transform them;

[0022] The transformation methods include:

[0023] initialization, , ;for Every point in ,exist Find the distance The nearest point ; seek and , so that the function value of the objective function is minimized;

[0024] Will and Apply to On the above, we get the transformed point cloud ;Will Set as , and repeat the transformation until the preset number of iterations is reached;

[0025] Calculate the registration degree of each pair of points, and select the point pairs with a registration degree greater than or equal to the preset threshold as stable registration points; the points that appear in more than b frames among the stable registration points constitute the 3D point cloud feature set;

[0026] The calculation formula of the registration degree is:

[0027] ;in, for and The degree of alignment; for and The total number of frames contained in the two point clouds; For the The rotation matrix of the frame's rotation transformation; For the The translation vector of the frame's translation transform.

[0028] Furthermore, the method of constructing a three-dimensional convolutional neural network includes:

[0029] Define the basic architecture of a 3D convolutional neural network, which includes an input layer, an encoder, a decoder, and an output layer;

[0030] The encoder includes s adaptive point convolution layers; each adaptive point convolution layer includes 3 adaptive point convolution units; the adaptive point convolution unit performs feature extraction for each point to obtain a preliminary feature map; and a clue node map is obtained according to the preliminary feature map;

[0031] The decoder is used to process the preliminary feature map to obtain an enhanced feature map;

[0032] The output layer includes an error regression branch and an analytical geometry branch, which are used to output the three-dimensional head parameters of the driver;

[0033] The input of the error regression branch is the enhanced feature map output by the decoder, which is mapped to the regression value through the fully connected layer to predict the coordinates of the key points of the head and the parameters of the skull structure;

[0034] The analytical geometry branch also takes the enhanced feature map output by the decoder as input, generates NURBS patch parameters, and reconstructs the facial mesh parameters of the driver's head surface according to the NURBS patch parameters;

[0035] Define Gaussian loss function And the projection loss function ; Perform weighted fusion of Gaussian loss function and projection loss function to obtain joint loss function ;in, is the Gaussian loss weight; is the projection loss weight;

[0036] Collect the 3D point cloud feature sets of v drivers’ heads as sample sets; divide the sample sets into point cloud training sets and point cloud verification sets;

[0037] The point cloud training set is divided into H batches, and the three-dimensional convolutional neural network is input into the batches, and the function value of the joint loss function is calculated; if the function values ​​no longer decrease after G consecutive times, the three-dimensional convolutional neural network is saved at this time, and the effect of the three-dimensional convolutional neural network is verified using the point cloud verification set; the three-dimensional convolutional neural network with the best effect is saved as the final three-dimensional convolutional neural network.

[0038] Furthermore, the processing to obtain the enhanced feature map includes:

[0039] According to the conditional random field modeling, the preliminary feature map and the clue node map are jointly upsampled to obtain an enhanced feature map;

[0040] Define the conditional random field energy function for conditional random field modeling ;

[0041] ;

[0042] in, For a single node feature First-order potential function; Node feature and The higher-order potential function between them; and is the node index in the preliminary feature graph;

[0043] The method of obtaining the clue node graph according to the preliminary feature graph includes:

[0044] Step 1: Apply the K-means-based vector quantization algorithm to the preliminary feature map to generate a codebook V with a size of 256; use the hash algorithm on the codebook V to generate clue nodes;

[0045] Step 2: Build a clue node graph, define the similarity between nodes, and connect nodes whose similarity is greater than a preset similarity threshold;

[0046] The calculation formula of the similarity is: ;in, Represents a clue node The eigenvector of Represents a clue node The eigenvector of for and similarity;

[0047] Repeat steps 1 and 2 to construct L-layer preliminary clue node graphs; merge all preliminary clue node graphs to form a clue node graph;

[0048] The formula for the joint upsampling is: ;in, To enhance the feature map; It is the clue node diagram; For a given clue node graph Under the condition of The conditional random field energy function of ;

[0049] The Gaussian loss function ;in, The real head key point coordinates / skull structure parameters; The predicted head key point coordinates / skull structure parameters; is the standard deviation of the Gaussian kernel;

[0050] The projection loss function ;

[0051] in, is the actual facial mesh parameter; are the predicted facial mesh parameters; is the lateral coordinate in the NURBS patch parameter space; It is the vertical coordinate in the NURBS patch parameter space.

[0052] Furthermore, the method of establishing the driver's three-dimensional head model includes:

[0053] Using the skull structure parameters, extract the basic skeleton framework of the driver's head; the basic skeleton framework includes the size, angle and relative position of each part of the skull;

[0054] According to the coordinates of the key points of the head, the key facial feature points are located on the basic skeleton frame; the key facial feature points include but are not limited to the center of the eyebrows, the corners of the eyes, the tip of the nose and the corners of the mouth; the key facial feature points reflect the typical features of the driver's face;

[0055] Based on the basic skeleton framework and referring to the facial mesh parameters, the triangular mesh of the driver's head is constructed using 3D modeling software. The triangular mesh constructs the 3D surface morphology of the driver's head through the combination of the basic elements of vertices, edges and faces.

[0056] The generated 3D surface morphology is subdivided into 4 rows, and the key facial feature points are used as control handles to locally deform and adjust the triangular patch mesh; thus, the construction of the driver's 3D head model is completed.

[0057] Furthermore, the method of obtaining the driver's forward field of vision includes:

[0058] Importing the driver's three-dimensional head model and the in-car environment object model into a virtual three-dimensional scene; the virtual three-dimensional scene is a virtual reality representation of a pre-established car driving interior scene;

[0059] According to the driver's 3D head model, the driver's binocular viewpoint is determined as , the vertical field of view angle is , the horizontal field of view angle is ;

[0060] Then the cone equation of the driver's binocular frontal field of view is obtained as: ;in, are the coordinates of any point in the frustum of the normal field of view;

[0061] Calculate the intersection points of the light and each object in the interior scene of the car driving; the calculation method of the intersection points includes:

[0062] According to the light equation ;in, is the driver's binocular viewpoint, is the light direction vector, is the ray parameter;

[0063] For any one An object composed of triangles, if the Triangles By three vertices , , Construct, obtain The unit normal vector ;

[0064] Then the light and Intersection Point The calculation formula is: ;

[0065] For each intersection point , check whether there is an occluding object, from the viewpoint Send shadow rays to , if the ray does not intersect any other object, then is a visible point; record all visible points;

[0066] Check whether the recorded visible point is within the emmetropia cone; if so, record the point as the emmetropia point;

[0067] The three-dimensional coordinates of all the front view points are projected onto a two-dimensional plane to obtain a point set on the two-dimensional plane, which is the driver's front view range.

[0068] Furthermore, the driver's sight direction prediction model is established in the following manner:

[0069] Collecting S groups of historical three-dimensional head posture data and in-vehicle environment data of the driver as initial data; the three-dimensional head posture data of the driver includes the head joint position and the head joint displacement; the in-vehicle environment data includes the vehicle speed and the steering wheel angle; annotating the initial data with the driver's line of sight direction; using the initial data annotated with the driver's line of sight direction as training data;

[0070] Define a network framework for the driver's line of sight direction prediction model; the network framework includes a multimodal data encoding layer, a modal interaction attention layer, a spatiotemporal modeling layer, and a line of sight direction prediction layer;

[0071] The sight direction prediction layer inputs the modeling feature vector into the fully connected layer and outputs a three-dimensional sight direction vector; performs L2 norm normalization on the three-dimensional sight direction vector to obtain a unit sight direction vector; discretizes the driver's sight direction into predetermined categories; the predetermined categories include but are not limited to a front view category, a left view category, a right view category, and a distracted sight category;

[0072] 3D sight direction vector ; In the formula, is the weight matrix of the fully connected layer; is the modeling feature vector; is the bias term;

[0073] The unit sight direction vector is used as the center sight of the frustum, and the angle range is set as the frustum volume. The angle range includes horizontal angle and vertical angle.

[0074] For each predetermined category, determine a corresponding three-dimensional space cone area; calculate the unit sight direction vector The angle distance with each three-dimensional space cone area; the unit sight direction vector is attributed to the three-dimensional space cone area corresponding to the predetermined category with the minimum angle distance value;

[0075] The calculation formula of the angle distance is: ; In the formula, The cone area in three-dimensional space The central sight unit vector of is the unit sight direction vector Cone area in three-dimensional space Angular distance;

[0076] Define the loss function of the driver's line of sight direction prediction model; for example, the cross entropy loss function; divide the training data into a distraction training set and a distraction validation set;

[0077] The distraction training set is divided into X1 batches, and the driver's line of sight direction prediction model is input into the batches. The function value of the loss function is recorded every time a batch is input; and an optimizer is added to optimize the parameters of the driver's line of sight direction prediction model; until the function value of the loss function does not decrease in X2 consecutive batches, the driver's line of sight direction prediction model corresponding to this time and the previous X3 batches is saved; the accuracy of the X3 driver's line of sight direction prediction models is verified using the distraction verification set; the driver's line of sight direction prediction model with the best accuracy is used as the driver's line of sight direction prediction model that has been finally trained.

[0078] Furthermore, the multimodal data encoding layer encodes the three-dimensional head posture data and the in-vehicle environment data respectively to obtain feature vector representations of the three-dimensional head posture data and the in-vehicle environment data;

[0079] The modal interaction attention layer is used to calculate the correlation score matrix between data; and use the correlation score matrix to calculate the weighted sum vector; and input the weighted sum vector into the GRU unit to perform feature selection and adaptive weight adjustment; finally, the weighted fusion feature vector corresponding to each data is output;

[0080] For any 3D head posture data and in-car environment data, calculate the correlation score ;

[0081] ; In the formula, is a learnable weight matrix; For data The transposed vector of the vector representation of ; For data The vector representation of ;

[0082] The correlation scores of the three-dimensional head posture data and the in-vehicle environment data are combined into a correlation score matrix;

[0083] For each data vector representation, calculate its weighted sum vector according to its correlation score with other data ; In the formula, For data The weighted sum vector of ; is the number of all data; for and data The attention weight represented by the vector;

[0084] ;in, is an index for summing; For data and data The correlation score between

[0085] The spatiotemporal modeling layer combines the weighted fused feature vectors into a feature sequence according to the corresponding data collection time; inputs the feature sequence into the spatiotemporal modeling network; uses a graph convolutional network to capture the spatial topological structure of the head joints; and uses the LSTM network in parallel to perform long-term modeling of the feature sequence context information; and obtains the modeling feature vector.

[0086] Furthermore, the multimodal warning feedback includes visual warning feedback, auditory warning feedback and tactile feedback;

[0087] The visual warning feedback is to display a striking visual warning signal on the dashboard or headrest display within the driver's field of vision;

[0088] Auditory warning feedback is played through the vehicle's audio system; the volume and frequency of the sound are adjusted according to the level of distraction;

[0089] The tactile feedback is to use a vibration generator to generate vibration at the driver's position; the frequency and intensity of the vibration are dynamically adjusted according to the distraction level;

[0090] The distraction level is determined according to the distance that the driver's line of sight deviates from the driver's forward field of vision.

[0091] The technical effects and advantages of a DMS driver distraction recognition system of the present invention are as follows:

[0092] The present invention utilizes a three-dimensional head model and a ray tracing algorithm to accurately calculate the driver's forward field of vision, providing an accurate reference field of vision for judging the driver's distracted behavior; the line of sight direction prediction model constructed based on a deep learning algorithm can accurately predict the driver's line of sight direction, and by comparing it with the forward field of vision, efficient detection of the driver's distracted behavior can be achieved; the present invention integrates a variety of advanced algorithms and network structures to greatly improve detection accuracy and robustness; in addition, the system proposes a multi-modal warning feedback mechanism, which can dynamically adjust visual, auditory and tactile alarms according to the severity of the distraction, effectively improving driving safety; it can effectively respond to driving safety hazards and has important practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 A schematic diagram of a DMS driver distraction recognition system of the present invention;

[0094] Figure 2 A schematic diagram of a DMS driver distraction recognition method of the present invention;

[0095] Figure 3 It is a schematic diagram of an electronic device of the present invention. DETAILED DESCRIPTION

[0096] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Example 1

[0097] See also Figure 1 As shown, a DMS driver distraction recognition system described in this embodiment includes:

[0098] A model parameter building module, used to collect the driver's three-dimensional head parameters and build the driver's three-dimensional head model according to the driver's three-dimensional head parameters;

[0099] A visual field calculation module, which is used to calculate and obtain the driver's visual field range facing forward by using a ray tracing algorithm based on the driver's three-dimensional head model;

[0100] A model building module, for building a driver's sight direction prediction model for predicting the driver's sight direction according to a deep learning algorithm;

[0101] The distraction judgment module is used to collect the driver's three-dimensional head posture data; input the driver's three-dimensional head posture data into the driver's line of sight direction prediction model; when the driver's line of sight direction predicted by the driver's line of sight direction prediction model is not within the field of vision directly ahead, the driver is distracted; and multi-modal warning feedback is performed; each module is connected by wire and / or wireless means to realize data transmission between modules.

[0102] Furthermore, the method of collecting the driver's three-dimensional head parameters includes:

[0103] Single IR infrared cameras are installed on the left and right sides of the vehicle terminal respectively. The single IR infrared cameras are aimed at the driver's head to collect basic depth images. The basic depth images contain a three-dimensional point cloud data set of depth information.

[0104] Through the radius search algorithm, feature points are found in the 3D point cloud dataset and the 3D point cloud features of the driver's head area are extracted.

[0105] The method for extracting the three-dimensional point cloud features includes:

[0106] A radius search algorithm based on KD tree (K-dimensional tree) was constructed; a KD tree K of a 3D point cloud dataset was constructed;

[0107] For a query point p, use a backtracking search algorithm to search for all points within a radius r of p on K.

[0108] The process of the backtracking search algorithm includes:

[0109] Initialize the priority queue Q; insert the root node of K into Q; if Q is not empty, pop up the first node m of the priority queue; if ||mp|| is less than or equal to r, then m is a point within the radius r of p; if ||mp|| is greater than r, insert the child node of m into Q; iterate to obtain the initial point cloud feature set;

[0110] Define a 3D point cloud continuous frame registration algorithm based on iterative closest point; use the 3D point cloud continuous frame registration algorithm to process the initial point cloud feature set to obtain a 3D point cloud feature set;

[0111] The three-dimensional point cloud continuous frame registration algorithm is to find a rigid body transformation between two partially overlapping three-dimensional point cloud data sets; the rigid body transformation includes rotation and pan , best align the two point cloud sets together.

[0112] Define the objective function of the 3D point cloud continuous frame registration algorithm ;

[0113] ;in, The rotation matrix representing the rotation transformation; The translation vector representing the translation transformation; Source point cloud The points; Target point cloud The points; is the Euclidean norm, which represents the distance between two points;

[0114] Divide the points in the initial point cloud feature set into source point cloud and target point cloud, and transform them;

[0115] The transformation methods include:

[0116] initialization, , ;for Every point in ,exist Find the distance The nearest point ; seek and , so that the function value of the objective function is minimized;

[0117] Will and Apply to On the above, we get the transformed point cloud ;Will Set as , and repeat the transformation until the preset number of iterations is reached.

[0118] Calculate the registration degree of each pair of points, and select the point pairs with a registration degree greater than or equal to the preset threshold as stable registration points; the points that appear in more than b frames among the stable registration points constitute the 3D point cloud feature set;

[0119] The calculation formula of the registration degree is:

[0120] ;in, for and The degree of alignment; for and The total number of frames contained in the two point clouds; For the The rotation matrix of the frame's rotation transformation; For the The translation vector of the frame's translation transformation;

[0121] A three-dimensional convolutional neural network is constructed, and the three-dimensional point cloud feature set is used as the input of the three-dimensional convolutional neural network, which outputs the three-dimensional head parameters of the driver.

[0122] Furthermore, the method of constructing a three-dimensional convolutional neural network includes:

[0123] Define the basic architecture of a 3D convolutional neural network, which includes an input layer, an encoder, a decoder, and an output layer;

[0124] The encoder includes s adaptive point convolution layers; each adaptive point convolution layer includes 3 adaptive point convolution units; the adaptive point convolution unit performs feature extraction in the local area of ​​each point to obtain a preliminary feature map;

[0125] The decoder is used to process the preliminary feature map to obtain an enhanced feature map.

[0126] The method of obtaining the enhanced feature map by processing includes:

[0127] According to the conditional random field modeling, the preliminary feature map and the clue node map are jointly upsampled to obtain an enhanced feature map;

[0128] Define the conditional random field energy function for conditional random field modeling ;

[0129] ;

[0130] in, For a single node feature First-order potential function, representing the characteristics of a single node potential energy; Node feature and The high-order potential function between them represents the node characteristics and The potential energy between nodes. It measures the node characteristics and The interaction between them reflects the spatial relationship or other association relationship between nodes; and is the node index in the preliminary feature map, indicating different positions in the preliminary feature map;

[0131] The node characteristics and Can be understood as a node and The associated features or variables;

[0132] A clue node graph is obtained according to the preliminary feature graph; the clue node graph uses feature vectors to represent nodes and defines the similarity between nodes; a high-dimensional clue information transmission channel is constructed to strengthen association learning.

[0133] The process of obtaining the clue node graph includes:

[0134] Step 1: Apply the K-average-based vector quantization algorithm to the preliminary feature map to generate a codebook V with a size of 256; use the hash algorithm to generate clue nodes on the codebook V; the features of the clue nodes are 64-dimensional; the K-average vector quantization algorithm maps continuous features to discrete codebooks through vector quantization, which can greatly reduce the amount of data and computational complexity, while retaining the main information and patterns of the original data; it is concise, efficient and theoretically guaranteed;

[0135] Step 2: Build a clue node graph, define the similarity between nodes, and connect nodes whose similarity is greater than a preset similarity threshold;

[0136] The calculation formula of the similarity is: ;in, Represents a clue node The eigenvector of Represents a clue node The eigenvector of for and similarity;

[0137] Repeat steps 1 and 2 to construct an L-layer preliminary clue node graph;

[0138] Finally, all preliminary clue node graphs are fused to form the clue node graph, which serves as the structural prior of the decoder.

[0139] The formula for the joint upsampling is: ;in, To enhance the feature map; It is the clue node diagram; For a given clue node graph Under the condition of The conditional random field energy function of .

[0140] Upsampling Maximize the matching of data items and smooth items while following the clue graph Given the structural constraints; through this optimization, high-quality feature representations that incorporate rich context can be obtained;

[0141] The output layer includes an error regression branch and an analytical geometry branch, which realizes the joint learning of parameter prediction and 3D reconstruction and outputs the driver's 3D head parameters.

[0142] The input of the error regression branch is the enhanced feature map output by the decoder, which is mapped to the regression value through the fully connected layer to predict the coordinates of the key points of the head and the parameters of the skull structure;

[0143] Use the probability model of the Gaussian kernel and define the Gaussian loss function ; Minimize the prediction error of head key point coordinates / skull structure parameters;

[0144] ;in, The real head key point coordinates / skull structure parameters; The predicted head key point coordinates / skull structure parameters; is the standard deviation of the Gaussian kernel;

[0145] The analytic geometry branch also takes as input the enhanced feature map output by the decoder to generate NURBS patch parameters;

[0146] Reconstruct the facial mesh parameters of the driver's head surface based on the NURBS patch parameters; define the projection loss function ;

[0147] in, is the actual facial mesh parameter; are the predicted facial mesh parameters; is the lateral coordinate in the NURBS patch parameter space; is the vertical coordinate in the NURBS patch parameter space;

[0148] The Gaussian loss function and the projection loss function are weightedly fused to obtain the joint loss function. ; is the Gaussian loss weight; is the projection loss weight;

[0149] The driver's three-dimensional head parameters include but are not limited to the coordinates of head key points, facial mesh parameters and skull structure parameters.

[0150] Collect the 3D point cloud feature sets of v drivers' heads as sample sets; divide the sample sets into point cloud training sets and point cloud verification sets; the specific division ratio can be determined according to the actual situation;

[0151] The point cloud training set is divided into H batches, and the three-dimensional convolutional neural network is input into the batches, and the function value of the joint loss function is calculated; if the function values ​​no longer decrease after G consecutive times, the three-dimensional convolutional neural network is saved at this time, and the effect of the three-dimensional convolutional neural network is verified using the point cloud verification set; the three-dimensional convolutional neural network with the best effect is saved as the final three-dimensional convolutional neural network.

[0152] Furthermore, the method of establishing the driver's three-dimensional head model includes:

[0153] Using the skull structure parameters, extract the basic skeleton framework of the driver's head; the basic skeleton framework includes the size, angle and relative position of each part of the skull;

[0154] According to the coordinates of the key points of the head, the key facial feature points are located on the basic skeleton frame; the key facial feature points include but are not limited to the center of the eyebrows, the corners of the eyes, the tip of the nose and the corners of the mouth; the key facial feature points reflect the typical features of the driver's face;

[0155] Based on the basic skeleton framework and referring to the facial mesh parameters, the triangular patch mesh of the driver's head is constructed using 3D modeling software (such as Maya, 3dMax, etc.); the triangular patch mesh constructs the 3D surface morphology of the driver's head through the combination of the basic elements of vertices, edges and faces;

[0156] The generated 3D surface morphology is subdivided into four rows to increase the mesh density reasonably and improve the expressiveness of facial details. At the same time, the topological structure of the mesh is optimized to ensure the best balance between fidelity and computational efficiency. The key facial feature points are used as control handles to locally deform and adjust the triangular patch mesh. This completes the construction of the driver's 3D head model.

[0157] Furthermore, the method of obtaining the driver's forward field of vision includes:

[0158] Importing the driver's three-dimensional head model and the in-car environment object model into a virtual three-dimensional scene; the virtual three-dimensional scene is a virtual reality representation of a pre-established car driving interior scene;

[0159] According to the driver's 3D head model, the driver's binocular viewpoint is determined as , the vertical field of view angle is , the horizontal field of view angle is ;

[0160] Then the cone equation of the driver's eye's front view cone is obtained as follows: ;in, are the coordinates of any point within the normal viewing frustum.

[0161] Because the influence of lighting must be taken into account; calculate the intersection points between light and objects in the car driving interior scene;

[0162] According to the light equation ;in, is the driver's binocular viewpoint, is the light direction vector, is the ray parameter;

[0163] For any one An object composed of triangles, if the Triangles By three vertices , , Construct, obtain The unit normal vector ;

[0164] Then the light and Intersection Point The calculation formula is: ;

[0165] For each intersection point , check whether there is an occluding object, from the viewpoint Send shadow rays to , if the ray does not intersect any other object, then is a visible point; record all visible points;

[0166] Check whether the recorded visible point is within the emmetropia cone; if so, record the point as the emmetropia point;

[0167] The three-dimensional coordinates of all the front view points are projected onto a two-dimensional plane to obtain a point set on the two-dimensional plane, which is the driver's front view range.

[0168] It should be noted that in computer graphics, shadow rays are a technique for calculating shadows; specifically, when determining whether a point is visible (not blocked by other objects), we need to emit a ray from the point toward the viewpoint (camera or observer position) to check whether the ray intersects with any object in the scene; if not, the point is visible; and the role of the shadow ray is to check whether the visible point is illuminated by direct light from the light source, that is, whether it is in the shadow; this requires emitting a shadow ray from the point toward the light source to check whether the ray intersects with any object in the scene; if so, the point is in the shadow; otherwise, it is a point of direct visible light.

[0169] Furthermore, the driver's sight direction prediction model is established in the following manner:

[0170] Collect S groups of historical drivers' three-dimensional head posture data and in-vehicle environment data as initial data; the driver's three-dimensional head posture data includes head joint position and head joint displacement; the in-vehicle environment data includes vehicle speed and steering wheel angle; the initial data is marked with the driver's line of sight direction; the initial data marked with the driver's line of sight direction is used as training data.

[0171] A network framework of a driver's gaze direction prediction model is defined; the network framework includes a multimodal data encoding layer, a modal interaction attention layer, a spatiotemporal modeling layer, and a gaze direction prediction layer.

[0172] The multimodal data encoding layer encodes the three-dimensional head posture data and the in-vehicle environment data respectively to obtain feature vector representations of the three-dimensional head posture data and the in-vehicle environment data;

[0173] The modal interaction attention layer is used to calculate the correlation score matrix between data; and use the correlation score matrix to calculate the weighted sum vector; and the weighted sum vector is input to the GRU unit to perform feature selection and adaptive weight adjustment; and finally output the weighted fusion feature vector corresponding to each data;

[0174] For any 3D head posture data and in-car environment data, calculate the correlation score ;

[0175] ; In the formula, is a learnable weight matrix; For data The transposed vector of the vector representation of ; For data The vector representation of the data is the head joint position or head joint displacement; data is vehicle speed or steering wheel angle; and They can all be replaced by numerical values; for example, 1 is the head joint position, 2 is the head joint displacement;

[0176] The correlation scores of the three-dimensional head posture data and the in-vehicle environment data are combined into a correlation score matrix;

[0177] For each data vector representation, calculate its weighted sum vector according to its correlation score with other data ; In the formula, For data The weighted sum vector of ; is the number of all data; for and data The attention weight represented by the vector of ; it is calculated by normalizing the relevance scores in the relevance matrix using the softmax function to ensure that the sum of the attention weights is 1;

[0178] ;in, is an index for summing; For data and data Correlation score between data Different from data ;

[0179] The spatiotemporal modeling layer combines the weighted fusion feature vectors according to the corresponding data collection time to form a feature sequence; the feature sequence is input into the spatiotemporal modeling network; a graph convolutional network is used to capture the spatial topological structure of the head joints; and an LSTM network is used in parallel to perform long-term modeling of the feature sequence context information; and a modeling feature vector is obtained;

[0180] The gaze direction prediction layer inputs the modeling feature vector into the fully connected layer and outputs a three-dimensional gaze direction vector; the three-dimensional gaze direction vector is normalized by L2 norm to obtain a unit gaze direction vector; based on the frustum target detection framework, the gaze direction is discretized into predetermined categories; the predetermined categories include but are not limited to the front field of view category, the left field of view category, the right field of view category, and the distracted field of view category.

[0181] 3D sight direction vector ; In the formula, is the weight matrix of the fully connected layer; is the modeling feature vector; is the bias term;

[0182] The three-dimensional sight line direction vector represents the unit direction vector of the driver's sight line in three-dimensional space.

[0183] The unit sight direction vector is used as the center sight of the frustum, and the angle range is set as the frustum volume. The angle range includes horizontal angle and vertical angle.

[0184] For each predetermined category, determine a corresponding three-dimensional space cone area; calculate the unit sight direction vector The angular distance with each three-dimensional space cone area; the unit sight direction vector is attributed to the three-dimensional space cone area corresponding to the predetermined category with the minimum angular distance value.

[0185] The calculation formula of the angle distance is: ; In the formula, The cone area in three-dimensional space The central sight unit vector of is the unit sight direction vector Cone area in three-dimensional space Angular distance;

[0186] Define the loss function of the driver's line of sight direction prediction model; for example, the cross entropy loss function; divide the training data into a distraction training set and a distraction validation set; the specific division ratio can be determined according to the actual situation; for example, 80% distraction training set and 20% distraction validation set;

[0187] The distraction training set is divided into X1 batches, and the driver's line of sight direction prediction model is input into the batches. The function value of the loss function is recorded every time a batch is input; and an optimizer is added to optimize the parameters of the driver's line of sight direction prediction model; until the function value of the loss function does not decrease in X2 consecutive batches, the driver's line of sight direction prediction model corresponding to this time and the previous X3 batches is saved; the accuracy of the X3 driver's line of sight direction prediction models is verified using the distraction verification set; the driver's line of sight direction prediction model with the best accuracy is used as the driver's line of sight direction prediction model that has been finally trained.

[0188] It should be noted that the spatiotemporal modeling network is designed to capture the dependencies and patterns of data in time and space at the same time. In the driver's line of sight prediction task, the temporal model such as LSTM can capture the dynamic changes of the driver's head posture over time; and the spatial model such as 3D convolutional network can extract the spatial structural features of a single frame of head point cloud. Through the spatiotemporal modeling network, the temporal dynamic and spatial structural information contained in the head posture data can be fully mined, providing a richer and more accurate feature representation for line of sight prediction.

[0189] The graph convolutional network is a new convolutional neural network architecture designed for non-Euclidean geometric data (such as graphs, grids, etc.); in the line of sight prediction task, the driver's head joints can be regarded as nodes in the graph structure, and the connections between the joints are edges, forming a topological graph; the graph convolutional network can learn to propagate and update node features along the graph structure, and capture local neighborhood patterns through convolution operations; it naturally processes non-Euclidean geometric data without projecting it into Euclidean space, avoiding the defects of manual feature design.

[0190] Furthermore, the multimodal warning feedback includes visual warning feedback, auditory warning feedback and tactile feedback;

[0191] The visual warning feedback is to display a striking visual warning signal on the dashboard or headrest display within the driver's field of vision, such as a red warning sign, text prompt, etc.;

[0192] The auditory warning feedback is to play a sound alarm, such as a ringtone, voice prompt, etc., through the vehicle audio system; the volume and frequency of the sound are adjusted according to the severity of the distraction; the auditory alarm can be played synchronously with the visual alarm to produce multi-sensory stimulation;

[0193] The tactile feedback is to use a vibration generator to generate vibration at the driver's location (such as a seat, a steering wheel); the frequency and intensity of the vibration are dynamically adjusted according to the distraction level;

[0194] The distraction level is determined based on the distance that the driver's line of sight deviates from the driver's forward field of vision; a smaller distance results in a smaller distraction level; a larger distance results in a larger distraction level; for example, if the distance is 1, the distraction level is determined to be 1; if the distance is 3, the distraction level is determined to be 2; and so on.

[0195] This embodiment uses a three-dimensional head model and a ray tracing algorithm to accurately calculate the driver's forward field of vision, providing an accurate reference field of vision for judging the driver's distracted behavior; the line of sight direction prediction model constructed based on the deep learning algorithm can accurately predict the driver's line of sight direction, and by comparing it with the forward field of vision, it can achieve efficient detection of the driver's distracted behavior; the system integrates a variety of advanced algorithms and network structures, greatly improving the detection accuracy and robustness; in addition, the system proposes a multi-modal warning feedback mechanism, which can dynamically adjust the visual, auditory and tactile alarms according to the severity of the distraction degree, thereby improving driving safety; it can effectively deal with driving safety hazards and has important practical application value. Example 2

[0196] See also Figure 2As shown, the part not described in detail in this embodiment is described in the description of embodiment 1, and a DMS driver distraction recognition method is provided, comprising: S1, collecting the driver's three-dimensional head parameters, and establishing the driver's three-dimensional head model according to the driver's three-dimensional head parameters;

[0197] S2. Calculate and obtain the driver's forward field of vision using a ray tracing algorithm based on the driver's three-dimensional head model;

[0198] S3. constructing a driver's sight direction prediction model for predicting the driver's sight direction according to a deep learning algorithm;

[0199] S4, collecting the driver's three-dimensional head posture data; inputting the driver's three-dimensional head posture data into the driver's line of sight direction prediction model; when the driver's line of sight direction predicted by the driver's line of sight direction prediction model is not within the forward field of vision, the driver is distracted; and performing multi-modal warning feedback. Example 3

[0200] See also Figure 3 As shown, this embodiment discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the operation mode of the DMS driver distraction recognition method provided above is implemented.

[0201] Since the electronic device introduced in this embodiment is an electronic device used to implement a DMS driver distraction identification method in the embodiment of the present application, based on the DMS driver distraction identification method introduced in the embodiment of the present application, a person skilled in the art can understand the specific implementation of the electronic device of the present embodiment and its various variations, so how the electronic device implements the method in the embodiment of the present application is not described in detail here. As long as a person skilled in the art implements the electronic device used in the DMS driver distraction identification method in the embodiment of the present application, it belongs to the scope of protection of the present application.

[0202] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and thresholds in the formula are set by technicians in this field according to actual conditions.

[0203] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technical users in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A DMS driver distraction recognition system, characterized in that: include: A model parameter building module, used to collect the driver's three-dimensional head parameters and build the driver's three-dimensional head model according to the driver's three-dimensional head parameters; A visual field calculation module, which is used to calculate and obtain the driver's visual field range facing forward by using a ray tracing algorithm based on the driver's three-dimensional head model; A model building module, for building a driver's sight direction prediction model for predicting the driver's sight direction according to a deep learning algorithm; Distraction judgment module, used to collect the driver's three-dimensional head posture data; The driver's three-dimensional head posture data is input into a driver's sight direction prediction model; when the driver's sight direction predicted by the driver's sight direction prediction model is not within the forward visual field, the driver is distracted; And provide multi-modal warning feedback; each module is connected by wired and / or wireless means to achieve data transmission between modules; The method of collecting the driver's three-dimensional head parameters includes: Single IR infrared cameras are installed on the left and right sides of the vehicle terminal respectively. The single IR infrared cameras are aimed at the driver's head to collect basic depth images. The basic depth images contain a three-dimensional point cloud data set of depth information. Through the radius search algorithm, feature points are found in the 3D point cloud data set, and the 3D point cloud feature set of the driver's head area is extracted; The method of extracting the three-dimensional point cloud feature set includes: A radius search algorithm based on KD tree is constructed; a KD tree K of a 3D point cloud dataset is constructed; for a query point p; a backtracking search algorithm is used to search for all points within a radius r of p on K; The process of the backtracking search algorithm includes: Initialize the priority queue Q; insert the root node of K into Q; if Q is not empty, pop up the first node m of the priority queue; if ||mp|| is less than or equal to r, then m is a point within the radius r of p; if ||mp|| is greater than r, insert the child node of m into Q; iterate to obtain the initial point cloud feature set; Define a 3D point cloud continuous frame registration algorithm based on iterative closest point; use the 3D point cloud continuous frame registration algorithm to process the initial point cloud feature set to obtain a 3D point cloud feature set; A three-dimensional convolutional neural network is constructed, and a three-dimensional point cloud feature set is used as the input of the three-dimensional convolutional neural network, and the three-dimensional convolutional neural network outputs the driver's three-dimensional head parameters; the driver's three-dimensional head parameters include head key point coordinates, facial mesh parameters and skull structure parameters.

2. A DMS driver distraction recognition system according to claim 1, characterized in that: The method of using the 3D point cloud continuous frame registration algorithm to process the initial point cloud feature set to obtain the 3D point cloud feature set includes: Define the objective function Lm of the 3D point cloud continuous frame registration algorithm; Where R represents the rotation matrix of the rotation transformation; t represents the translation vector of the translation transformation; P Ai is the source point cloud P A The i-th point in P Bi The target point cloud P B The i-th point in ; || ||2 is the Euclidean norm; Divide the points in the initial point cloud feature set into source point cloud and target point cloud, and transform them; The transformation methods include: Initialization, R = I, t = 0; for P B Every point P in Bi , in P A Find the distance P Bi The nearest point P Ai ; Find R and t to minimize the function value of the objective function; Apply R and t to P A On the top, we get the transformed point cloud P A ';P A 'Set P A , and repeat the transformation until the preset number of iterations is reached; Calculate the registration degree of each pair of points, and select the point pairs with a registration degree greater than or equal to the preset threshold as stable registration points; the points that appear in more than b frames among the stable registration points constitute the 3D point cloud feature set; The calculation formula of the registration degree is: Among them, H(P Ai ,P Bi ) is P Ai and P Bi The degree of registration of P A and P B The total number of frames contained in the two point clouds; R f is the rotation matrix of the rotation transformation of the fth frame; t f is the translation vector of the translation transformation of the f-th frame.

3. A DMS driver distraction recognition system according to claim 2, characterized in that: The method of constructing a three-dimensional convolutional neural network includes: Define the basic architecture of a 3D convolutional neural network, which includes an input layer, an encoder, a decoder, and an output layer; The encoder includes s adaptive point convolution layers; each adaptive point convolution layer includes 3 adaptive point convolution units; the adaptive point convolution unit performs feature extraction for each point to obtain a preliminary feature map; and a clue node map is obtained according to the preliminary feature map; The decoder is used to process the preliminary feature map to obtain an enhanced feature map; The output layer includes an error regression branch and an analytical geometry branch, which are used to output the three-dimensional head parameters of the driver; The input of the error regression branch is the enhanced feature map output by the decoder, which is mapped to the regression value through the fully connected layer to predict the coordinates of the key points of the head and the parameters of the skull structure; The analytical geometry branch also takes the enhanced feature map output by the decoder as input, generates NURBS patch parameters, and reconstructs the facial mesh parameters of the driver's head surface according to the NURBS patch parameters; Define the Gaussian loss function L reg And the projection loss function L recon ; The Gaussian loss function and the projection loss function are weightedly fused to obtain the joint loss function L lan =λ1·L reg +λ2·L recon ; Among them, λ1 is the Gaussian loss weight; λ2 is the projection loss weight; Collect the 3D point cloud feature sets of v drivers’ heads as sample sets; divide the sample sets into point cloud training sets and point cloud verification sets; The point cloud training set is divided into H batches, and the three-dimensional convolutional neural network is input into the batches, and the function value of the joint loss function is calculated; if the function values ​​no longer decrease after G consecutive times, the three-dimensional convolutional neural network is saved at this time, and the effect of the three-dimensional convolutional neural network is verified using the point cloud verification set; the three-dimensional convolutional neural network with the best effect is saved as the final three-dimensional convolutional neural network.

4. A DMS driver distraction recognition system according to claim 3, characterized in that: The method of obtaining the enhanced feature map by processing includes: According to the conditional random field modeling, the preliminary feature map and the clue node map are jointly upsampled to obtain an enhanced feature map; Define the conditional random field energy function E(X) for conditional random field modeling; E(X)=∑ j ψ u (X j )+∑ j＜k ψ p (X j ,X k ); Among them, ψ u (X j ) is a single node feature X j First-order potential function; ψ p (X j ,X k ) is the node feature X j and X k The high-order potential function between them; j and k are the node indices in the preliminary feature graph; The method of obtaining the clue node graph according to the preliminary feature graph includes: Step 1: Apply the K-means-based vector quantization algorithm to the preliminary feature map to generate a codebook V with a size of 256; use the hash algorithm on the codebook V to generate clue nodes; Step 2: Build a clue node graph, define the similarity between nodes, and connect nodes whose similarity is greater than a preset similarity threshold; The calculation formula of the similarity is: Among them, I r Represents the feature vector of clue node r; I e Represents the feature vector of clue node e; Sim(I r ,I e ) is I r and I e similarity; Repeat steps 1 and 2 to construct L-layer preliminary clue node graphs; merge all preliminary clue node graphs to form a clue node graph; The formula for the joint upsampling is: X'=argmax E(X,U); wherein X' is the enhanced feature map; U is the clue node map; E(X,U) is the conditional random field energy function of the initial feature map X under the condition of the given clue node map U; The Gaussian loss function: Among them, yk is the real head key point coordinates / skull structure parameters; yk' is the predicted head key point coordinates / skull structure parameters; σ is the standard deviation of the Gaussian kernel; The projection loss function L recon =∑||Pl(q,d)-Pl'(q,d)|| 2 ; Among them, Pl(q,d) is the real facial mesh parameter; Pl'(q,d) is the predicted facial mesh parameter; q is the horizontal coordinate in the space of NURBS patch parameters; d is the vertical coordinate in the space of NURBS patch parameters.

5. A DMS driver distraction recognition system according to claim 4, characterized in that: The method for establishing the driver's three-dimensional head model includes: Using the skull structure parameters, extract the basic skeleton framework of the driver's head; the basic skeleton framework includes the size, angle and relative position of each part of the skull; According to the coordinates of the key points of the head, the key facial feature points are located on the basic skeleton frame; the key facial feature points include the center of the eyebrows, the corners of the eyes, the tip of the nose and the corners of the mouth; the key facial feature points reflect the typical features of the driver's face; Based on the basic skeleton framework and referring to the facial mesh parameters, the triangular mesh of the driver's head is constructed using 3D modeling software. The triangular mesh constructs the 3D surface morphology of the driver's head through the combination of the basic elements of vertices, edges and faces. The generated three-dimensional surface morphology is subdivided, and the key facial feature points are used as control handles to locally deform and adjust the triangular face mesh; thus, the construction of the driver's three-dimensional head model is completed.

6. A DMS driver distraction recognition system according to claim 5, characterized in that: The method of obtaining the driver's forward field of vision includes: Importing the driver's three-dimensional head model and the in-car environment object model into a virtual three-dimensional scene; the virtual three-dimensional scene is a virtual reality representation of a pre-established car driving interior scene; According to the three-dimensional head model of the driver, the binocular viewpoint of the driver is determined as PO, the vertical field of view angle is αV, and the horizontal field of view angle is αH; Then the cone equation of the driver's binocular frontal field of view is obtained as: Among them, x', y', z' are the coordinates of any point in the frustum of the normal field of view; Calculate the intersection points of the light and each object in the interior scene of the car driving; the calculation method of the intersection points includes: According to the light equation P'=PO+T'·D'; where PO is the driver's binocular viewpoint, D' is the light direction vector, and T' is the ray parameter; For any object composed of N triangles, if the g-th triangle Tg is composed of three vertices Vg1, Vg2, and Vg3, obtain the unit normal vector Ng of Tg; Then the calculation formula for the point Pg where the light intersects with Tg is: For each intersection point Pg, check whether there is an occluding object, emit a shadow ray from the viewpoint PO to Pg, if the ray does not intersect with any other object, then Pg is a visible point; record all visible points; Check whether the recorded visible point is within the emmetropia cone; if so, record the point as the emmetropia point; The three-dimensional coordinates of all the front view points are projected onto a two-dimensional plane to obtain a point set on the two-dimensional plane, which is the driver's front view range.

7. A DMS driver distraction recognition system according to claim 6, characterized in that: The method for establishing the driver's sight direction prediction model includes: Collecting S groups of historical three-dimensional head posture data and in-vehicle environment data of the driver as initial data; the three-dimensional head posture data of the driver includes the head joint position and the head joint displacement; the in-vehicle environment data includes the vehicle speed and the steering wheel angle; annotating the initial data with the driver's line of sight direction; using the initial data annotated with the driver's line of sight direction as training data; Define the network framework of the driver's line of sight direction prediction model; the network framework includes a multimodal data encoding layer, a modal interaction attention layer, a spatiotemporal modeling layer, and a line of sight direction prediction layer; The sight direction prediction layer inputs the modeling feature vector into the fully connected layer and outputs a three-dimensional sight direction vector; performs L2 norm normalization on the three-dimensional sight direction vector to obtain a unit sight direction vector; discretizes the driver's sight direction into predetermined categories; the predetermined categories include forward vision category, left vision category, right vision category, and distracted vision category; Three-dimensional sight direction vector Z' = W_fc·H" + b_fc; where W_fc is the weight matrix of the fully connected layer; H" is the modeling feature vector; b_fc is the bias term; The unit sight direction vector is used as the center sight of the frustum, and the angle range is set as the frustum volume. The angle range includes horizontal angle and vertical angle. For each predetermined category, a corresponding three-dimensional space cone area is determined; the angle distance between the unit sight direction vector V' and each three-dimensional space cone area is calculated; the unit sight direction vector is attributed to the three-dimensional space cone area corresponding to the predetermined category with the smallest angle distance value; The calculation formula of the angle distance is: D χ =arccos(V'·N' χ );where N' χ is the central sight line unit vector of the three-dimensional cone area χ; D χ is the angle distance between the unit sight line direction vector V' and the three-dimensional space cone area χ; Define the loss function of the driver's line of sight direction prediction model; divide the training data into a distraction training set and a distraction validation set; The distraction training set is divided into X1 batches, and the driver's line of sight direction prediction model is input into the batches. The function value of the loss function is recorded every time a batch is input; and an optimizer is added to optimize the parameters of the driver's line of sight direction prediction model; until the function value of the loss function does not decrease in X2 consecutive batches, the driver's line of sight direction prediction model corresponding to this time and the previous X3 batches is saved; the accuracy of the X3 driver's line of sight direction prediction models is verified using the distraction verification set; the driver's line of sight direction prediction model with the best accuracy is used as the driver's line of sight direction prediction model that has been finally trained.

8. The DMS driver distraction recognition system according to claim 7, characterized in that: The multimodal data encoding layer encodes the three-dimensional head posture data and the in-vehicle environment data respectively to obtain feature vector representations of the three-dimensional head posture data and the in-vehicle environment data; The modal interaction attention layer is used to calculate the correlation score matrix between data; and use the correlation score matrix to calculate the weighted sum vector; and input the weighted sum vector into the GRU unit to perform feature selection and adaptive weight adjustment; finally, the weighted fusion feature vector corresponding to each data is output; For any 3D head posture data and in-vehicle environment data, calculate the correlation score R β,ω ; R β,ω =f β T ·W'·f ω ;W' is the learnable weight matrix; f β T is the transposed vector of the vector representation of data β; f ω is the vector representation of the data ω; The correlation scores of the three-dimensional head posture data and the in-vehicle environment data are combined into a correlation score matrix; For each data vector representation, calculate its weighted sum vector according to its correlation score with other data In the formula, g' β is the weighted sum vector of data β; M is the total number of data; α βω f ω and the attention weights represented by the vector of data β; Where k' is an index for summation; R β,k' is the correlation score between data β and data k'; The spatiotemporal modeling layer combines the weighted fused feature vectors into a feature sequence according to the corresponding data collection time; inputs the feature sequence into the spatiotemporal modeling network; uses a graph convolutional network to capture the spatial topological structure of the head joints; and uses the LSTM network in parallel to perform long-term modeling of the feature sequence context information; and obtains the modeling feature vector.

9. The DMS driver distraction recognition system according to claim 8, characterized in that: The multimodal warning feedback includes visual warning feedback, auditory warning feedback and tactile feedback; The visual warning feedback is to display a striking visual warning signal on the dashboard or headrest display within the driver's field of vision; Auditory warning feedback is played through the vehicle's audio system; the volume and frequency of the sound are adjusted according to the level of distraction; The tactile feedback is to use a vibration generator to generate vibration at the driver's position; the frequency and intensity of the vibration are dynamically adjusted according to the distraction level; The distraction level is determined according to the distance that the driver's line of sight deviates from the driver's forward field of vision.

Images