A driver state monitoring method based on a three-dimensional morphable model

Abstract

The application discloses a driver state monitoring method based on a three-dimensional deformation model, which monitors the driver state from multiple angles through processing three modes of physiological information, head posture estimation and 3D gaze estimation; when it is judged that the driver is in an abnormal state, a warning reminder can be timely sent to prevent the driver from performing dangerous behaviors, and relevant traffic management personnel can be notified when necessary; the abnormal state includes a fatigue state, a road rage state, a distraction state and an abnormal state; the application fuses multiple monitoring modes and uses multiple computer vision and deep learning technologies, improves the accuracy and reliability of the driver state monitoring, and can be widely applied to scenes requiring long-time driving.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of driver state monitoring, and particularly relates to a driver state monitoring method based on a three-dimensional deformation model. BACKGROUND

[0002] With the rapid growth of the number of cars and the popularity of transportation tools, road traffic safety has become one of the important problems in the world. A large number of traffic accidents occur every year, many of which are caused by poor driver state, such as fatigue, distraction, road rage, and illegal driving. In order to improve traffic safety, driver state monitoring technology has become a popular research direction.

[0003] Driver state monitoring technology is a technology that collects and analyzes physiological information, behavioral characteristics and other data of drivers to determine whether the driver is in a normal state. At present, some technologies for monitoring the state of drivers have appeared, such as fatigue monitoring based on physiological signals, distraction state monitoring based on image processing, and abnormal driving behavior detection based on behavioral characteristics.

[0004] Among them, fatigue monitoring based on physiological signals is the most widely used technology. This technology collects physiological signals of drivers, such as heart rate and brain waves, to determine whether the driver is in a state of fatigue. The advantage of this technology is high accuracy, but it requires contact sensors and is easily disturbed by the external environment, and may also interfere with the driver's driving operation.

[0005] In recent years, with the development of artificial intelligence, computer vision and other technologies, more and more new driver state monitoring technologies have been widely applied. For example, driver state monitoring technology based on deep learning can collect image data of drivers and use neural network models to determine the state of drivers. This technology has high accuracy, but requires a large amount of data and computing power support.

[0006] In addition to the above technologies, some other technologies are also used for driver state monitoring. For example, abnormal driving behavior detection technology based on behavioral characteristics uses vehicle sensors to detect the driving behavior of drivers and uses specific algorithms to determine whether the driver is in an abnormal state. This technology has high real-time performance and accuracy, but it needs appropriate sensors and algorithm support.

[0007] In general, driver state monitoring technology is a very important technology that can effectively improve the level of traffic safety, so it is necessary to study a method that can accurately monitor the state of drivers to reduce the probability of accidents. SUMMARY

[0008] The application aims to provide a driver state monitoring method based on a three-dimensional morphable model to solve the above technical problems.

[0009] The driver state monitoring method based on a three-dimensional morphable model comprises the following steps:

[0010] S1: data collection: collecting driver physiological information and driver operation information; the driver physiological information includes body temperature, heart rate, blood pressure, blood oxygen saturation, respiratory rate, voice, eye position, pupil size, blink frequency, retinal center, fixation point and eye movement information; the driver operation information includes driving video, steering wheel rotation angle, brake pedal force and accelerator pedal force.

[0011] S2: data processing:

[0012] S2.1: processing and analyzing driver physiological information;

[0013] S2.2: head posture estimation, estimating the posture of the driver's head through a region proposal network, a face alignment network and a three-dimensional morphable model;

[0014] S2.3: 3D gaze estimation, estimating the gaze direction of the driver through computer vision technology.

[0015] S3: state judgment: judging whether the driver is in an abnormal state according to the physiological information analysis result, head posture estimation and 3D gaze estimation, respectively.

[0016] Taking the union of the abnormal state information judged by the above three methods.

[0017] When the judgment result is that the driver's state is abnormal, timely warning and reminding the driver to drive safely, and notifying relevant traffic management personnel if necessary.

[0018] The abnormal state includes a fatigue state, a road rage state, a distraction state and an abnormal state.

[0019] The fatigue state is a state in which the driver appears obvious fatigue, including a decrease in eye opening, an increase in blink frequency, a decrease in fixation time, a deviation of fixation point and a decrease in eye movement frequency.

[0020] The road rage state is a state in which the driver appears obvious emotional excitement and impulsive behavior, including a significant increase in heart rate or blood pressure, or showing angry behavior, including roaring, shouting, making angry sounds, glaring at other drivers, frequently honking, frequently braking and hard acceleration;

[0021] The distraction state is a state in which the driver's attention is distracted during driving, which may have a potential impact on driving safety.

[0022] The abnormal state is an abnormal state of the driver during driving, including physiological parameter abnormality and behavior abnormality; the physiological parameter abnormality includes that the heart rate, breathing rate, blood pressure or blood oxygen of the driver exceeds the normal range; the behavior abnormality includes that the driver frequently changes lanes, drives on the solid line, runs a red light, excessively accelerates, frequently and sharply brakes, leaves the main driving position, changes the driver, intentionally interferes with the work of the monitoring device or the sensor.

[0023] S4: data storage and analysis:

[0024] S4.1: according to the characteristics of the collected data and the application requirements, a suitable data model and database structure are designed;

[0025] S4.2: the processed data are stored in the database, including the physiological information and operation information of the driver, and the results of head posture estimation and 3D gaze estimation;

[0026] S4.3: clean the data, remove abnormal data and noise;

[0027] S4.4: process the data, including data preprocessing, feature extraction, feature selection and dimension reduction;

[0028] The data preprocessing includes data deduplication, missing value processing, abnormal value processing and data normalization;

[0029] S4.5: according to the data obtained by data preprocessing, feature extraction, feature selection and dimension reduction, use data mining and statistical analysis techniques to analyze and model the state of the driver, including clustering analysis, association rule mining, classification and prediction;

[0030] S4.5.1: use K-means algorithm for clustering analysis;

[0031] S4.5.2: use Apriori algorithm for association rule mining, the steps are as follows:

[0032] (1) define the threshold of minimum support and minimum confidence;

[0033] (2) scan the data set to generate candidate 1-item set;

[0034] (3) remove the item set in the candidate 1-item set whose support is less than the minimum support;

[0035] (4) generate candidate 2-item set according to the remaining item set;

[0036] (5) remove the item set in the candidate 2-item set whose support is less than the minimum support;

[0037] (6) generate candidate 3-item set according to the remaining item set;

[0038] (7) remove the item sets with support less than the minimum support from the candidate 3-item sets;

[0039] (8) in this way, until no candidate item set can be generated;

[0040] (9) generate association rules according to the frequent item sets;

[0041] (10) remove the association rules with confidence less than the minimum confidence;

[0042] The support calculation formula is:

[0043]

[0044] Wherein, count(X) represents the number of times of the item set X appearing in the data set, and N represents the total number of transactions in the data set;

[0045] The confidence calculation formula is:

[0046]

[0047] Wherein, X represents the premise item set, and Y represents the conclusion item set;

[0048] Generate candidate k-item sets according to the frequent k-1-item sets, wherein k is greater than or equal to 2;

[0049] The candidate item set generation formula is:

[0050] C k = U i k-1 j | i k-1 ∈ L

[0051] , | i k ∪ j | = k, i k-1 < j

[0052] S4.5.3: classification using decision tree algorithm;

[0053] S4.5.4: data prediction: input the real-time collected driver physiological information and driver operation information into the trained classification model during the driving of the vehicle, to obtain the corresponding classification result.

[0054] S4.6: according to the analysis result, formulate corresponding early warning strategy and action plan, so as to timely remind the driver to drive safely, and notify the relevant traffic management personnel if necessary.

[0055] The head pose estimation estimates the pose of the driver's head in three-dimensional space through computer vision technology, and the specific steps are as follows:

[0056] S1: Face detection: Use a face detection algorithm based on convolutional neural network to detect faces in the image and output a rectangular box to represent the position and size of the detected face in the image:

[0057] S1.1: Image preprocessing: Preprocess the input image, including size normalization and color space conversion;

[0058] S1.2: Extract feature map: Extract features from the preprocessed image through a convolutional neural network to obtain a set of feature maps;

[0059] S1.3: Generate candidate boxes using region proposal network: Generate multiple candidate boxes on the feature map using sliding window method, each candidate box i has a feature vector x i ;

[0060] S1.4: Classify and regress each candidate box: Classify and regress each candidate box through a classifier and a regressor to determine whether the region contains a face and the position and size of the face box:

[0061] The parameters of the classifier and regressor are W c ,W x ,W y ,W w ,W h and b c ,b x ,b y ,b w ,b h , the probability of whether the region contains a face and the left top horizontal and vertical coordinates of the face box and the width and height can be calculated as follows:

[0062]

[0063]

[0064]

[0065]

[0066]

[0067] where σ represents the Sigmoid function;

[0068] S1.5: Non-maximum suppression: Remove overlapping face boxes through a non-maximum suppression algorithm to obtain the final face box:

[0069] (x min ,ymin ,x max ,y max )=(xw / 2,yh / 2,x+w / 2,y+h / 2)

[0070] Where x, y are the coordinates of the center point of the prediction box; w, h are the width and height of the prediction box; (x min ,y min ) and (x max ,y max ) represent the coordinates of the top left and bottom right corners of the face bounding box;

[0071] S2: Feature point detection: Based on the face image detected in the previous step, a face alignment network is used to detect the feature points on the face;

[0072] y i =F(x,θ) i )

[0073] Where x is the input face image, F is the face alignment network model, and y i Let θ be the position of the i-th keypoint in the feature map. i Let be the network parameters for the i-th key point;

[0074] Supervised learning is used to train the face alignment network model, which optimizes the model parameters by minimizing the difference between the predicted and the true values.

[0075] The result of feature point detection can be represented as a two-dimensional coordinate array, i.e.:

[0076] P = (x1, y1), (x2, y2), ..., (x n ,y n )

[0077] Where n is the number of key points, (x i ,y i Let be the position of the i-th keypoint in the image;

[0078] S3: Pose Estimation: A 3D deformable model is used to represent the face shape and texture information. Then, the detected feature points are mapped onto the 3D deformable model to calculate the head pose in 3D space.

[0079] S3.1: Convert 2D feature point coordinates to 3D spatial coordinates: For each estimated feature point p i =(x i ,y i The depth value d of the corresponding point in the 3D deformation model can be used. i Convert it to 3D spatial coordinates (X i ,Y i Zi ):

[0080]

[0081] where c x and c y are the image center point coordinates, and f is the camera focal length;

[0082] S3.2: Registration of the estimated 3D coordinates with the control point coordinates in the 3D morphable model: the estimated 3D coordinates are registered with the control point coordinates in the 3D morphable model by the iterative closest point algorithm to obtain the registered 3D coordinates;

[0083] S3.3: Obtaining the pose parameters in the 3D morphable model by linear regression: for the registered 3D coordinates and the corresponding control point coordinates in the 3D morphable model, the pose parameters θ in the 3D morphable model are obtained by linear regression; that is, a linear equation system Aθ = b is constructed, where A is a 3n x 6 matrix, n is the number of control points, and b is a 3n-dimensional vector representing the registered 3D coordinates; θ is solved by least squares method to obtain the pose parameters of the head in the 3D morphable model.

[0084] The specific steps of the 3D gaze estimation are as follows:

[0085] S1: rendering the known 2D human eye center point position and the shape and texture of the 3D morphable model to obtain the corresponding 2D image of the 3D head model:

[0086] I 2D = R(s, R yaw , R pitch , R roll )MC

[0087] where I 2D is the rendered 2D image, R is the projection transformation matrix, s is the scale parameter, R yaw , R pitch , and R roll are the head pose parameters, M is the 3D morphable model, and C is the texture color of the 3D morphable model;

[0088] S2: detecting the human eye center point position in the 2D image and calculating the pixel coordinates (x c , y c ) of the human eye center point in the image;

[0089] S3: calculating the gaze direction vector v according to the human eye center point position and the rotation angle of the head pose:

[0090] v = R yaw R pitch v0

[0091] wherein v0 is a reference gaze direction vector;

[0092] S4: selecting a point cloud point closest to the gaze direction vector, and calculating its color value c in the three-dimensional morphable model texture space;

[0093] S5: calculating the average color value c according to the color value c of the point cloud point and the weight w i mean :

[0094]

[0095] S6: multiplying the average color value c mean with the gaze direction vector v to obtain a 3D gaze direction vector v 3D :

[0096] v 3D = c mean · v

[0097] S7: calculating the coordinates p of the gaze point in the 3D coordinate system, wherein p0 is the center point coordinates of the head model;

[0098] p = p0 + v 3D · d

[0099] wherein d is the gaze distance;

[0100] S8: converting the coordinates of the gaze point into coordinates p cam in the camera coordinate system:

[0101] p cam = R cam (t cam -p)

[0102] wherein R cam is the rotation matrix of the camera, and t cam is the translation vector of the camera.

[0103] The beneficial effects of the present application are:

[0104] 1. Multi-dimensional state monitoring: the present application monitors the state of the driver in multiple ways, comprehensively judges the state of the driver by monitoring the physiological information, head posture and gaze direction of the driver in real time, and discovers the abnormal state of the driver in time, which can greatly improve the accuracy and reliability of the monitoring.

[0105] 2. Improve the safety driving awareness of the driver: the present application timely warns and reminds the driver after judging the abnormal state of the driver, so that the driver pays attention to safe driving and improves the safety driving awareness.

[0106] ​3. Improve traffic management efficiency: when the driver's state is found to be abnormal, the application can also notify the relevant traffic management personnel for processing, improving the efficiency of traffic management.

[0107] 4. Wide application: the application can be widely used in the fields of transportation and public safety, such as taxi, passenger car, freight car, police car and other transportation industries. BRIEF DESCRIPTION OF DRAWINGS

[0108] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the specific embodiments or the prior art description will be briefly introduced as follows.

[0109] Figure 1 A system framework diagram of a driver state monitoring method based on a three-dimensional deformation model according to the present application;

[0110] Figure 2 A flowchart of data storage and analysis steps of a driver state monitoring method based on a three-dimensional deformation model according to the present application. DETAILED DESCRIPTION

[0111] The embodiments of the technical solutions of the present application will be described in detail below with reference to the drawings, and the following embodiments are only used to more clearly illustrate the technical solutions of the present application, and therefore only serve as examples, but cannot limit the protection scope of the present application.

[0112] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in the present application should be understood as the usual meanings understood by the skilled in the art to which the present application belongs.

[0113] Referring to Figure 1 The embodiment of the present application provides a driver state monitoring method based on a three-dimensional deformation model, which comprises the following steps:

[0114] S1: Data acquisition: acquiring driver physiological information and driver operation information; the driver physiological information includes body temperature, heart rate, blood pressure, blood oxygen saturation, respiratory rate, sound, eye position, pupil size, blink frequency, retinal center, fixation point and eye movement information; the driver operation information includes driving video, steering wheel rotation angle, brake pedal force and throttle pedal force.

[0115] S2: Data processing:

[0116] S2.1: processing and analyzing driver physiological information;

[0117] S2.2: head posture estimation, estimating the posture of the driver's head through a region proposal network, a face alignment network and a three-dimensional deformation model;

[0118] S2.3: 3D gaze estimation is performed to estimate the gaze direction of the driver by computer vision technology.

[0119] S3: State judgment: whether the driver is in an abnormal state is judged according to the physiological information analysis result, head pose estimation and 3D gaze estimation, respectively.

[0120] The abnormal state information determined by the above three methods is taken as a union set.

[0121] When the judgment result is that the driver's state is abnormal, timely warning is given to remind the driver to drive safely, and relevant traffic management personnel are notified if necessary.

[0122] The abnormal state includes fatigue state, road rage state, distraction state and abnormal state.

[0123] The fatigue state is a state in which the driver appears obvious fatigue, including eye opening degree becoming smaller, blinking frequency increasing, fixation time becoming shorter, fixation point deviating and eye movement frequency slowing down.

[0124] The road rage state is a state in which the driver appears obvious emotional excitement and impulsive behavior, including heart rate significantly increasing or blood pressure significantly increasing, or showing angry behavior, including roaring, shouting, making angry sound, glaring at other drivers, frequently honking, frequently braking and hard acceleration.

[0125] The distraction state is a state in which the driver's attention is distracted during driving, which may have potential impact on driving safety.

[0126] The abnormal state is a state in which the driver appears abnormal during driving, including physiological parameter abnormality and behavior abnormality; the physiological parameter abnormality includes that the driver's heart rate, breathing rate, blood pressure or blood oxygen is out of the normal range; the behavior abnormality includes that the driver frequently changes lanes, drives on solid line, runs red light, over-accelerates, frequently brakes suddenly, leaves the main driving position, changes driver, intentionally interferes with the work of monitoring equipment or sensor.

[0127] S4: Data storage and analysis: refer to Figure 2 , the specific steps are as follows:

[0128] S4.1: According to the characteristics of the collected data and application requirements, design appropriate data model and database structure;

[0129] S4.2: Store the processed data into the database, including the driver's physiological information and operation information, as well as the results of head pose estimation and 3D gaze estimation;

[0130] S4.3: Clean the data to remove abnormal data and noise;

[0131] S4.4: Process the data, including data preprocessing, feature extraction, feature selection, and dimension reduction;

[0132] The data preprocessing includes data deduplication, missing value processing, outlier processing, and data normalization;

[0133] S4.5: Based on the data obtained from data preprocessing, feature extraction, feature selection, and dimension reduction, use data mining and statistical analysis techniques to analyze and model the driver's state, including clustering analysis, association rule mining, classification, and prediction;

[0134] S4.5.1: Use the K-means algorithm for clustering analysis;

[0135] S4.5.2: Use the Apriori algorithm for association rule mining, with the following steps:

[0136] (1) Define the minimum support and minimum confidence thresholds;

[0137] (2) Scan the data set to generate candidate 1-item sets;

[0138] (3) Remove item sets in the candidate 1-item set with support less than the minimum support;

[0139] (4) Generate candidate 2-item sets based on the remaining item sets;

[0140] (5) Remove item sets in the candidate 2-item set with support less than the minimum support;

[0141] (6) Generate candidate 3-item sets based on the remaining item sets;

[0142] (7) Remove item sets in the candidate 3-item set with support less than the minimum support;

[0143] (8) Continue in this manner until no more candidate item sets can be generated;

[0144] (9) Generate association rules based on frequent item sets;

[0145] (10) Remove association rules with confidence less than the minimum confidence;

[0146] The support calculation formula is:

[0147]

[0148] Where count(X) represents the number of times item set X appears in the data set, and N represents the total number of transactions in the data set;

[0149] The confidence calculation formula is:

[0150]

[0151] wherein X represents a premise item set, Y represents a conclusion item set;

[0152] generating candidate k-item sets from frequent k-1-item sets, wherein k>=2;

[0153] The candidate item set generation formula is:

[0154] C k = i U j | i e L k-1 , j e L k-1 , | i U j | = k, i < j

[0155] wherein C k represents a candidate k-item set, L k-1 represents a frequent k-1-item set;

[0156] S4.5.3: Classification using decision tree algorithm;

[0157] S4.5.4: Data prediction: input the real-time collected driver physiological information and driver operation information into the trained classification model during vehicle driving, to obtain the corresponding classification result.

[0158] S4.6: According to the analysis result, formulate corresponding early warning strategy and action plan, so as to timely remind the driver to standardize the safe driving, and notify the relevant traffic management personnel when necessary.

[0159] The head pose estimation estimates the pose of the driver's head in three-dimensional space through computer vision technology, and the specific steps are as follows:

[0160] S1: Face detection: use a face detection algorithm based on convolutional neural network to detect faces in an image and output a rectangular box to represent the position and size of the detected face in the image:

[0161] S1.1: Image preprocessing: pre-process the input image, including size normalization and color space conversion;

[0162] S1.2: Extract feature map: extract features from the pre-processed image through a convolutional neural network to obtain a set of feature maps;

[0163] S1.3: Generate candidate boxes using region proposal network: generate multiple candidate boxes on the feature map using a sliding window method, each candidate box i has a feature vector x i ;

[0164] S1.4: Classify and regress each candidate box: classify and regress each candidate box through a classifier and regressor to determine whether the region contains a face and the position and size of the face box:

[0165] The parameters of the classifier and regressor are W c ,W x ,W y ,W w ,W h and b c ,b x ,b y ,b w ,b h The probability that the region contains a face and the left top coordinates and width and height of the face box can be calculated as follows, respectively:

[0166]

[0167]

[0168]

[0169]

[0170]

[0171] where σ represents the Sigmoid function;

[0172] S1.5: Non-maximum suppression: the overlapping face boxes are de-duplicated by a non-maximum suppression algorithm to obtain the final face box:

[0173] (x min ,y min ,x max ,y max ) = (x-w / 2, y-h / 2, x+w / 2, y+h / 2)

[0174] where x, y are the center point coordinates of the prediction box; w, h are the width and height of the prediction box; (x min ,y min ) and (x max ,y max ) are the left top and right bottom coordinates of the face box, respectively;

[0175] S2: Feature point detection: based on the face image detected in the previous step, a face alignment network is used to detect the feature points on the face;

[0176] y i = F(x, θ i )

[0177] where x is the input face image, F is the face alignment network model, y iLet θ be the position of the i-th keypoint in the feature map. i Let be the network parameters for the i-th key point;

[0178] Supervised learning is used to train the face alignment network model, which optimizes the model parameters by minimizing the difference between the predicted and the true values.

[0179] The result of feature point detection can be represented as a two-dimensional coordinate array, i.e.:

[0180] P = (x1, y1), (x2, y2), ..., (x n ,y n )

[0181] Where n is the number of key points, (x i ,y i Let be the position of the i-th keypoint in the image;

[0182] S3: Pose Estimation: A 3D deformable model is used to represent the face shape and texture information. Then, the detected feature points are mapped onto the 3D deformable model to calculate the head pose in 3D space.

[0183] S3.1: Convert 2D feature point coordinates to 3D spatial coordinates: For each estimated feature point p i =(x i ,y i The depth value d of the corresponding point in the 3D deformation model can be used. i Convert it to 3D spatial coordinates (X i ,Y i Z i ):

[0184]

[0185] Among them, c x and c y These are the coordinates of the image center point, and f is the camera focal length;

[0186] S3.2: Register the estimated 3D coordinates with the control point coordinates in the 3D deformation model: Use the iterative nearest point algorithm to register the estimated 3D coordinates with the control point coordinates in the 3D deformation model to obtain the registered 3D coordinates;

[0187] S3.3: Obtain the pose parameters in the three-dimensional morphable model by linear regression: for the registered 3D coordinates and the corresponding control point coordinates in the three-dimensional morphable model, obtain the pose parameters θ in the three-dimensional morphable model by linear regression; that is, construct a linear equation group Aθ = b, where A is a 3n x 6 matrix, n is the number of control points, and b is a 3n-dimensional vector representing the registered 3D coordinates; solve θ by least squares method to obtain the pose parameters of the head in the three-dimensional morphable model.

[0188] The specific steps of the 3D gaze estimation are as follows:

[0189] S1: Render the known 2D human eye center point position and the shape and texture of the three-dimensional morphable model to obtain the corresponding 2D image of the 3D head model:

[0190] I 2D = R(s, R yaw , R pitch , R roll )MC

[0191] where I 2D is the rendered 2D image, R is the projection transformation matrix, s is the scale parameter, R yaw , R pitch , R roll are the head pose parameters, M is the three-dimensional morphable model, and C is the texture color of the three-dimensional morphable model;

[0192] S2: Detect the human eye center point position in the 2D image and calculate the pixel coordinates (x c , y c ) of the human eye center point in the image;

[0193] S3: Calculate the gaze direction vector v according to the human eye center point position and the rotation angle of the head pose:

[0194] v = R yaw R pitch v0

[0195] where v0 is the reference gaze direction vector;

[0196] S4: Select the point cloud point closest to the gaze direction vector and calculate its color value c in the texture space of the three-dimensional morphable model;

[0197] S5: Calculate the average color value c i according to the color value c of the point cloud point and the weight w mean :

[0198]

[0199] S6: Calculate the average color value cmean v is multiplied by the gaze direction vector v to obtain a 3D gaze direction vector v 3D :

[0200] v 3D = c mean ·v

[0201] S7: coordinates p of the gaze point in the 3D coordinate system are calculated, wherein p0 is the coordinate of the center point of the head model;

[0202] p = p0 + v 3D ·d

[0203] wherein d is the gaze distance;

[0204] S8: the coordinates of the gaze point are converted into coordinates p in the camera coordinate system cam :

[0205] p cam = R cam (t cam -p)

[0206] wherein R cam is the rotation matrix of the camera, and t cam is the translation vector of the camera.

[0207] The preferred embodiments of the present application are described in detail above in combination with the drawings, but the present application is not limited to the specific details in the above-described embodiments, and various simple modifications can be made to the technical solutions of the present application within the technical concept of the present application, and these simple modifications all belong to the protection scope of the present application.

[0208] In addition, it should be noted that each specific technical feature described in the above-described specific embodiments can be combined in any appropriate manner without contradiction, and in order to avoid unnecessary repetition, the present application will not describe various possible combinations again.

Claims

1. A driver state monitoring method based on a three-dimensional morphable model, characterized by, The method comprises the following steps: S1: data collection: collecting driver physiological information and driver operation information; The driver physiological information includes body temperature, heart rate, blood pressure, blood oxygen saturation, respiratory rate, voice, eye position, pupil size, blink rate, retinal center, fixation point and eye movement information; the driver operation information includes driving video, steering wheel rotation angle, brake pedal force and accelerator pedal force; S2: data processing: S2.1: processing and analyzing driver physiological information; S2.2: head posture estimation, estimating the posture of the driver's head through a region proposal network, a face alignment network and a three-dimensional deformation model; S2.3: 3D gaze estimation, estimating the gaze direction of the driver through computer vision technology; S3: state judgment: judging whether the driver is in an abnormal state according to the physiological information analysis result, head posture estimation and 3D gaze estimation respectively; Taking the union of the abnormal state information judged by the above three methods; When the judgment result is that the driver state is abnormal, timely warning reminds the driver to drive safely, and notifies the relevant traffic management personnel if necessary; The abnormal state includes fatigue state, road rage state, distraction state and abnormal state; The fatigue state is the state in which the driver appears obvious fatigue, including eye opening degree becoming smaller, blink rate increasing, fixation time becoming shorter, fixation point deviating and eye movement frequency slowing down; The road rage state is the state in which the driver appears obvious emotional excitement and impulsive behavior, including heart rate significantly increasing or blood pressure significantly increasing, or showing angry behavior, including roaring, shouting, making angry sound, glaring at other drivers, frequently honking, frequently braking and hard acceleration; The distraction state is the state in which the driver's attention is distracted during driving, which may have potential impact on driving safety; The abnormal state is the state in which the driver appears abnormal during driving, including physiological parameter abnormality and behavior abnormality; The physiological parameter abnormality includes that the heart rate, respiratory rate, blood pressure or blood oxygen of the driver exceeds the normal range; the behavior abnormality includes that the driver frequently changes lanes, drives on the solid line, runs a red light, accelerates excessively, frequently brakes suddenly, leaves the main driving position, changes the driver, intentionally interferes with the work of the monitoring equipment or sensor; S4: data storage and analysis: S4.1: designing appropriate data model and database structure according to the characteristics of the collected data and application requirements; S4.2: storing the processed data into the database, including driver physiological information and operation information, and the results of head posture estimation and 3D gaze estimation; S4.3: cleaning data, removing abnormal data and noise; S4.4: processing data, including data preprocessing, feature extraction, feature selection and dimension reduction; The data preprocessing includes data deduplication, missing value processing, abnormal value processing and data normalization; S4.5: Based on the data obtained from data preprocessing, feature extraction, feature selection and dimension reduction, use data mining and statistical analysis techniques to analyze and model the driver's state, including clustering analysis, association rule mining, classification and prediction; S4.5.1: Use the K-means algorithm for clustering analysis; S4.5.2: Use the Apriori algorithm for association rule mining, the steps are as follows: (1) Define the minimum support and minimum confidence thresholds; (2) Scan the data set to generate candidate 1-item sets; (3) Remove item sets in the candidate 1-item set whose support is less than the minimum support; (4) Generate candidate 2-item sets based on the remaining item sets; (5) Remove item sets in the candidate 2-item set whose support is less than the minimum support; (6) Generate candidate 3-item sets based on the remaining item sets; (7) Remove item sets in the candidate 3-item set whose support is less than the minimum support; (8) Continue in this way until no more candidate item sets can be generated; (9) Generate association rules based on frequent item sets; (10) Remove association rules whose confidence is less than the minimum confidence; The support calculation formula is: where count(X) represents the number of times item set X appears in the data set, and N represents the total number of transactions in the data set; The confidence calculation formula is: where X represents the premise item set and Y represents the conclusion item set; Generate candidate k-item sets based on frequent k-1-item sets, where k≥2; The candidate item set generation formula is: C k = i∪j|i∈L k-1 ,j∈L k-1 ,|i∪j|=k,i<j where C k denotes a candidate k-itemset, L k-1 denotes a frequent k-1-itemset; S4.5.3: Use the decision tree algorithm for classification; S4.5.4: Data prediction: input the real-time collected driver physiological information and driver operation information into the trained classification model during vehicle driving to obtain the corresponding classification result; S4.6: According to the analysis results, formulate corresponding warning strategies and action plans to timely remind the driver to drive safely and notify relevant traffic management personnel if necessary.

2. The driver state monitoring method based on a three-dimensional morphable model according to claim 1, wherein, The head pose estimation estimates the pose of the driver's head in three-dimensional space through computer vision technology, the specific steps are as follows: S1: Face detection: use a face detection algorithm based on a convolutional neural network to detect a face in an image and output a rectangular box to represent the position and size of the detected face in the image: S1.1: Image preprocessing: preprocess the input image, including size normalization and color space conversion; S1.2: Extract feature maps: extract feature maps from the preprocessed image through a convolutional neural network; S1.3: Generate candidate boxes with region proposal network: generate multiple candidate boxes on the feature map using a sliding window approach, each candidate box i has a feature vector x i ; S1.4: Classify and regress each candidate box: classify and regress each candidate box through a classifier and regressor to determine whether the region contains a face and the position and size of the face box: The parameters of the classifier and regressor are W c ,W x ,W y ,W w ,W h and b c ,b x ,b y ,b w ,b h The probability that the region contains a face and the left-top horizontal and vertical coordinates and the width and height of the face frame can be calculated as follows, respectively: where σ represents the Sigmoid function; S1.5: Non-maximum suppression: use a non-maximum suppression algorithm to remove overlapping face boxes to obtain the final face box: (x min ,y min ,x max ,y max ) = (x - w / 2, y - h / 2, x + w / 2, y + h / 2) Where x, y are the coordinates of the center point of the prediction box; w, h are the width and height of the prediction box; (x min ,y min ) and (x max ,y max ) represent the coordinates of the top left and bottom right corners of the face bounding box; S2: Feature point detection: based on the face image detected in the previous step, use a face alignment network to detect feature points on the face; y i = F(x, θ i ) Wherein, x is an input face image, F is a face alignment network model, y i is the position of the i th key point in the feature map, θ i is the network parameter of the i th key point; The method of supervised learning is used to train the face alignment network model, that is, the model parameters are optimized by minimizing the difference between the predicted value and the true value; The result of feature point detection can be represented as a two-dimensional coordinate array, that is: P = (xl,yl), (x2,y2),..., (xn,yn) n n P = (xl,yl), (x2,y2),..., (xn,yn)​ where n is the number of key points, (x i ,y i ) is the position of the i-th key point in the image; S3: Pose estimation: use a three-dimensional morphable model to represent the shape and texture information of the face, then map the detected feature points to the three-dimensional morphable model, and calculate the pose of the head in the three-dimensional space: S3.1: Convert 2D feature point coordinates to 3D space coordinates: For each feature point p i estimated, i ,y i ), it can be converted to 3D space coordinates (X i ,Y i ,Z i ) by the depth value d i of the corresponding point in the three-dimensional morphable model: where c x and c y are the image center point coordinates, and f is the camera focal length. S3.2: Register the estimated 3D coordinates with the control point coordinates in the three-dimensional morphable model: by using the iterative closest point algorithm, the estimated 3D coordinates are registered with the control point coordinates in the three-dimensional morphable model, and the registered 3D coordinates are obtained; S3.3: Obtain the pose parameters in the three-dimensional morphable model by linear regression: for the registered 3D coordinates and the corresponding control point coordinates in the three-dimensional morphable model, the pose parameters θ in the three-dimensional morphable model are obtained by linear regression; that is, a linear equation system Aθ = b is constructed, where A is a 3n×6 matrix, n is the number of control points, and b is a 3n-dimensional vector representing the registered 3D coordinates; θ is solved by least squares method, and the pose parameters of the head in the three-dimensional morphable model are obtained.

3. The driver state monitoring method based on a three-dimensional morphable model according to claim 1, wherein, The specific steps of the 3D gaze estimation are as follows: S1: Render the known 2D eye center point position and the shape and texture of the three-dimensional morphable model to obtain the 2D image corresponding to the 3D head model: I 2D = R(s, R yaw , R pitch , R roll )MC wherein I 2D is the rendered 2D image, R is a projection transformation matrix, s is a scale parameter, R yaw is a rotation matrix, and pitch is a translation matrix. roll is a head pose parameter, M is a three-dimensional morphable model, and C is a texture color of the three-dimensional morphable model. S2: detecting the human eye center point position in the 2D image, calculating the pixel coordinates (x c ,y c ) of the human eye center point in the image; S3: According to the position of the eye center point and the rotation angle of the head pose, the gaze direction vector v is calculated: v = R yaw R pitch v0 Where v0 is the reference gaze direction vector; S4: Select the point cloud point closest to the gaze direction vector, and calculate its color value c in the texture space of the three-dimensional morphable model; S5: calculating an average color value c from the color values c and the weights w of the point cloud points i mean :​ S6: average color value c is calculated from the color values c of the pixels in the gaze direction vector v mean is multiplied with the gaze direction vector v, resulting in a 3D gaze direction vector v 3D : v 3D = c mean ·v S7: Calculate the coordinates p of the gaze point in the 3D coordinate system, where p0 is the center point coordinates of the head model; p = po + v 3D • d Where d is the gaze distance; S8: convert the coordinates of the gaze point into coordinates p in the camera coordinate system cam : p cam = R cam (t cam -p) where R cam is the rotation matrix of the camera, t cam is the translation vector of the camera.

Images