A method for detecting driver fatigue based on 3D head modeling

Abstract

The present invention belongs to the field of vehicle driving control technology and relates to a method for detecting a driver's fatigue driving status based on 3D head modeling, comprising step S1: using a depth camera to obtain a point cloud of the driver's head's external contours and a portable ultrasound probe to scan the driver's blood vessel locations to obtain a 3D model of the driver's head; step S2: obtaining a physiological indicator test of the driver based on the 3D head model; step S3: defining an input tensor and constructing a one-dimensional convolutional neural network for processing multimodal time series data; step S4: designing an output layer to determine the driver's fatigue driving level; and step S5: implementing a corresponding vehicle control strategy based on the obtained driver's fatigue driving level. By using a convolutional neural network to calculate the driver's fatigue level score, it is determined whether the driver is driving fatigued. After determining the driver's fatigue driving status, the driver is reminded to take a rest through an in-vehicle voice broadcast and an external warning light, and other vehicles are reminded to avoid the driver.

Description

Technical Field

[0001] The present invention belongs to the technical field of vehicle driving control for detecting driver fatigue status; in particular, it relates to a method for detecting driver fatigue driving status based on 3D head modeling. Background Art

[0002] Driver fatigue is one of the main causes of traffic accidents. When fatigued, a driver's physical function and judgment abilities decline significantly, manifesting as increased braking reaction time, distracted attention, and reduced road dynamics perception. This significantly increases the probability of rear-end collisions and vehicle deviation. Therefore, timely and accurate detection of driver fatigue is crucial. Fatigue detection can significantly reduce the risk of traffic accidents caused by fatigue. When signs of driver fatigue are detected, prompting the driver to rest and other vehicles to carefully avoid them can effectively protect the lives of road users, reduce damage to vehicles and road infrastructure, and save manpower and resources required for accident rescue, handling, and post-accident recovery. Furthermore, continuous fatigue monitoring helps cultivate driver safety awareness, cultivating habits of consciously paying attention to their condition and proactively preventing fatigued driving, thereby fundamentally improving road safety.

[0003] At present, the commonly used indicators for detecting fatigue driving status include: EEG signals, eye information, blood pressure, heart rate, facial information, head pitch angle, steering wheel information and speech rate. Fatigue driving status detection usually does not use only a single indicator, but integrates the above indicators to comprehensively judge whether the driver is driving fatigued. For example, Chinese patent CN119479194A discloses a fatigue driving detection method based on facial recognition analysis and taking into account the driver's heart rate monitoring data, blood oxygen monitoring data and other data; Chinese patent CN119478898A discloses a fatigue driving status monitoring method based on vital signs such as heart rate, respiratory rate, blood oxygen saturation, blood pressure and heart rate variability; Chinese patent CN118279878B discloses a multi-modal physiological information fusion vehicle driver fatigue status intelligent identification method, which comprehensively considers various indicators and can reduce the misjudgment rate of fatigue driving status, but still fails to overcome the following shortcomings of each indicator:

[0004] (1) EEG signals: They are susceptible to environmental interference. The vibration of the vehicle during driving, the electromagnetic waves of electronic equipment, and even the slight shaking of the driver's head can cause a significant increase in signal noise.

[0005] (2) Eye information: Light or frequently changing lighting conditions may induce non-fatigue-related blinking movements, leading to misjudgment.

[0006] (3) Blood pressure: Diet, mood, low / high blood pressure, etc. have a great impact on blood pressure and affect the judgment of fatigue driving status.

[0007] (4) Heart rate: Emotional fluctuations such as anxiety, tension, or excitement can significantly affect heart rate, leading to misjudgment of fatigue status.

[0008] (5) Facial information: This is often obtained using image processing technology, but the accuracy of this indicator depends on the clarity of the image. In completely dark environments, strong light environments, and when the vehicle is violently bumpy, the image will inevitably become blurred, affecting the accuracy of the detection results. In addition, when the interior environment is complex and the driver's face is obscured, this indicator cannot be obtained.

[0009] (6) Head pitch angle: Like facial information, it cannot be obtained when the vehicle is bumpy or blocked.

[0010] (7) Steering wheel information: When driving on bumpy roads or curves, the natural shaking of the steering wheel may lead to misjudgment.

[0011] (8) Speaking speed: There are large differences in speaking speed and tone between individuals, and the speaking speed and tone of the same individual are also easily affected by emotions. Therefore, the accuracy of fatigue driving status detection based on this indicator is not high.

[0012] In summary, the existing methods for detecting fatigue driving status are insufficient in terms of anti-interference, detection accuracy, and environmental adaptability. They cannot guarantee detection accuracy in strong light, complete darkness, and vehicle bumps, and they easily ignore individual differences among drivers. Summary of the Invention

[0013] In view of the above problems, the purpose of the present invention is to provide a method for detecting the driver's fatigue driving status based on 3D head modeling. A 3D model of the driver's head is established through a depth camera, an ultrasonic probe and an IMU, and the driver's head is modeled through a detector to obtain the driver's head center of gravity position, facial information, and prefrontal blood flow. According to the above-mentioned driver's physiological information indicators, a convolutional neural network is used to calculate the driver's fatigue level score to determine whether the driver is fatigued driving. After determining the driver's fatigue driving status, the driver is reminded to take a rest and to avoid other vehicles through voice broadcasts in the car and warning lights outside the car.

[0014] The present invention provides a method for detecting a driver's fatigue driving state based on 3D head modeling, which specifically includes the following steps:

[0015] Step S1: The depth camera obtains the external contour point cloud of the driver's head, and the portable ultrasound probe scans the location of the driver's blood vessels to obtain a 3D model of the driver's head;

[0016] Step S2: Based on the 3D model of the driver's head, physiological indicators such as eyelid closure, mouth corner position deviation, head center of gravity deviation, frontal lobe blood flow, and tongue status are detected to obtain the driver's physiological indicators;

[0017] Step S3: defining an input tensor, wherein the input is time series data containing multimodal physiological indicators, and constructing a one-dimensional convolutional neural network for processing multimodal time series data;

[0018] Step S4, output layer design: design the output layer based on the input tensor and the input layer of the convolutional neural network (CNN) architecture to obtain a probability vector of the driver's fatigue driving state and determine the driver's fatigue driving degree;

[0019] Step S5: Control the vehicle of the driver who is detected to be in a fatigue driving state, and implement a corresponding vehicle control strategy based on the obtained driver's fatigue driving level.

[0020] As a preferred embodiment of the present invention, step S1 further includes the following steps:

[0021] Step S1.1: Driver head data collection,

[0022] Step S1.1.1: External contour modeling:

[0023] Use a depth camera to obtain 3D point cloud data of the driver's head surface. The calculation formula is:

[0024] ;

[0025] : Driver’s head surface point cloud data; : Number of collected point clouds; : No. The three-dimensional coordinates of a point in the global coordinate system;

[0026] Use Delaunay triangulation algorithm to convert point cloud data into continuous surfaces :

[0027] ;

[0028] : a set of vertices, : No. vertices, : number of vertices, :vertex coordinates of

[0029] : a collection of triangles, : No. One-sided piece, : number of patches, A collection from vertices , is a triangular patch The vertex of

[0030] Step S1.1.2, internal structure mapping:

[0031] Install a portable ultrasound probe on the headrest of the driver's seat to perform ultrasound testing on the driver's head to obtain the blood vessel distribution of the driver's head;

[0032] Step S1.1.2.1, blood vessel centerline extraction:

[0033] The vascular structure module is enhanced using a three-dimensional Hessian matrix filter to extract the centerline coordinates:

[0034] ;

[0035] : The coordinates of the midpoint of the blood vessel centerline and the set of blood vessel radius;

[0036] , , : No. The three-dimensional coordinates of the midpoint of the segment vessel centerline;

[0037] : No. The radius of the segmental vessel;

[0038] : total number of vascular segments;

[0039] Step S1.1.2.2: Construction of vascular fractal model:

[0040] Based on fractal dimension Describe the complexity of vascular branching:

[0041] ;

[0042] : Radius is less than The number of blood vessels;

[0043] : fractal dimension;

[0044] Step S1.1.2.3, Inertial Unit Measurement:

[0045] The head posture is measured by the IMU sensor, and the rotation is represented by the posture quaternion:

[0046] Attitude quaternion :

[0047] ;

[0048] : Real part, indicating the rotation angle;

[0049] , , : Imaginary part, indicating the direction of the rotation axis;

[0050] :along The unit vector in the positive direction of the axis;

[0051] :along The unit vector in the positive direction of the axis;

[0052] :along The unit vector in the positive direction of the axis;

[0053] Step S1.2, coordinate system alignment:

[0054] Step S1.2.1, global coordinate system definition:

[0055] Head center of gravity As the origin, the vertical direction is the vertical axis The direction of the axis, The positive direction of the axis points to the roof, perpendicular to Axis and pointing directly in front of the driver's face (nose direction) For front and rear axles The positive direction of the axis is perpendicular to Axis and The plane formed by the axes is the left and right axes The direction of the axis, The positive direction of the axis points to the right, defining the global three-dimensional rectangular coordinate system :

[0056]

[0057] : Coordinates of the center of gravity of the head in the initial state;

[0058] : Number of collected point clouds;

[0059] Step S1.2.2, sensor external parameter calibration:

[0060] Align the coordinate systems of the depth camera, portable ultrasound probe, and IMU sensor to the global coordinate system:

[0061] Each transformation matrix is ​​of the form:

[0062] ;

[0063] : Rotation matrix;

[0064] : translation vector;

[0065] Step S1.3, point cloud preprocessing,

[0066] Step S1.3.1, outlier filtering:

[0067] Calculate the distance and standard deviation of all points in the neighborhood of each point. If the distance between a point in the neighborhood and the point exceeds the threshold, it is considered an outlier and removed.

[0068] Neighborhood radius Average distance of inliers and standard deviation :

[0069] ;

[0070] ;

[0071] :Point cloud points The neighborhood set of In radius Points within millimeters;

[0072] : the number of points in the neighborhood;

[0073] :To be evaluated 3D point coordinates;

[0074] : Neighborhood The 3D point coordinates;

[0075] The threshold is set as:

[0076] ;

[0077] Step S1.3.2, voxel grid downsampling:

[0078] By evenly dividing the point cloud into voxels;

[0079] Downsampled point cloud collection for:

[0080] ;

[0081] :To be evaluated 3D point coordinates;

[0082] : No. The set of points within a voxel;

[0083] : the number of points after downsampling;

[0084] Step S1.4, model fusion and final output,

[0085] Step S1.4.1: Fusion of triangulated mesh surface and blood vessels:

[0086] The blood vessel centerline data Mapping to the triangular mesh surface to obtain a 3D head model including blood vessel distribution , guiding NIRS sensor placement:

[0087] ;

[0088] : The coordinates of the midpoint of the blood vessel centerline and the set of blood vessel radius;

[0089] , , : No. 3D coordinates of the segmented vessel centerline;

[0090] : No. The radius of the segmental vessel;

[0091] Step S1.4.2, head dynamic posture compensation:

[0092] pass Rotation Matrix Update the model pose in real time:

[0093] ;

[0094] : Quaternion representation, describing the rotation joint lock problem of the head in three-dimensional space;

[0095] : Real part, indicating the rotation angle;

[0096] , , : Imaginary part, indicating the direction of the rotation axis;

[0097] :along Unit vector in the direction of the axis;

[0098] :along Unit vector in the direction of the axis;

[0099] :along Unit vector in the direction of the axis;

[0100] Step S1.4.3, model output:

[0101] The final model contains:

[0102] Geometric surface, i.e. triangular mesh ;

[0103] Vascular network, i.e., classification model ;

[0104] Dynamic posture parameters, quaternions With the rotation matrix .

[0105] As a preference of the present invention, step S2 further includes the following steps:

[0106] Step S2.1: Eyelid closure Detection,

[0107] 3D keypoint detection:

[0108] Locate the upper eyelid point With lower eyelid point is the pupil center and The intersection of the vertical line and the upper and lower eyelids;

[0109] Closure calculation :

[0110] ;

[0111] Reference distance when eyes are open :

[0112] ;

[0113] Reference distance when eyes are closed :

[0114] ;

[0115] in Upper eyelid point , Lower eyelid point ;

[0116] : Real-time three-dimensional coordinates of upper eyelid points;

[0117] : Real-time 3D coordinates of lower eyelid points;

[0118] : When the eyes are fully open, the upper eyelid is pointed coordinate;

[0119] : Lower eyelid point when eyes are fully open coordinate;

[0120] : Upper eyelid point when eyes are completely closed z coordinate;

[0121] : Lower eyelid point when eyes are completely closed z coordinate;

[0122] : baseline distance when eyes are open;

[0123] : Reference distance when eyes are closed;

[0124] Step S2.2: Mouth corner position offset Detection,

[0125] 3D keypoint extraction:

[0126] Locate the left and right corners of the mouth and , and the reference position of the left corner of the mouth in the resting state and the reference position of the right corner of the mouth in the resting state ;

[0127] Dynamic offset :

[0128] ;

[0129] : Left corner of mouth ;

[0130] : Right corner of mouth ;

[0131] : Real-time three-dimensional coordinates of the left corner of the mouth;

[0132] : Real-time three-dimensional coordinates of the right mouth corner;

[0133] :Right now , the reference position of the left corner of the mouth in the resting state;

[0134] :Right now , the reference position of the right corner of the mouth in the resting state;

[0135] : three-dimensional coordinates of the left corner of the mouth in the resting state;

[0136] : three-dimensional coordinates of the right mouth corner in the resting state;

[0137] Step S2.3: Head center of gravity shift ,

[0138] Real-time center of gravity of the head calculate:

[0139] ;

[0140] : Number of collected point clouds;

[0141] , , : No. The real-time coordinates of each point in the global coordinate system;

[0142] offset :

[0143] ;

[0144] : Real-time center of gravity coordinates of the head;

[0145] : Coordinates of the center of gravity of the head in the initial state;

[0146] Step S2.4: Prefrontal blood flow Detection,

[0147] NIRS sensor deployment:

[0148] Based on the 3D model of the driver's head, the driver is advised to place the NIRS sensor in the area with dense blood vessels;

[0149] blood flow Estimate:

[0150] ;

[0151] : Change in oxygenated hemoglobin concentration;

[0152] : baseline concentration of oxygenated hemoglobin at rest;

[0153] : calibration coefficient;

[0154] Step S2.5, tongue status ( 、 、 、 、 ) detection,

[0155] The oral area is 3D modeled separately, using a near-infrared structured light camera installed above the dashboard with a wavelength of 850 mm to penetrate the lip surface and collect dynamic point cloud data of the lip at 30 frames per second. :

[0156] ;

[0157] : No. The coordinates of the points;

[0158] : The total number of lip dynamic point cloud data collected;

[0159] Use the U-Net network to perform semantic segmentation of the oral cavity and extract the coordinates of the tongue tip ;

[0160] Three-dimensional displacement trajectory analysis:

[0161] Displacement amplitude

[0162] The reference position relative to the tip of the tongue

[0163] The instantaneous displacement of:

[0164] ;

[0165] : displacement amplitude of tongue tip;

[0166] : Coordinates of the tongue tip at the moment;

[0167] : tongue tip reference position;

[0168] Yawning is large and continuous. , the amplitude is small and periodic when speaking, ;

[0169] Direction of movement:

[0170] For continuous K Tongue tip coordinates of the frame Perform polynomial fitting to obtain the motion trajectory equation:

[0171] ;

[0172] in 、 、 is the trajectory equation parameter, which is a three-dimensional vector;

[0173] : trajectory motion equation;

[0174] Calculate the instantaneous motion direction vector :

[0175] ;

[0176] In the global coordinate system, calculate the vertical axis Angle :

[0177] ;

[0178] : The angle between the instantaneous motion direction vector and the vertical axis;

[0179] : instantaneous motion direction vector;

[0180] Yawning: mainly vertically downward, ;

[0181] Speaking: Mainly horizontal swing, ;

[0182] Instantaneous velocity and acceleration:

[0183] Instantaneous speed :

[0184] ;

[0185] : The coordinates of the tip of the tongue at that moment;

[0186] : The coordinates of the tip of the tongue at the moment ;

[0187] Yawning: low speed, , showing a trend of slow rise and slow fall;

[0188] Speak: high speed, , accompanied by short-duration pulses;

[0189] acceleration :

[0190] ;

[0191] : The instantaneous velocity of the tongue tip at that moment;

[0192] : The instantaneous velocity of the tongue tip at that moment;

[0193] Yawning: the absolute value of acceleration is small, ;

[0194] Speech: The absolute value of acceleration changes dramatically. ;

[0195] Motion trajectory complexity:

[0196] The complexity is quantified by calculating the approximate entropy of the tongue movement path:

[0197] Parameter setting: Embedding dimension , tolerance ;

[0198] Constructing a time series vector :

[0199] ;

[0200] : No. The displacement amplitude of the tongue tip at each time point;

[0201] : No. The displacement amplitude of the tongue tip at each time point;

[0202] : index of time series, range in is the total data length, is the embedding dimension;

[0203] Statistical similarity vector ratio :

[0204] ;

[0205] : Tolerance is the threshold for judging whether two time series vectors are similar;

[0206] : Within tolerance Inside, with vectors Similar proportions, is a time series vector;

[0207] : total data length;

[0208] : embedding dimension;

[0209] : index of time series, range in is the total data length, is the embedding dimension, Indicates the vectors;

[0210] Calculating entropy :

[0211] ;

[0212] :The embedding dimension is The ratio of similar vectors when ;

[0213] : Complexity of tongue movement trajectory, i.e. entropy value ;

[0214] Yawn: Simple trajectory, low value, ;

[0215] Speech: Complex trajectory, high value, .

[0216] As a preference of the present invention, step S3 further includes the following steps:

[0217] Step S3.1, input data definition,

[0218] The input is time series data containing multimodal physiological indicators, each time window contains sampling points, the feature dimension is 9;

[0219] Input Tensor ;

[0220] Input Example The matrix:

[0221] ;

[0222] : Sampling point The degree of eyelid closure at ;

[0223] : Sampling point The mouth corner position shifts when ;

[0224] : Sampling point The head center of gravity shifts when ;

[0225] : Sampling point Prefrontal blood flow, ;

[0226] : Sampling point The tongue displacement amplitude at ;

[0227] : Sampling point The direction of tongue movement, ;

[0228] : Sampling point The tongue movement speed, ;

[0229] : Sampling point The tongue acceleration at ;

[0230] : Sampling point The complexity of tongue movement trajectory, ;

[0231] Step S3.2, Convolutional Neural Network (CNN) architecture,

[0232] A one-dimensional convolutional neural network was used to process multimodal time series data. The network architecture starts with the input layer, receiving a feature sequence of 60 time steps and 9 dimensions. It then extracts spatiotemporal features through two layers of convolution and pooling modules. The first layer uses 64 convolution kernels of size 5 to slide along the time dimension, capturing local temporal patterns through the ReLU activation function. This is followed by maximum pooling with a stride of 2 to compress the feature dimension and enhance translation invariance. The second layer further stacks 128 convolution kernels of size 3 to fuse cross-modal associations at a higher level of abstraction. After further pooling, high-level spatiotemporal features are obtained. The feature maps are flattened and then connected to the fully connected layer. Global feature integration is first performed through a 256-dimensional hidden layer. Finally, a three-neuron output layer combined with a Softmax function generates the probability distribution of wakefulness, mild fatigue, and severe fatigue.

[0233] Among them, the convolution layer 1 is:

[0234] ;

[0235] : Convolutional layer 1;

[0236] : No. The convolution kernel is Input channel, The weight of the time step;

[0237] : No. The bias value of the convolution kernel;

[0238] : input tensor;

[0239] : convolution kernel index, range , indicating that there are 64 convolution kernels in the first layer;

[0240] : Input feature channel index, range , corresponding to 9 input feature dimensions;

[0241] : Output feature map time step index, input time window length is 60, output dimension is ;

[0242] Max Pooling Layer 1:

[0243] ;

[0244] : Maximum pooling layer 1;

[0245] : The first output feature map of convolutional layer 1 channels at time step The value of

[0246] : The first output feature map of convolutional layer 1 channels at time step The value of

[0247] : The time step index after pooling, the pooling step size is 2, and the output dimension is ;

[0248] Convolutional layer 2:

[0249] ;

[0250] : Convolutional layer 2;

[0251] : No. The convolution kernel is Input channel, The weight of the time step;

[0252] : The first output of pooling layer 1 channels at time step The value of

[0253] : convolution kernel index, range , indicating that there are 128 convolution kernels in the first layer;

[0254] : Input feature channel index, range , corresponding to 64 output channels of layer 1;

[0255] : The time step index of the output feature map, the output dimension is ;

[0256] Max Pooling Layer 2:

[0257] ;

[0258] : Maximum pooling layer 2;

[0259] : The first output feature map of convolutional layer 2 channels at time step The value of

[0260] : The first output feature map of convolutional layer 2 channels at time step The value of

[0261] : The time step index after pooling, the pooling step size is 2, and the output dimension is ;

[0262] Fully connected layer 1:

[0263] ;

[0264] : Fully connected layer 1;

[0265] : The weight matrix of the fully connected layer 1, the dimension is ;

[0266] : Output of pooling layer 2 The dimension is , the length of the flattened vector is 1920;

[0267] : The bias term of the fully connected layer 1, with a dimension of 256;

[0268] Fully connected layer 2:

[0269] ;

[0270] : Fully connected layer 2;

[0271] : The weight matrix of the fully connected 2 layers, the dimension is ;

[0272] : The output vector of the fully connected layer 1, with a dimension of 256;

[0273] : Bias term of fully connected layer 2, dimension is 3.

[0274] As a preference of the present invention, step S4 also includes the following steps:

[0275] Step S4.1, output layer design,

[0276] The output layer is designed as a fully connected layer containing three neurons, corresponding to the three states of the driver: alert, mildly fatigued, and severely fatigued. Each neuron uses the Softmax activation function to convert the linear output of the previous layer into a normalized probability value, indicating the confidence level of the state. Specifically, the raw output value of the last layer of the network is transformed by Softmax to generate a three-element probability vector, where each element represents the predicted probability of the corresponding category, and the sum of all probabilities equals 1. By setting a probability threshold, if the probabilities of all categories are lower than the set probability threshold, the state is marked as uncertain and the multimodal data review mechanism module is activated for review.

[0277] Output dimension: 3D probability vector , corresponding to three types of fatigue states;

[0278] :wide awake, ;

[0279] : Mild fatigue, ;

[0280] : Severe fatigue, ;

[0281] : Karolinska Sleepiness Scale;

[0282] Softmax formula:

[0283] ;

[0284] , is the original output value of the fully connected layer 2, and e is a natural constant;

[0285] Step S4.2, judgment rules,

[0286] Direct classification:

[0287] Take the category with the highest probability as the prediction result:

[0288]

[0289] : The classification result of the driver's current fatigue level is a discrete category label with values ​​of 0, 1, and 2;

[0290] : The traversed variable takes values ​​0, 1, and 2. It is used to compare the probability values ​​of the three categories and return the index of the category with the highest probability.

[0291] As a preference of the present invention, step S5 also includes the following steps:

[0292] Step S5.1: Based on the detected driver fatigue level, adopt the following vehicle control strategy:

[0293] When the driver is slightly fatigued, a low-frequency voice broadcast is activated to remind the driver to stop and rest, reduce the throttle opening to 4 / 5 of the current opening, and adjust the throttle opening according to the fatigue probability value. Dynamically adjust the throttle response curve to lower the acceleration limit:

[0294] ;

[0295] is the current maximum acceleration of the vehicle, is the original maximum acceleration of the vehicle, : Mild fatigue, ;

[0296] When severe driver fatigue is detected, a high-frequency in-car voice broadcast is activated to remind the driver to stop and rest, the lights are controlled to flash to remind other drivers to take shelter, the throttle opening is reduced to 5 / 6 of the current opening, the automatic transmission is locked, downshifting and acceleration are prohibited, and the engine output torque is directly limited to a safe threshold:

[0297] ;

[0298] in

[0299] is the current torque of the engine,

[0300] is the rated torque of the engine,

[0301] The current engine output torque safety threshold.

[0302] The beneficial effects of the present invention are as follows:

[0303] 1. This invention addresses the issue of uneven signal quality caused by individual physiological differences in drivers, such as skin transmittance and subcutaneous fat thickness, by collecting blood flow. By using a depth camera and a portable ultrasound probe to create a 3D model of the driver's head and scan the driver's vascular distribution, the system helps drivers more accurately locate areas with dense blood vessels, improving signal quality and detection accuracy. Furthermore, the use of a detector with a smaller contact area and softer material reduces interference with the driver, thereby enhancing detection accuracy.

[0304] 2. The present invention addresses the problems existing in traditional face recognition and solves the problems of being unable to obtain complete eyelid closure and mouth corner position deviation in strong light and occlusion conditions. First, the application of head 3D modeling can work all day and night, is completely independent of the visible light environment, and can work in strong backlight and complete darkness. In addition, the head 3D modeling has better stability in complex scenes and can work normally in typical scenes such as windshield reflections, drivers wearing sunglasses, and objects occluding the vehicle. Finally, the head 3D modeling can also realize self-correction of head posture and dynamic compensation of expression detection. For example, when the driver turns his head to look in the rearview mirror, the 3D model restores the frontal expression through geometric transformation, making up for the defect of misjudgment of 2D images due to perspective distortion.

[0305] 3. This invention introduces two new discriminant indicators, prefrontal cortex blood flow and tongue position, as the basis for detecting fatigue driving, reducing the false positive rate and increasing the reliability of the results. This is because the prefrontal cortex is responsible for higher-level cognitive functions such as decision-making and attention, and its activity decreases during fatigue. Furthermore, the tongue position differs when yawning due to fatigue and when speaking normally. BRIEF DESCRIPTION OF THE DRAWINGS

[0306] By referring to the following description in conjunction with the accompanying drawings, and with a more complete understanding of the present invention, other objects and results of the present invention will become more clear and easy to understand. In the accompanying drawings:

[0307] Figure 1 Schematic diagram of the global coordinate system of the present invention;

[0308] Figure 2 Schematic diagrams of the upper and lower eyelids when the eyes are open (right) and closed (left) according to the present invention;

[0309] Figure 3 This is a schematic diagram of input data for the input tensor of the present invention;

[0310] Figure 4 Schematic diagram of the convolutional neural network structure of the present invention. DETAILED DESCRIPTION

[0311] See Figure 1-2 The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0312] An embodiment of the present invention provides a method for detecting a driver's fatigue driving state based on 3D head modeling, comprising the following steps:

[0313] Step S1: The depth camera obtains the external contour point cloud of the driver's head, and the portable ultrasound probe scans the location of the driver's blood vessels to obtain a 3D model of the driver's head;

[0314] Step S1.1: Driver head data collection,

[0315] Step S1.1.1: External contour modeling:

[0316] Use a depth camera to obtain 3D point cloud data of the driver's head surface. The calculation formula is:

[0317] ;

[0318] : Driver’s head surface point cloud data; : Number of collected point clouds; : No. The three-dimensional coordinates of a point in the global coordinate system;

[0319] Use Delaunay triangulation algorithm to convert point cloud data into continuous surfaces :

[0320] ;

[0321] : a set of vertices, : No. vertices, : number of vertices, :vertex coordinates of

[0322] : a collection of triangles, : No. One-sided piece, : number of patches, A collection from vertices , is a triangular patch The vertex of

[0323] Step S1.1.2, internal structure mapping:

[0324] A portable ultrasound probe is installed on the headrest of the driver's seat to perform ultrasound examination of the driver's head, supplemented by standard anatomical atlases to obtain the distribution of blood vessels in the driver's head;

[0325] Step S1.1.2.1, blood vessel centerline extraction:

[0326] The vascular structure module is enhanced using a three-dimensional Hessian matrix filter to extract the centerline coordinates:

[0327] ;

[0328] : The coordinates of the midpoint of the blood vessel centerline and the set of blood vessel radius;

[0329] , , : No. The three-dimensional coordinates of the midpoint of the segment vessel centerline (in millimeters);

[0330] : No. The radius of the segmental vessel (in millimeters);

[0331] : total number of vascular segments;

[0332] Step S1.1.2.2: Construction of vascular fractal model:

[0333] Based on fractal dimension Describe the complexity of vascular branching:

[0334] ;

[0335] : Radius is less than The number of blood vessels;

[0336] : Fractal dimension (typical value ); parameters describing the self-similarity and structural complexity of vascular networks, The larger the value, the more complex the blood vessel branches.

[0337] Step S1.1.2.3, Inertial Unit Measurement:

[0338] The head posture is measured by the IMU sensor, and the rotation is represented by the posture quaternion:

[0339] Attitude quaternion :

[0340] ;

[0341] : Real part, indicating the rotation angle;

[0342] , , : Imaginary part, indicating the direction of the rotation axis;

[0343] :along The unit vector in the positive direction of the axis;

[0344] :along The unit vector in the positive direction of the axis;

[0345] :along The unit vector in the positive direction of the axis;

[0346] Step S1.2, coordinate system alignment:

[0347] Step S1.2.1, global coordinate system definition:

[0348] Head center of gravity As the origin, the vertical direction is the vertical axis The direction of the axis, The positive direction of the axis points to the roof, perpendicular to Axis and pointing directly in front of the driver's face (nose direction) For front and rear axles The positive direction of the axis is perpendicular to Axis and The plane formed by the axes is the left and right axes The direction of the axis, The positive direction of the axis points to the right, defining the global three-dimensional rectangular coordinate system :

[0349]

[0350] : Coordinates of the center of gravity of the head in the initial state (unit: mm);

[0351] : Number of collected point clouds;

[0352] Step S1.2.2, sensor external parameter calibration:

[0353] Align the coordinate systems of the depth camera, portable ultrasound probe, and IMU sensor to the global coordinate system:

[0354] Each transformation matrix is ​​of the form:

[0355] ;

[0356] : Rotation matrix;

[0357] : translation vector (in millimeters);

[0358] Step S1.3, point cloud preprocessing,

[0359] Step S1.3.1, outlier filtering:

[0360] Calculate the distance and standard deviation of all points in the neighborhood of each point. If the distance between a point in the neighborhood and the point exceeds the threshold, it is considered an outlier and removed.

[0361] Neighborhood radius Inside (here take mm) average distance between points and standard deviation :

[0362] ;

[0363] ;

[0364] :Point cloud points The neighborhood set of In radius Points within millimeters;

[0365] : The number of points in the neighborhood ( The value changes dynamically, depending on the local point density);

[0366] :To be evaluated Coordinates of three-dimensional points (in millimeters);

[0367] : Neighborhood The Coordinates of three-dimensional points (in millimeters);

[0368] The threshold is set as:

[0369] ;

[0370] Step S1.3.2, voxel grid downsampling:

[0371] By evenly dividing the point cloud into voxels (cube grid), the point cloud density can be reduced and the amount of data can be reduced; for example, the voxel size can be set to 2 mm, and only the center point or mean point of each voxel is retained.

[0372] Downsampled point cloud collection for:

[0373] ;

[0374] :To be evaluated Coordinates of three-dimensional points (in millimeters);

[0375] : No. The set of points within a voxel;

[0376] : the number of points after downsampling;

[0377] Step S1.4, model fusion and final output,

[0378] Step S1.4.1: Fusion of triangulated mesh surface and blood vessels:

[0379] The blood vessel centerline data Mapping to the triangular mesh surface to obtain a 3D head model including blood vessel distribution , guiding NIRS (Near Infrared Spectrometer) sensor layout:

[0380] ;

[0381] : The coordinates of the midpoint of the blood vessel centerline and the set of blood vessel radius;

[0382] , , : No. 3D coordinates of the centerline of the segmented blood vessel (in millimeters);

[0383] : No. The radius of the segmental vessel (in millimeters);

[0384] Step S1.4.2, head dynamic posture compensation:

[0385] pass Rotation Matrix Update the model pose in real time:

[0386] ;

[0387] : Quaternion representation, describing the rotation joint lock problem of the head in three-dimensional space;

[0388] : Real part, indicating the rotation angle;

[0389] , , : Imaginary part, indicating the direction of the rotation axis;

[0390] :along Unit vector in the direction of the axis;

[0391] :along Unit vector in the direction of the axis;

[0392] :along Unit vector in the direction of the axis.

[0393] Step S1.4.3, model output:

[0394] The final model contains:

[0395] Geometric surface, i.e. triangular mesh ;

[0396] Vascular network, i.e., classification model ;

[0397] Dynamic posture parameters, quaternions With the rotation matrix .

[0398] Step S2: Based on the 3D model of the driver's head, physiological indicators such as eyelid closure, head center of gravity offset, mouth corner position offset, frontal lobe blood flow, and tongue status are detected to obtain the driver's physiological indicators;

[0399] Step S2.1: Eyelid closure Detection,

[0400] 3D keypoint detection:

[0401] Locate the upper eyelid point With lower eyelid point is the pupil center and The intersection of the vertical line and the upper and lower eyelids,

[0402] Closure calculation :

[0403] ;

[0404] Reference distance when eyes are open :

[0405] ;

[0406] Reference distance when eyes are closed :

[0407] ;

[0408] in Upper eyelid point , Lower eyelid point ;

[0409] : Real-time three-dimensional coordinates of upper eyelid points;

[0410] : Real-time 3D coordinates of lower eyelid points;

[0411] : When the eyes are fully open, the upper eyelid is pointed coordinate;

[0412] : Lower eyelid point when eyes are fully open coordinate;

[0413] : Upper eyelid point when eyes are completely closed z coordinate;

[0414] : Lower eyelid point when eyes are completely closed z coordinate;

[0415] : baseline distance when eyes are open;

[0416] : Reference distance when eyes are closed;

[0417] Step S2.2: Mouth corner position offset Detection,

[0418] 3D keypoint extraction:

[0419] Locate the left and right corners of the mouth and , and the reference position of the left corner of the mouth in the resting state and the reference position of the right corner of the mouth in the resting state ;

[0420] Dynamic offset :

[0421] ;

[0422] : Left corner of mouth ;

[0423] : Right corner of mouth ;

[0424] : Real-time three-dimensional coordinates of the left corner of the mouth;

[0425] : Real-time three-dimensional coordinates of the right mouth corner;

[0426] :Right now , the reference position of the left corner of the mouth in the resting state;

[0427] :Right now , the reference position of the right corner of the mouth in the resting state;

[0428] : three-dimensional coordinates of the left corner of the mouth in the resting state;

[0429] : three-dimensional coordinates of the right mouth corner in the resting state;

[0430] Step S2.3: Head center of gravity shift ,

[0431] Real-time center of gravity of the head calculate:

[0432] ;

[0433] : Number of collected point clouds;

[0434] , , : No. The real-time coordinates of each point in the global coordinate system;

[0435] offset :

[0436] ;

[0437] : Real-time center of gravity coordinates of the head;

[0438] : Coordinates of the center of gravity of the head in the initial state;

[0439] Step S2.4: Prefrontal blood flow Detection,

[0440] NIRS (Near Infrared Spectrometer) Sensor Deployment:

[0441] Based on the 3D model of the driver's head, the driver is advised to place the NIRS sensor in the area with dense blood vessels;

[0442] blood flow Estimate:

[0443] ;

[0444] : Change in oxygenated hemoglobin concentration (measured by NIRS sensor, unit: );

[0445] : Resting oxygenated hemoglobin baseline concentration (unit: );

[0446] : calibration coefficient;

[0447] Step S2.5, tongue status ( 、 、 、 、 ) detection,

[0448] The oral area is 3D modeled separately, using a near-infrared structured light camera installed above the dashboard with a wavelength of 850 mm to penetrate the lip surface and collect dynamic point cloud data of the lip at 30 frames per second. :

[0449] ;

[0450] : No. The coordinates of the points;

[0451] : The total number of lip dynamic point cloud data collected;

[0452] Use the U-Net network to perform semantic segmentation of the oral cavity and extract the coordinates of the tongue tip ;

[0453] Three-dimensional displacement trajectory analysis:

[0454] Displacement amplitude The reference position relative to the tip of the tongue The instantaneous displacement of:

[0455] ;

[0456] : displacement amplitude of tongue tip;

[0457] : Coordinates of the tongue tip at the moment;

[0458] : tongue tip reference position;

[0459] Yawning is large and continuous. , the amplitude is small and periodic when speaking, ;

[0460] Movement direction (calculate the angle with the reference axis):

[0461] For continuous K Tongue tip coordinates of the frame Perform polynomial fitting to obtain the motion trajectory equation:

[0462] ;

[0463] in 、 、 is the trajectory equation parameter, which is a three-dimensional vector;

[0464] : trajectory motion equation;

[0465] Calculate the instantaneous motion direction vector :

[0466] ;

[0467] In the global coordinate system, calculate the vertical axis Angle :

[0468] ;

[0469] : The angle between the instantaneous motion direction vector and the vertical axis;

[0470] : instantaneous motion direction vector;

[0471] Yawning: mainly vertical downward (mouth expansion), ;

[0472] Speaking: Mainly horizontal swing, ;

[0473] Instantaneous velocity and acceleration:

[0474] Instantaneous speed :

[0475] ;

[0476] : The coordinates of the tip of the tongue at that moment;

[0477] : The coordinates of the tip of the tongue at the moment ;

[0478] Yawning: low speed, , showing a trend of slow rise and slow fall;

[0479] Speak: high speed, , accompanied by short-duration pulses;

[0480] acceleration :

[0481] ;

[0482] : The instantaneous velocity of the tongue tip at that moment;

[0483] : The instantaneous velocity of the tongue tip at that moment;

[0484] Yawning: the absolute value of acceleration is small, ;

[0485] Speech: The absolute value of acceleration changes dramatically. ;

[0486] Motion trajectory complexity:

[0487] The complexity is quantified by calculating the approximate entropy of the tongue movement path:

[0488] Parameter setting: Embedding dimension , tolerance ;

[0489] Constructing a time series vector :

[0490] ;

[0491] : No. The displacement amplitude of the tongue tip at each time point (in millimeters);

[0492] : No. The displacement amplitude of the tongue tip at each time point (in millimeters);

[0493] : index of time series, range in is the total data length, is the embedding dimension;

[0494] Statistical similarity vector ratio :

[0495] ;

[0496] : Tolerance is the threshold for judging whether two time series vectors are similar;

[0497] : Within tolerance Inside, with vectors Similar proportions, is a time series vector;

[0498] : total data length;

[0499] : embedding dimension;

[0500] : index of time series, range in is the total data length, is the embedding dimension, Indicates the vectors;

[0501] Calculating entropy :

[0502] ;

[0503] :The embedding dimension is The ratio of similar vectors when ;

[0504] : Complexity of tongue movement trajectory, i.e. entropy value ;

[0505] Yawn: Simple trajectory, low value, ;

[0506] Speech: Complex trajectory, high value, .

[0507] Step S3: defining an input tensor, wherein the input is time series data containing multimodal physiological indicators, and constructing a one-dimensional convolutional neural network for processing multimodal time series data;

[0508] Step S3.1, input data definition,

[0509] The input is time series data containing multimodal physiological indicators, each time window contains sampling points (e.g. Y=60, corresponding to 5 seconds of data, sampling frequency is 12Hz), feature dimension is 9;

[0510] Input Tensor ,like Figure 3 As shown:

[0511] Input Example The matrix:

[0512] ;

[0513] : Sampling point The degree of eyelid closure at ;

[0514] : Sampling point The mouth corner position shifts when ;

[0515] : Sampling point The head center of gravity shifts when ;

[0516] : Sampling point Prefrontal blood flow, ;

[0517] : Sampling point The tongue displacement amplitude at ;

[0518] : Sampling point The direction of tongue movement, ;

[0519] : Sampling point The tongue movement speed, ;

[0520] : Sampling point The tongue acceleration at ;

[0521] : Sampling point The complexity of tongue movement trajectory, ;

[0522] Step S3.2, Convolutional Neural Network (CNN) architecture,

[0523] The present invention adopts a one-dimensional convolutional neural network (1D-CNN) to process multimodal time series data. The network architecture receives a feature sequence with a length of 60 time steps and a dimension of 9 from the input layer, and extracts spatiotemporal features through two layers of convolution and pooling modules. The first layer uses 64 convolution kernels of size 5 to slide along the time dimension, and captures local temporal patterns (such as the gradual process of eyelid closure or a sudden drop in blood flow) through the ReLU activation function, followed by a maximum pooling with a step size of 2 to compress the feature dimension and enhance translation invariance; the second layer further stacks 128 large The convolution kernel is as small as 3, and cross-modal associations are integrated at a higher level of abstraction (such as the coordinated changes in the center of gravity offset of the head and the movement of the tongue). After pooling again, high-level spatiotemporal features are obtained. The feature map is flattened and then connected to the fully connected layer. Global feature integration is first performed through a 256-dimensional hidden layer. Finally, the 3-neuron output layer is combined with the Softmax function to generate the probability distribution of wakefulness, mild fatigue and severe fatigue. The entire network gradually expands the perception through hierarchical progressive convolution operations to extract multi-scale temporal patterns. The pooling operation reduces the impact of local fluctuations, and finally realizes end-to-end fatigue state mapping. Figure 4 As shown:

[0524] in:

[0525] Convolutional layer 1, namely:

[0526] ;

[0527] : Convolutional layer 1;

[0528] : No. The convolution kernel is Input channel, The weight of the time step;

[0529] : No. The bias value of the convolution kernel;

[0530] : input tensor;

[0531] : convolution kernel index, range , indicating that there are 64 convolution kernels in the first layer;

[0532] : Input feature channel index, range , corresponding to 9 input feature dimensions;

[0533] : Output feature map time step index, input time window length is 60, output dimension is ;

[0534] Max Pooling Layer 1:

[0535] ;

[0536] : Maximum pooling layer 1;

[0537] : The first output feature map of convolutional layer 1 channels at time step The value of

[0538] : The first output feature map of convolutional layer 1 channels at time step The value of

[0539] : The time step index after pooling, the pooling step size is 2, and the output dimension is ;

[0540] Convolutional layer 2:

[0541] ;

[0542] : Convolutional layer 2;

[0543] : No. The convolution kernel is Input channel, The weight of the time step;

[0544] : The first output of pooling layer 1 channels at time step The value of

[0545] : convolution kernel index, range , indicating that there are 128 convolution kernels in the first layer;

[0546] : Input feature channel index, range , corresponding to 64 output channels of layer 1;

[0547] : The time step index of the output feature map, the output dimension is ;

[0548] Max Pooling Layer 2:

[0549] ;

[0550] : Maximum pooling layer 2;

[0551] : The first output feature map of convolutional layer 2 channels at time step The value of

[0552] : The first output feature map of convolutional layer 2 channels at time step The value of

[0553] : The time step index after pooling, the pooling step size is 2, and the output dimension is ;

[0554] Fully connected layer 1:

[0555] ;

[0556] : Fully connected layer 1;

[0557] : The weight matrix of the fully connected layer 1, with a dimension of ;

[0558] : Output of pooling layer 2 The dimension is , the length of the flattened vector is 1920;

[0559] : The bias term of the fully connected layer 1, with a dimension of 256;

[0560] Fully connected layer 2:

[0561] ;

[0562] : Fully connected layer 2;

[0563] : The weight matrix of the fully connected 2 layers, the dimension is ;

[0564] : The output vector of the fully connected layer 1, with a dimension of 256;

[0565] : Bias term of fully connected layer 2, dimension is 3.

[0566] Step S4, output layer design: design the output layer based on the input tensor and the convolutional neural network input layer to obtain the probability vector of the driver's fatigue driving state and determine the driver's fatigue driving degree;

[0567] Step S4.1, output layer design,

[0568] The output layer is designed as a fully connected layer containing 3 neurons, corresponding to the three states of the driver: awake, mildly fatigued, and severely fatigued. Each neuron converts the linear output of the previous layer into a normalized probability value through the Softmax activation function, indicating the confidence level of the occurrence of that state. Specifically, the original output value (logits) of the last layer of the network is transformed by Softmax to generate a three-element probability vector, in which each element represents the predicted probability of the corresponding category, and the sum of all probabilities equals 1. In actual applications, the system will select the category with the highest probability value as the final judgment result. For example, if the output probability vector is [0.05, 0.25, 0.70], the driver is judged to be in a severely fatigued state. To further improve reliability, a probability threshold is set (for example, the highest probability must exceed 0.7). If the probabilities of all categories are lower than the set probability threshold, they are marked as uncertain and the multimodal data review mechanism module is activated for review. This design not only ensures the intuitiveness of classification decisions, but also quantifies the confidence level of judgments through probability, providing an explainable decision-making basis for subsequent graded warnings (such as sound prompts, seat vibrations, or forced parking).

[0569] Output dimension: 3D probability vector , corresponding to three types of fatigue states;

[0570] :wide awake, ;

[0571] : Mild fatigue, ;

[0572] : Severe fatigue, ;

[0573] : Karolinska Sleepiness Scale;

[0574] Softmax formula:

[0575] ;

[0576] , is the original output value of the fully connected layer 2, and e is a natural constant;

[0577] Step S4.2, judgment rules,

[0578] Direct classification:

[0579] Take the category with the highest probability as the prediction result:

[0580]

[0581] : The classification result of the driver's current fatigue level is a discrete category label with values ​​of 0, 1, and 2;

[0582] : The traversed variable takes values ​​0, 1, and 2. It is used to compare the probability values ​​of the three categories and return the index of the category with the highest probability.

[0583] Step S5: Control the vehicle of the driver who is detected to be in a fatigue driving state, and implement a corresponding vehicle control strategy based on the obtained driver's fatigue driving level;

[0584] Step S5.1: Based on the detected driver fatigue level, adopt the following vehicle control strategy:

[0585] When the driver is slightly fatigued, a low-frequency voice broadcast is activated to remind the driver to stop and rest, reduce the throttle opening to 4 / 5 of the current opening, and adjust the throttle opening according to the fatigue probability value. Dynamically adjust the throttle response curve to lower the acceleration limit:

[0586] ;

[0587] is the current maximum acceleration of the vehicle, is the original maximum acceleration of the vehicle, : Mild fatigue, ;

[0588] When severe driver fatigue is detected, a high-frequency in-car voice broadcast is activated to remind the driver to stop and rest, the lights are controlled to flash to remind other drivers to take shelter, the throttle opening is reduced to 5 / 6 of the current opening, the automatic transmission is locked, downshifting and acceleration are prohibited, and the engine output torque is directly limited to a safe threshold:

[0589] ;

[0590] in is the current torque of the engine, is the rated torque of the engine, The current engine output torque safety threshold.

[0591] In the specific implementation process,

[0592] The driver's initial head 3D model is obtained through a depth camera and a portable ultrasound probe, and the driver's subsequent real-time head external contour point cloud data is obtained.

[0593] Based on the driver's real-time head external contour point cloud data and the time window and sampling frequency set in the neural network, the driver's current physiological indicators are calculated.

[0594] After the calculation is completed, the driver's physiological indicator matrix is ​​input into the neural network, and the neural network outputs a 3D probability vector of the driver's fatigue judgment.

[0595] The driver's current fatigue level is judged through the 3D probability vector, and the corresponding vehicle control strategy is adopted according to the driver's current fatigue level.

[0596] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for detecting driver fatigue driving status based on 3D head modeling, characterized in that: The following steps are involved: Step S1: The depth camera obtains the external contour point cloud of the driver's head, and the portable ultrasound probe scans the location of the driver's blood vessels to obtain a 3D model of the driver's head; Step S2: Based on the 3D model of the driver's head, physiological indicators such as eyelid closure, mouth corner position deviation, head center of gravity deviation, frontal lobe blood flow, and tongue status are detected to obtain the driver's physiological indicators; Step S3: defining an input tensor, wherein the input is time series data containing multimodal physiological indicators, and constructing a one-dimensional convolutional neural network for processing multimodal time series data; Step S4, output layer design: design the output layer based on the input tensor and the input layer of the convolutional neural network (CNN) architecture to obtain a probability vector of the driver's fatigue driving state and determine the driver's fatigue driving degree; Step S5: Control the vehicle of the driver who is detected to be in a fatigue driving state, and implement a corresponding vehicle control strategy based on the obtained driver's fatigue driving level.

2. The method for detecting driver fatigue driving status based on 3D head modeling according to claim 1, characterized in that: Step S1 also includes the following steps: Step S1.1: Driver head data collection, Step S1.1.1: External contour modeling: Use a depth camera to obtain 3D point cloud data of the driver's head surface. The calculation formula is: ； : Driver’s head surface point cloud data; : Number of collected point clouds; : No. The three-dimensional coordinates of a point in the global coordinate system; Use Delaunay triangulation algorithm to convert point cloud data into continuous surfaces : ； : a set of vertices, : No. vertices, : number of vertices, :vertex coordinates of : a collection of triangles, : No. One-sided piece, : number of patches, A collection from vertices , is a triangular patch The vertex of Step S1.1.2, internal structure mapping: Install a portable ultrasound probe on the headrest of the driver's seat to perform ultrasound testing on the driver's head to obtain the blood vessel distribution of the driver's head; Step S1.1.2.1, blood vessel centerline extraction: The vascular structure module is enhanced using a three-dimensional Hessian matrix filter to extract the centerline coordinates: ； : The coordinates of the midpoint of the blood vessel centerline and the set of blood vessel radius; , , : No. The three-dimensional coordinates of the midpoint of the segment vessel centerline; : No. The radius of the segmental vessel; : total number of vascular segments; Step S1.1.2.2: Construction of vascular fractal model: Based on fractal dimension Describe the complexity of vascular branching: ； : Radius is less than The number of blood vessels; : fractal dimension; Step S1.1.2.3, Inertial Unit Measurement: The head posture is measured by the IMU sensor, and the rotation is represented by the posture quaternion: Attitude quaternion : ； : Real part, indicating the rotation angle; , , : Imaginary part, indicating the direction of the rotation axis; :along The unit vector in the positive direction of the axis; :along The unit vector in the positive direction of the axis; :along The unit vector in the positive direction of the axis; Step S1.2, coordinate system alignment: Step S1.2.1, global coordinate system definition: Head center of gravity As the origin, the vertical direction is the vertical axis The direction of the axis, The positive direction of the axis points to the roof, perpendicular to The axis points directly in front of the driver's face as the front and rear axis The positive direction of the axis is perpendicular to Axis and The plane formed by the axes is the left and right axes The direction of the axis, The positive direction of the axis points to the right, defining the global three-dimensional rectangular coordinate system : ； : Coordinates of the center of gravity of the head in the initial state; : Number of collected point clouds; Step S1.2.2, sensor external parameter calibration: Align the coordinate systems of the depth camera, portable ultrasound probe, and IMU sensor to the global coordinate system: Each transformation matrix is ​​of the form: ； : Rotation matrix; : translation vector; Step S1.3, point cloud preprocessing, Step S1.3.1, outlier filtering: Calculate the distance and standard deviation of all points in the neighborhood of each point. If the distance between a point in the neighborhood and the point exceeds the threshold, it is considered an outlier and removed. Neighborhood radius Average distance of inliers and standard deviation : ； ； :Point cloud points The neighborhood set of In radius Points within millimeters; : the number of points in the neighborhood; :To be evaluated 3D point coordinates; : Neighborhood The 3D point coordinates; The threshold is set as: ； Step S1.3.2, voxel grid downsampling: By evenly dividing the point cloud into voxels; Downsampled point cloud collection for: ； :To be evaluated 3D point coordinates; : No. The set of points within a voxel; : the number of points after downsampling; Step S1.4, model fusion and final output, Step S1.4.1: Fusion of triangulated mesh surface and blood vessels: The blood vessel centerline data Mapping to the triangular mesh surface to obtain a 3D head model including blood vessel distribution , guiding NIRS sensor placement: ； : The coordinates of the midpoint of the blood vessel centerline and the set of blood vessel radius; , , : No. 3D coordinates of the segmented vessel centerline; : No. The radius of the segmental vessel; Step S1.4.2, head dynamic posture compensation: pass Rotation Matrix Update the model pose in real time: ； : Quaternion representation, describing the rotation joint lock problem of the head in three-dimensional space; : Real part, indicating the rotation angle; , , : Imaginary part, indicating the direction of the rotation axis; :along Unit vector in the direction of the axis; :along Unit vector in the direction of the axis; :along Unit vector in the direction of the axis; Step S1.4.3, model output: The final model contains: Geometric surface, i.e. triangular mesh ; Vascular network, i.e., classification model ; Dynamic posture parameters, quaternions With the rotation matrix .

3. The method for detecting driver fatigue driving status based on 3D head modeling according to claim 1, characterized in that: Step S2 also includes the following steps: Step S2.1: Eyelid closure Detection, 3D keypoint detection: Locate the upper eyelid point With lower eyelid point is the pupil center and The intersection of the vertical line and the upper and lower eyelids; Closure calculation : ； Reference distance when eyes are open : ； Reference distance when eyes are closed : ； in Upper eyelid point , Lower eyelid point ; : Real-time three-dimensional coordinates of upper eyelid points; : Real-time 3D coordinates of lower eyelid points; : When the eyes are fully open, the upper eyelid is pointed coordinate; : Lower eyelid point when eyes are fully open coordinate; : Upper eyelid point when eyes are completely closed z coordinate; : Lower eyelid point when eyes are completely closed z coordinate; : baseline distance when eyes are open; : Reference distance when eyes are closed; Step S2.2: Mouth corner position offset Detection, 3D keypoint extraction: Locate the left and right corners of the mouth and , and the reference position of the left corner of the mouth in the resting state and the reference position of the right corner of the mouth in the resting state ; Dynamic offset : ； : Left corner of mouth ; : Right corner of mouth ; : Real-time three-dimensional coordinates of the left corner of the mouth; : Real-time three-dimensional coordinates of the right mouth corner; :Right now , the reference position of the left corner of the mouth in the resting state; :Right now , the reference position of the right corner of the mouth in the resting state; : three-dimensional coordinates of the left corner of the mouth in the resting state; : three-dimensional coordinates of the right mouth corner in the resting state; Step S2.3: Head center of gravity shift , Real-time center of gravity of the head calculate: ； : Number of collected point clouds; , , : No. The real-time coordinates of each point in the global coordinate system; offset : ； : Real-time center of gravity coordinates of the head; : Coordinates of the center of gravity of the head in the initial state; Step S2.4: Prefrontal blood flow Detection, NIRS sensor deployment: Based on the 3D model of the driver's head, the driver is advised to place the NIRS sensor in the area with dense blood vessels; blood flow Estimate: ； : Change in oxygenated hemoglobin concentration; : baseline concentration of oxygenated hemoglobin at rest; : calibration coefficient; Step S2.5, tongue status ( 、 、 、 、 ) detection, The oral area is 3D modeled separately, using a near-infrared structured light camera installed above the dashboard with a wavelength of 850 mm to penetrate the lip surface and collect dynamic point cloud data of the lip at 30 frames per second. : ； : No. The coordinates of the points; : The total number of lip dynamic point cloud data collected; Use the U-Net network to perform semantic segmentation of the oral cavity and extract the coordinates of the tongue tip ; Three-dimensional displacement trajectory analysis: Displacement amplitude The reference position relative to the tip of the tongue The instantaneous displacement of: ； : displacement amplitude of tongue tip; : Coordinates of the tongue tip at the moment; : tongue tip reference position; Yawning is large and continuous. , the amplitude is small and periodic when speaking, ; Direction of movement: For continuous K Tongue tip coordinates of the frame Perform polynomial fitting to obtain the motion trajectory equation: ； in 、 、 is the trajectory equation parameter, which is a three-dimensional vector; : trajectory motion equation; Calculate the instantaneous motion direction vector : ； In the global coordinate system, calculate the vertical axis Angle : ； : The angle between the instantaneous motion direction vector and the vertical axis; : instantaneous motion direction vector; Yawning: mainly vertically downward, ; Speaking: Mainly horizontal swing, ; Instantaneous velocity and acceleration: Instantaneous speed : ； : The coordinates of the tip of the tongue at that moment; : The coordinates of the tip of the tongue at the moment ; Yawning: low speed, , showing a trend of slow rise and slow fall; Speech: high speed, , accompanied by short-duration pulses; acceleration : ； : The instantaneous velocity of the tongue tip at that moment; : The instantaneous velocity of the tongue tip at that moment; Yawning: the absolute value of acceleration is small, ; Speech: The absolute value of acceleration changes dramatically. ; Motion trajectory complexity: The complexity is quantified by calculating the approximate entropy of the tongue movement path: Parameter setting: Embedding dimension , tolerance ; Constructing a time series vector : ； : No. The displacement amplitude of the tongue tip at each time point; : No. The displacement amplitude of the tongue tip at each time point; : index of time series, range in is the total data length, is the embedding dimension; Statistical similarity vector ratio : ； : Tolerance is the threshold for judging whether two time series vectors are similar; : Within tolerance Inside, with vectors Similar proportions, is a time series vector; : total data length; : embedding dimension; : index of time series, range in is the total data length, is the embedding dimension, Indicates the vectors; Calculating entropy : ； :The embedding dimension is The ratio of similar vectors when ; : Complexity of tongue movement trajectory, i.e. entropy value ; Yawn: Simple trajectory, low value, ; Speech: Complex trajectory, high value, .

4. The method for detecting driver fatigue driving status based on 3D head modeling according to claim 1, characterized in that: Step S3 also includes the following steps: Step S3.1, input data definition, The input is time series data containing multimodal physiological indicators, each time window contains sampling points, the feature dimension is 9; Input Tensor ; Input Example The matrix: ； : Sampling point The degree of eyelid closure at ; : Sampling point The mouth corner position shifts when ; : Sampling point The head center of gravity shifts when ; : Sampling point Prefrontal blood flow, ; : Sampling point The tongue displacement amplitude at ; : Sampling point The direction of tongue movement, ; : Sampling point The tongue movement speed, ; : Sampling point The tongue acceleration at ; : Sampling point The complexity of tongue movement trajectory, ; Step S3.2, Convolutional Neural Network (CNN) architecture, A one-dimensional convolutional neural network was used to process multimodal time series data. The network architecture starts with the input layer, receiving a feature sequence of 60 time steps and 9 dimensions. It then extracts spatiotemporal features through two layers of convolution and pooling modules. The first layer uses 64 convolution kernels of size 5 to slide along the time dimension, capturing local temporal patterns through the ReLU activation function. This is followed by maximum pooling with a stride of 2 to compress the feature dimension and enhance translation invariance. The second layer further stacks 128 convolution kernels of size 3 to fuse cross-modal associations at a higher level of abstraction. After further pooling, high-level spatiotemporal features are obtained. The feature maps are flattened and then connected to the fully connected layer. Global feature integration is first performed through a 256-dimensional hidden layer. Finally, a three-neuron output layer combined with a Softmax function generates the probability distribution of wakefulness, mild fatigue, and severe fatigue. Among them, the convolution layer 1 is: ； : Convolutional layer 1; : No. The convolution kernel is Input channel, The weight of the time step; : No. The bias value of the convolution kernel; : input tensor; : convolution kernel index, range , indicating that there are 64 convolution kernels in the first layer; : Input feature channel index, range , corresponding to 9 input feature dimensions; : Output feature map time step index, input time window length is 60, output dimension is ; Max Pooling Layer 1: ； : Maximum pooling layer 1; : The first output feature map of convolutional layer 1 channels at time step The value of : The first output feature map of convolutional layer 1 channels at time step The value of : The time step index after pooling, the pooling step size is 2, and the output dimension is ; Convolutional layer 2: ； : Convolutional layer 2; : No. The convolution kernel is Input channel, The weight of the time step; : The first output of pooling layer 1 channels at time step The value of : convolution kernel index, range , indicating that there are 128 convolution kernels in the first layer; : Input feature channel index, range , corresponding to 64 output channels of layer 1; : The time step index of the output feature map, the output dimension is ; Max Pooling Layer 2: ； : Maximum pooling layer 2; : The first output feature map of convolutional layer 2 channels at time step The value of : The first output feature map of convolutional layer 2 channels at time step The value of : The time step index after pooling, the pooling step size is 2, and the output dimension is ; Fully connected layer 1: ； : Fully connected layer 1; : The weight matrix of the fully connected layer 1, with a dimension of ; : Output of pooling layer 2 The dimension is , the length of the flattened vector is 1920; : The bias term of the fully connected layer 1, with a dimension of 256; Fully connected layer 2: ； : Fully connected layer 2; : The weight matrix of the fully connected 2 layers, the dimension is ; : The output vector of the fully connected layer 1, with a dimension of 256; : Bias term of fully connected layer 2, dimension is 3.

5. The method for detecting driver fatigue driving status based on 3D head modeling according to claim 1, characterized in that: Step S4 also includes the following steps: Step S4.1, output layer design, The output layer is designed as a fully connected layer containing three neurons, corresponding to the three states of the driver: alert, mildly fatigued, and severely fatigued. Each neuron uses the Softmax activation function to convert the linear output of the previous layer into a normalized probability value, indicating the confidence level of the state. Specifically, the raw output value of the last layer of the network is transformed by Softmax to generate a three-element probability vector, where each element represents the predicted probability of the corresponding category, and the sum of all probabilities equals 1. By setting a probability threshold, if the probabilities of all categories are lower than the set probability threshold, the state is marked as uncertain and the multimodal data review mechanism module is activated for review. Output dimension: 3D probability vector , corresponding to three types of fatigue states; :wide awake, ; : Mild fatigue, ; : Severe fatigue, ; : Karolinska Sleepiness Scale; Softmax formula: ； , is the original output value of the fully connected layer 2, and e is a natural constant; Step S4.2, judgment rules, Direct classification: Take the category with the highest probability as the prediction result: （ ）； : The classification result of the driver's current fatigue level is a discrete category label with values ​​of 0, 1, and 2; : The traversed variable takes values ​​0, 1, and 2. It is used to compare the probability values ​​of the three categories and return the index of the category with the highest probability.

6. The method for detecting driver fatigue driving status based on 3D head modeling according to claim 1, characterized in that: Step S5 also includes the following steps: Step S5.1: Based on the detected driver fatigue level, adopt the following vehicle control strategy: When the driver is slightly fatigued, a low-frequency voice broadcast is activated to remind the driver to stop and rest, reduce the throttle opening to 4 / 5 of the current opening, and adjust the throttle opening according to the fatigue probability value. Dynamically adjust the throttle response curve to lower the acceleration limit: ； is the current maximum acceleration of the vehicle, is the original maximum acceleration of the vehicle, : Mild fatigue, ; When severe driver fatigue is detected, a high-frequency in-car voice broadcast is activated to remind the driver to stop and rest, the lights are controlled to flash to remind other drivers to take shelter, the throttle opening is reduced to 5 / 6 of the current opening, the automatic transmission is locked, downshifting and acceleration are prohibited, and the engine output torque is directly limited to a safe threshold: ； in is the current torque of the engine, is the rated torque of the engine, The current engine output torque safety threshold.

Images