Method for detecting fatigue driving state of driver based on head 3D modeling

Abstract

The invention belongs to the technical field of vehicle driving control, and relates to a driver fatigue driving state detection method based on head 3D modeling, which comprises the following steps: S1, a depth camera obtains the external contour point cloud of the head of a driver, and a portable ultrasonic probe scans the blood vessel position of the driver to obtain the head 3D modeling of the driver; s2, according to 3D modeling of the head of the driver, physiological index detection of the driver is obtained; s3, defining an input tensor, and constructing a one-dimensional convolutional neural network for processing the multi-modal time sequence data; s4, designing an output layer, and judging the fatigue driving degree of the driver; and S5, implementing a corresponding vehicle control strategy according to the obtained fatigue driving degree of the driver. The fatigue degree score of the driver is calculated through the convolutional neural network, whether the driver is in fatigue driving or not is judged, and after the driver is judged to be in the fatigue driving state, the driver is reminded to pay attention to rest and other vehicles are reminded to pay attention to avoidance through in-vehicle voice broadcasting and an out-vehicle reminding lamp.

Description

Technical Field

[0001] The present invention belongs to the technical field of vehicle driving control for detecting the fatigue state of drivers; specifically, it relates to a method for detecting the fatigue driving state of drivers based on 3D head modeling. Background Art

[0002] Drivers' fatigue driving is one of the main causes of traffic accidents. In a fatigued state, the physical functions and judgment abilities of drivers decrease significantly, specifically manifested as an increase in braking reaction time, distraction of attention, and a decrease in the dynamic perception ability of the road, etc. This will lead to a substantial increase in the probability of rear-end collisions and vehicle yaw. Therefore, it is crucial to detect the fatigue state of drivers in a timely and accurate manner. Through the detection of the fatigue driving state, the risk of traffic accidents caused by fatigue driving can be greatly reduced. When it is found that a driver shows signs of fatigue, the driver can be reminded to rest in time and other vehicles can be reminded to drive carefully to avoid traffic accidents, which can effectively protect the lives of road users, reduce the damage to vehicles and road infrastructure, and save the manpower and material resources consumed in accident rescue, accident handling, and post-accident recovery. In addition, continuously detecting the fatigue state helps to cultivate drivers' safety awareness and enables drivers to form the behavioral habit of consciously paying attention to their own state and actively preventing fatigue driving, thus fundamentally improving the road safety level.

[0003] Currently, the common indicators for detecting the fatigue driving state are: electroencephalogram signals, eye information, blood pressure, heart rate, facial information, head pitch angle, steering wheel information, speech rate, etc. The detection of the fatigue driving state usually does not use a single indicator only, but fuses the above-mentioned indicators to comprehensively judge whether a driver is fatigued. For example, Chinese Patent CN119479194A discloses a fatigue driving detection method based on facial recognition analysis and simultaneously considering data such as the heart rate monitoring data and blood oxygen monitoring data of drivers; Chinese Patent CN119478898A discloses a fatigue driving state monitoring method based on vital signs such as heart rate, respiratory rate, blood oxygen saturation, blood pressure, and heart rate variability; Chinese Patent CN118279878B discloses an intelligent recognition method for the fatigue state of on-vehicle drivers by fusing multi-mode physiological information. Considering various indicators comprehensively can reduce the misjudgment rate of the fatigue driving state, but it still fails to overcome the following defects of each indicator: (1) Electroencephalogram signals: Prone to environmental interference. Vibration during vehicle driving, electromagnetic waves of electronic devices, or even slight head shaking of the driver will cause a significant increase in signal noise.

[0004] (2) Eye information: Light or frequently changing lighting conditions may induce non-fatigue-related blinking actions, resulting in misjudgment.

[0005] (3) Blood pressure: Diet, emotion, low / high blood pressure, etc. have a greater interference on blood pressure, affecting the judgment of the fatigue driving state.

[0006] (4)Heart rate: Emotional fluctuations such as anxiety, tension or excitement can significantly affect the heart rate, leading to misjudgment of the fatigue state.

[0007] (5)Facial information: It is often obtained by using image processing technology, but the accuracy of this indicator depends on the clarity of the image. In a completely dark environment, strong light environment and when the vehicle is violently jolting, the image will inevitably become blurred, affecting the accuracy of the detection result. In addition, when the in-vehicle environment is complex and the driver's face is blocked, this indicator cannot be obtained.

[0008] (6)Head pitch angle: Like facial information, it cannot be obtained when the vehicle is jolting or there is occlusion.

[0009] (7)Steering wheel information: When driving on a bumpy road or turning, the natural jitter of the steering wheel may lead to misjudgment.

[0010] (8)Speech rate: The gap in speech rate and tone between individuals is relatively large, and the speech rate and tone of the same individual are also easily affected by emotions. Therefore, the accuracy of detecting the fatigue driving state based on this indicator is not high.

[0011] In summary, the existing fatigue driving state detection methods have deficiencies in terms of anti-interference, detection accuracy, environmental adaptability, etc. They cannot guarantee the detection accuracy in strong light, completely dark and vehicle jolting situations, and are prone to ignoring individual differences of drivers. Summary of the Invention

[0012] In view of the above problems, the purpose of the present invention is to provide a method for detecting the fatigue driving state of a driver based on head 3D modeling. A 3D model of the driver's head is established through a depth camera, an ultrasonic probe and an IMU, and the driver's head is modeled by a detector to obtain the center of gravity position of the driver's head, facial information, and blood flow in the prefrontal lobe. According to the above physiological information indicators of the driver, a convolutional neural network is used to calculate the fatigue degree score of the driver, determine whether the driver is fatigued, and after determining the fatigue driving state of the driver, the driver is reminded to take a rest through in-vehicle voice broadcast and an external warning light outside the vehicle, and other vehicles are reminded to avoid.

[0013] A method for detecting the fatigue driving state of a driver based on head 3D modeling provided by the present invention specifically includes the following steps: Step S1: The depth camera obtains the point cloud of the external contour of the driver's head, and the portable ultrasonic probe scans the blood vessel position of the driver to obtain the 3D modeling of the driver's head; Step S2: According to the 3D modeling of the driver's head, physiological index detections of the driver's eyelid closure degree, corner of the mouth position offset, head center of gravity offset, blood flow in the prefrontal lobe and tongue state are carried out to obtain the driver's physiological indexes; Step S3: Define the input tensor. The input is the time series data containing multi-modal physiological indicators, and construct a one-dimensional convolutional neural network for processing multi-modal time series data. Step S4: Output layer design. Design the output layer based on the input tensor and the input layer of the convolutional neural network CNN architecture to obtain the probability vector of the driver's fatigue driving state, and judge the degree of the driver's fatigue driving. Step S5: Control the vehicle of the driver detected to be in a fatigue driving state, and implement corresponding vehicle control strategies according to the obtained degree of the driver's fatigue driving.

[0014] As a preference of the present invention, the following steps are further included in step S1: Step S1.1: Driver's head data collection Step S1.1.1: External contour modeling: Use a depth camera to obtain the 3D point cloud data of the driver's head surface. Its calculation formula is: ; : Point cloud data of the driver's head surface; : Number of collected point clouds; : The -th point's three-dimensional coordinates in the global coordinate system; Adopt the Delaunay triangulation algorithm to convert the point cloud data into a continuous surface : ; : Set of vertices, : The -th vertex, : Number of vertices, : Coordinates of vertex ; : Set of triangular patches, : The -th patch, : Number of patches, comes from the set of vertices , and is the vertex of the triangular patch ; Step S1.1.2: Internal structure mapping: Install a portable ultrasonic probe on the headrest of the driver's seat to perform ultrasonic detection on the driver's head and obtain the blood vessel distribution of the driver's head. Step S1.1.2.1: Blood vessel centerline extraction: Adopt a three-dimensional Hessian matrix filtering and enhancing blood vessel structure module to extract the centerline coordinates: ; : The set of the midpoint coordinates of the blood vessel centerline and the blood vessel radius; , , : The three-dimensional coordinates of the midpoint of the centerline of the th blood vessel segment; : The radius of the th blood vessel segment; : The total number of blood vessel segments; Step S1.1.2.2, Construction of the blood vessel fractal model: Based on the fractal dimension to describe the complexity of blood vessel branching: ; The number of blood vessels with a radius less than ; : The fractal dimension; Step S1.1.2.3, Measurement of the inertial unit: Measure the head pose through the IMU sensor and represent the rotation using the attitude quaternion: The attitude quaternion : ; : The real part, representing the rotation angle; , , : The imaginary part, representing the direction of the rotation axis; : The unit vector along the positive direction of the axis; : The unit vector along the positive direction of the axis; : The unit vector along the positive direction of the axis; Step S1.2, Coordinate system alignment: Step S1.2.1, Definition of the global coordinate system: With the center of gravity of the head as the origin, with the vertical direction as the direction of the vertical axis axis, the positive direction of the axis points to the roof of the car, and perpendicular to the axis and pointing directly in front of the driver's face (Nose direction) is the positive direction of the front-back axis axis, and perpendicular to The plane formed by the axis and the axis is the left - right axis. The positive direction of the axis points to the right, defining the global three - dimensional rectangular coordinate system

[0015] : The coordinates of the center of gravity of the head in the initial state; : The number of point clouds collected; Step S1.2.2, Extrinsic calibration of the sensor: Align the coordinate systems of the depth camera, the portable ultrasonic probe, and the IMU sensor to the global coordinate system: The form of each transformation matrix is: ; : Rotation matrix; : Translation vector; Step S1.3, Point cloud pre - processing, Step S1.3.1, Outlier filtering: Calculate the distance sum and standard deviation of all points within the neighborhood of each point. If the distance between a certain point in the neighborhood and this point exceeds the threshold, it is judged as an outlier and removed; Neighborhood radius Average distance of inliers and standard deviation : ; ; : The th point in the point cloud 's neighborhood set, that is, all points within a radius of millimeters from ; : The number of points in the neighborhood; : The coordinates of the th three - dimensional point to be evaluated; : The coordinates of the th th three - dimensional point in the neighborhood The threshold is set to: ; Step S1.3.2, Voxel grid downsampling: By evenly dividing the point cloud into voxels; The downsampled point cloud set is: ; : The coordinates of the th three-dimensional point to be evaluated; : The set of points within the th voxel; : The number of points after downsampling; Step S1.4, Model fusion and final output, Step S1.4.1, Triangular mesh surface and blood vessel fusion: Map the blood vessel centerline data to the triangular mesh surface to obtain a 3D model of the head with blood vessel distribution , guiding the layout of NIRS sensors: ; : The set of the midpoint coordinates of the blood vessel centerline and the blood vessel radius; , , : The three-dimensional coordinates of the th segment of the blood vessel centerline; : The radius of the th segment of the blood vessel; Step S1.4.2, Head dynamic pose compensation: By the rotation matrix update the model pose in real time: ; : Quaternion representation, describing the rotation of the head in three-dimensional space to solve the gimbal lock problem; : The real part, representing the rotation angle; , , : The imaginary part, representing the rotation axis direction; : The unit vector along the axis direction; : The unit vector along the axis direction; : The unit vector along the axis direction; Step S1.4.3, Model Output: The final model includes: A geometric surface, i.e., a triangular mesh ; A blood vessel network, i.e., a classification model ; Dynamic pose parameters, i.e., quaternions and rotation matrices .

[0016] As an optimization of the present invention, the following steps are further included in step S2: Step S2.1, Eyelid closure degree Detection, 3D key point detection: Locate the upper eyelid point and the lower eyelid point as the intersection points of the straight line passing through the pupil center and perpendicular to the axis with the upper and lower eyelids; Closure degree calculation : ; The reference distance when the eyes are open : ; The reference distance when the eyes are closed : ; where is the upper eyelid point , is the lower eyelid point ; : Real-time three-dimensional coordinates of the upper eyelid point; : Real-time three-dimensional coordinates of the lower eyelid point; : Coordinates of the upper eyelid point when the eyes are fully open ; : Coordinates of the lower eyelid point when the eyes are fully open ; : Coordinates of the upper eyelid point when the eyes are fully closed z ; : Coordinates of the lower eyelid point when the eyes are fully closed z ; : Reference distance when the eyes are open; : Reference distance when the eyes are closed; Step S2.2, Offset of the corner of the mouth Detection, 3D key point extraction: Locate the left and right corner points of the mouth and , and the reference position of the left corner of the mouth at rest and the reference position of the right corner of the mouth at rest ; Dynamic offset : ; : Left corner point of the mouth ; : Right corner point of the mouth ; : Real-time three-dimensional coordinates of the left corner of the mouth; : Real-time three-dimensional coordinates of the right corner of the mouth; : That is , the reference position of the left corner of the mouth at rest; : That is , the reference position of the right corner of the mouth at rest; : Three-dimensional coordinates of the left corner of the mouth at rest; : Three-dimensional coordinates of the right corner of the mouth at rest; Step S2.3, Offset of the center of gravity of the head , Real-time center of gravity of the head Calculation: ; : Number of point clouds collected; , , : The th point's real-time coordinates in the global coordinate system; Offset : ; : Real-time center of gravity coordinates of the head; : Center of gravity coordinates of the head in the initial state; Step S2.4, Blood flow in the prefrontal lobe Detection, NIRS sensor deployment: Based on the 3D modeling of the driver's head, prompt the driver to deploy NIRS sensors in areas with dense blood vessel distribution; Blood flow Estimation: ; : Change in oxyhemoglobin concentration; : Baseline concentration of oxyhemoglobin at rest; : Calibration coefficient; Step S2.5, Tongue state( 、 、 、 、 ) Detection, Perform a separate 3D modeling of the oral cavity area, using a near-infrared structured light camera, installed above the dashboard, with a wavelength of 850 mm to penetrate the lip surface, and collect lip dynamic point cloud data at 30 frames per second : ; : The th point coordinate; : Total number of collected lip dynamic point cloud data; Use the U-Net network to perform semantic segmentation on the internal area of the oral cavity and extract the coordinates of the tip of the tongue ; Three-dimensional displacement trajectory analysis: Displacement amplitude is the instantaneous displacement relative to the reference position of the tip of the tongue : ; : Displacement amplitude of the tip of the tongue; : Coordinates of the tip of the tongue at time : Reference position of the tip of the tongue; The amplitude is large and continuously increasing during yawning, , and the amplitude is small and periodically fluctuating during speaking, ; Movement direction: For consecutive K frames of the coordinates of the tip of the tongue Perform polynomial fitting to obtain the motion trajectory equation: ; where 、 、 are the parameters of the trajectory equation and are three - dimensional vectors; : the trajectory motion equation; Calculate the instantaneous motion direction vector : ; In the global coordinate system, calculate the angle with the vertical axis : ; : the angle between the instantaneous motion direction vector and the vertical axis; : the instantaneous motion direction vector; Yawning: mainly downward vertically, ; Speaking: mainly horizontally swinging, ; Instantaneous velocity and acceleration: Instantaneous velocity : ; : the coordinates of the tip of the tongue body at time : the coordinates of the tip of the tongue body at time, taking ; Yawning: low speed, , showing a slow - rising and slow - falling trend; Speaking: high speed, , accompanied by short - time pulses; Acceleration : ; : the instantaneous velocity of the tip of the tongue body at time : the instantaneous velocity of the tip of the tongue body at time Yawning: the absolute value of acceleration is small, ; Speaking: the absolute value of acceleration changes violently, ; Complexity of movement trajectory: Quantify the complexity by calculating the approximate entropy of the tongue movement path: Parameter setting: Embedding dimension , tolerance ; Construct a time series vector : ; : Displacement amplitude of the tip of the tongue at the -th time point; : Displacement amplitude of the tip of the tongue at the -th time point; : Index of the time series, range where is the total data length, is the embedding dimension; Statistical proportion of similar vectors : ; : Threshold for the tolerance to judge whether two time series vectors are similar; : Proportion similar to the -th vector within the tolerance , is the time series vector; : Total data length; : Embedding dimension; : Index of the time series, range where is the total data length, is the embedding dimension, represents the -th vector; Calculate the entropy value : ; : Proportion of similar vectors when the embedding dimension is ; : Complexity of the tongue movement trajectory, i.e., the entropy value ; Yawning: The trajectory is simple, with a low value, ; Speech: Complex trajectory, high value, .

[0017] As a preferred embodiment of the present invention, step S3 also includes the following steps: Step S3.1, input data definition, The input is time series data containing multimodal physiological indicators, and each time window contains sampling points, the feature dimension is 9; Input Tensor ; Input Example The matrix: ; : Sampling point The degree of eyelid closure at ; : Sampling point The mouth corner position shifts when ; : Sampling point The head center of gravity shifts when ; : Sampling point Prefrontal blood flow at ; : Sampling point The tongue displacement amplitude at ; : Sampling point The direction of tongue movement, ; : Sampling point The tongue movement speed, ; : Sampling point The tongue movement acceleration at ; : Sampling point The complexity of the tongue movement trajectory at ; Step S3.2, Convolutional Neural Network (CNN) architecture, A one-dimensional convolutional neural network is used to process multi-modal time series data. The network architecture receives a feature sequence with a length of 60 time steps and a dimension of 9 from the input layer, and sequentially extracts spatio-temporal features through two convolutional and pooling modules. The first layer uses 64 convolutional kernels of size 5 to slide along the time dimension, captures local time patterns through the ReLU activation function, and then performs max pooling with a stride of 2 to compress the feature dimension and enhance translational invariance; the second layer further stacks 128 convolutional kernels of size 3 to fuse cross-modal associations at a higher abstraction level, and after pooling again, high-level spatio-temporal features are obtained. The feature map is flattened and then connected to a fully connected layer. First, global feature integration is performed through a 256-dimensional hidden layer, and finally, a 3-neuron output layer combines with the Softmax function to generate the probability distributions of wakefulness, mild fatigue, and severe fatigue. Among them, convolutional layer 1, that is: ; : Convolutional layer 1; : The th convolutional kernel's weight at the th input channel and the th time step; : The bias value of the th convolutional kernel; : Input tensor; : Convolutional kernel index, range , indicating that there are 64 convolutional kernels in the first layer; : Input feature channel index, range , corresponding to 9 input feature dimensions; : Time step index of the output feature map. The input time window length is 60, and the output dimension is ; Max pooling layer 1: ; : Max pooling layer 1; : The value of the th channel of the output feature map of convolutional layer 1 at time step ; : The value of the th channel of the output feature map of convolutional layer 1 at time step ; : Pooled time step index. The pooling stride is 2, and the output dimension is ; Convolution layer 2: ; : Convolution layer 2; : The weight of the th convolutional kernel at the input channel and the th time step; : The value of the th channel in the output of pooling layer 1 at time step ; : Convolutional kernel index, ranging from , indicating that there are 128 convolutional kernels in the first layer; : Input feature channel index, ranging from , corresponding to 64 output channels of the first layer; : Time step index of the output feature map, with an output dimension of ; Max pooling layer 2: ; : Max pooling layer 2; : The value of the th channel in the output feature map of convolution layer 2 at time step ; : The value of the th channel in the output feature map of convolution layer 2 at time step ; : Pooled time step index, with a pooling stride of 2 and an output dimension of ; Fully connected layer 1: ; : Fully connected layer 1; : Weight matrix of fully connected layer 1, with a dimension of ; : Output of pooling layer 2 with a dimension of , and the length of the flattened vector is 1920; : Bias term of fully connected layer 1, with a dimension of 256; Fully connected layer 2: ; : Fully connected layer 2; : Weight matrix of the second fully connected layer, with dimension ; : Output vector of the first fully connected layer, with dimension 256; : Bias term of the second fully connected layer, with dimension 3.

[0018] As a preference of the present invention, the following steps are further included in step S4: Step S4.1, Output layer design The output layer is designed as a fully connected layer containing 3 neurons, corresponding to three states of the driver, namely awake, mildly fatigued, and severely fatigued. Each neuron converts the linear output of the previous layer into a normalized probability value through the Softmax activation function, representing the confidence of the occurrence of this state. Specifically, it includes: after the original output value of the last layer of the network undergoes the Softmax transformation, a three-element probability vector is generated, where each element represents the prediction probability of the corresponding category, and the sum of all probabilities is equal to 1; by setting a probability threshold, if the probabilities of all categories are lower than the set probability threshold, it is marked as an uncertain state and the multimodal data review mechanism module is activated for review; Output dimension: 3D probability vector , corresponding to three types of fatigue states; : Awake, ; : Mildly fatigued, ; : Severely fatigued, ; : Karolinska Sleepiness Scale; Softmax formula: ; , is the original output value of the second fully connected layer, and e is the natural constant; Step S4.2, Judgment rule Direct classification: Take the category with the maximum probability as the prediction result:

[0019] : The classification result of the driver's current fatigue level, which is a discrete category label with values of 0, 1, and 2; : The variable to be traversed, with values of 0, 1, and 2, used to compare the probability values of three categories and return the category index with the highest probability.

[0020] As an optimization of the present invention, the following steps are further included in step S5: Step S5.1, according to the detected driver fatigue level, adopt the following vehicle control strategies: When it is detected that the driver is slightly fatigued, start a low-frequency in-vehicle voice broadcast to remind the driver to stop and rest, reduce the throttle opening to 4 / 5 of the current opening, and dynamically adjust the throttle response curve according to the fatigue probability value to reduce the upper limit of acceleration: ; is the current maximum acceleration of the vehicle, is the original maximum acceleration of the vehicle, : Slight fatigue, ; When it is detected that the driver is severely fatigued, start a high-frequency in-vehicle voice broadcast to remind the driver to stop and rest, control the vehicle lights to flash to remind other drivers to pay attention and avoid, reduce the throttle opening to 5 / 6 of the current opening, lock the automatic transmission, prohibit downshifting and accelerating, and directly limit the engine output torque to the safety threshold: ; Where is the current torque of the engine, is the rated torque of the engine, is the current engine output torque safety threshold.

[0021] The beneficial effects of the present invention are as follows: 1. For the collection of blood flow, the present invention solves the problem of uneven signal quality caused by differences in individual physiological characteristics such as the skin light transmittance and subcutaneous fat layer thickness of the driver. By using a depth camera and a portable ultrasonic probe, a 3D model of the driver's head is established and the blood vessel distribution of the driver is scanned, which helps the driver to more accurately locate the area with dense blood vessel distribution, improves the signal quality and detection accuracy, and uses a detector with less contact and softer material to reduce the interference to the driver and improve the detection accuracy.

[0022] 2. The present invention addresses the problems existing in traditional face recognition, and solves the problems of being unable to obtain the complete eyelid closure degree and the deviation of the mouth corner position under strong light and occlusion conditions. First, the head 3D modeling can work all-weather, is completely independent of the visible light environment, and can work in strong light, backlight and completely dark scenes. In addition, the head 3D modeling has better stability in complex scenes and can also work normally in typical scenes such as windshield reflection, the driver wearing sunglasses, and vehicle interior items occlusion. Finally, the head 3D modeling can also achieve head pose self-correction, dynamic compensation, and expression detection. For example, when the driver turns his head to observe the rearview mirror, the 3D model restores the frontal expression through geometric transformation, making up for the defect of misjudgment caused by perspective distortion in 2D images.

[0023] 3. The present invention reduces the misjudgment rate and increases the reliability of the results by introducing two new discriminant indicators, namely the blood flow in the prefrontal cortex and the tongue state, as the basis for detecting the fatigue driving state. Specifically, based on the fact that the prefrontal cortex is responsible for high-level cognitive functions such as decision-making and attention, its activity will decline during fatigue, and the state and position of the tongue are different when yawning due to fatigue and normal speaking. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] By referring to the following description in conjunction with the accompanying drawings, and with a more comprehensive understanding of the present invention, other objects and results of the present invention will become more apparent and easier to understand. In the drawings: Figure 1 is a schematic diagram of the global coordinate system of the present invention; Figure 2 is a schematic diagram of the upper and lower eyelids when the eyes are open (right) and closed (left) of the present invention; Figure 3 is a schematic diagram of the input data of the input tensor of the present invention; Figure 4 is a schematic diagram of the convolutional neural network structure of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0025] Refer to Figure 1-2 , and the following further describes the present invention in detail in conjunction with the accompanying drawings and specific embodiments.

[0026] The embodiment of the present invention provides a method for detecting the fatigue driving state of a driver based on head 3D modeling, including the following steps: Step S1. The depth camera obtains the point cloud of the external contour of the driver's head, and the portable ultrasonic probe scans the blood vessel position of the driver to obtain the 3D modeling of the driver's head; Step S1.1. Data collection of the driver's head, Step S1.1.1. External contour modeling: Using a depth camera, obtain the 3D point cloud data on the surface of the driver's head, and its calculation formula is: ; : Point cloud data on the surface of the driver's head; : Number of collected point clouds; : The three-dimensional coordinates of the Use the Delaunay triangulation algorithm to convert the point cloud data into a continuous surface : ; : Set of vertices, : The vertex, : Number of vertices, : Vertex coordinates; : Set of triangular patches, : The triangular patch, : Number of patches, from the set of vertices , which are the vertices of the triangular patch ; Step S1.1.2, Internal structure mapping: Install the portable ultrasonic probe on the headrest of the driver's seat, perform ultrasonic detection on the driver's head, and obtain the blood vessel distribution of the driver's head with the assistance of a standard anatomical atlas; Step S1.1.2.1, Blood vessel centerline extraction: Use a three-dimensional Hessian matrix filter to enhance the blood vessel structure module and extract the centerline coordinates: ; : Set of midpoint coordinates of the blood vessel centerline and blood vessel radii; , , : The three-dimensional coordinates (in millimeters) of the midpoint of the : The radius of the : Total number of blood vessel segments; Step S1.1.2.2, Blood vessel fractal model construction: Based on the fractal dimension to describe the complexity of blood vessel branching: ; : With a radius less than The number of blood vessels; : Fractal dimension (typical value ); A parameter describing the self-similarity and structural complexity of the blood vessel network, The larger the value, the more complex the blood vessel branches.

[0027] Step S1.1.2.3, Inertial unit measurement: Measure the head posture through the IMU sensor, and represent the rotation using the attitude quaternion: Attitude quaternion : ; : The real part, representing the rotation angle; , , : The imaginary part, representing the direction of the rotation axis; : Along the axis positive direction unit vector; : Along the axis positive direction unit vector; : Along the axis positive direction unit vector; Step S1.2, Coordinate system alignment: Step S1.2.1, Global coordinate system definition: Taking the center of gravity of the head as the origin, with the vertical direction as the vertical axis axis direction, axis positive direction points to the roof of the car, perpendicular to axis and pointing to the driver's face directly forward (Nose direction) as the front-rear axis axis positive direction, perpendicular to axis and axis forms a plane as the left-right axis axis direction, axis positive direction points to the right, defining the global three-dimensional rectangular coordinate system :

[0028] : The center of gravity coordinates of the head in the initial state (unit: millimeter); : The number of point clouds collected; Step S1.2.2, Sensor extrinsic parameter calibration: Align the coordinate systems of the depth camera, portable ultrasound probe, and IMU sensor to the global coordinate system: Each transformation matrix is in the form of: ; : Rotation matrix; : Translation vector (in millimeters); Step S1.3, Point cloud preprocessing, Step S1.3.1, Outlier filtering: Calculate the distance sum and standard deviation of all points within the neighborhood of each point. If the distance of a certain point within the neighborhood from this point exceeds the threshold, it is judged as an outlier and removed; Neighborhood radius within (here take millimeters) the average distance of points and standard deviation : ; ; : The th point in the point cloud, that is, all points within a radius of from millimeters; : The number of points within the neighborhood ( value changes dynamically, depending on the local point density); : The coordinates of the th 3D point to be evaluated (in millimeters); : The coordinates of the th 3D point in the neighborhood (in millimeters); The threshold is set to: ; Step S1.3.2, Voxel grid downsampling: Reduce the point cloud density and data volume by uniformly dividing the point cloud into voxels (cube grids); for example, the voxel size can be set to 2 millimeters, and only the center point or the mean point of each voxel is retained.

[0029] The downsampled point cloud set is: ; : The Three-dimensional point coordinates (in millimeters); : The point set within the ith voxel; : The number of points after downsampling; Step S1.4, Model Fusion and Final Output, Step S1.4.1, Triangular Mesh Surface and Vessel Fusion: Map the vessel centerline data onto the triangular mesh surface to obtain a 3D head model with vessel distribution , guiding the layout of NIRS (Near-Infrared Spectrometer) sensors: ; : The set of midpoint coordinates of the vessel centerline and vessel radii; , , : The three-dimensional coordinates (in millimeters) of the ith segment of the vessel centerline; : The radius (in millimeters) of the ith segment of the vessel; Step S1.4.2, Head Dynamic Pose Compensation: Real-time update the model pose through the rotation matrix : ; : Quaternion representation, describing the rotation of the head in three-dimensional space to avoid gimbal lock problems; : The real part, representing the rotation angle; , , : The imaginary part, representing the rotation axis direction; : The unit vector along the x-axis direction; : The unit vector along the y-axis direction; : The unit vector along the z-axis direction.

[0030] Step S1.4.3, Model Output: The final model includes: The geometric surface, i.e., the triangular mesh ; Vascular network, i.e., the classification model ; Dynamic pose parameters, i.e., quaternions and rotation matrices 。

[0031] Step S2: Based on the 3D modeling of the driver's head, detect the physiological indicators of the driver's eyelid closure degree, head center of gravity offset, mouth corner position offset, prefrontal blood flow, and tongue state to obtain the driver's physiological indicators; Step S2.1: Eyelid closure degree Detection, 3D key point detection: Locate the upper eyelid point and the lower eyelid point as the intersection points of the line passing through the pupil center and perpendicular to the axis with the upper and lower eyelids, Closure degree calculation : ; Reference distance when the eyes are open : ; Reference distance when the eyes are closed : ; Wherein i.e., the upper eyelid point , i.e., the lower eyelid point ; : Real-time three-dimensional coordinates of the upper eyelid point; : Real-time three-dimensional coordinates of the lower eyelid point; : Coordinates of the upper eyelid point when the eyes are fully open ; : Coordinates of the lower eyelid point when the eyes are fully open ; : Coordinates of the upper eyelid point when the eyes are fully closed z ; : Coordinates of the lower eyelid point when the eyes are fully closed z ; : Reference distance when the eyes are open; : Reference distance when the eyes are closed; Step S2.2: Mouth corner position offset Detection, 3D key point extraction: Locate the left and right mouth corner points and the left mouth corner reference position at rest and the right mouth corner reference position at rest ; Dynamic offset : ; : Left mouth corner point ; : Right mouth corner point ; : Real-time three-dimensional coordinates of the left mouth corner; : Real-time three-dimensional coordinates of the right mouth corner; : That is , the left mouth corner reference position at rest; : That is , the right mouth corner reference position at rest; : Three-dimensional coordinates of the left mouth corner at rest; : Three-dimensional coordinates of the right mouth corner at rest; Step S2.3, Head center of gravity offset , Head real-time center of gravity Calculation: ; : Number of point clouds collected; , , : The th point's real-time coordinates in the global coordinate system; Offset : ; : Head real-time center of gravity coordinates; : Head center of gravity coordinates in the initial state; Step S2.4, Prefrontal blood flow Detection, NIRS (Near-infrared spectroscopy) sensor deployment: Based on the 3D modeling of the driver's head, prompt the driver to deploy NIRS sensors in areas with dense blood vessel distribution; Blood flow volume Estimation: ; : Change in oxyhemoglobin concentration (measured by NIRS sensor, unit: ) : Baseline concentration of oxyhemoglobin at rest (unit: ) : Calibration coefficient Step S2.5, tongue state( 、 、 、 、 ) Detection Perform separate 3D modeling on the oral cavity area, using a near-infrared structured light camera, installed above the dashboard, with a wavelength of 850 mm to penetrate the lip surface, and collect lip dynamic point cloud data at 30 frames per second : ; : Coordinates of the th point : Total number of lip dynamic point cloud data collected Use the U-Net network to perform semantic segmentation on the internal oral cavity area and extract the coordinates of the tip of the tongue ; Three-dimensional displacement trajectory analysis: Displacement amplitude is the instantaneous displacement relative to the reference position of the tip of the tongue : ; : Displacement amplitude of the tip of the tongue : Coordinates of the tip of the tongue at time : Reference position of the tip of the tongue The amplitude is large and continuously increasing during yawning , and the amplitude is small and periodically fluctuating during speaking ; Movement direction (calculate the angle with the reference axis): For consecutive K coordinates of the tip of the tongue frames, perform polynomial fitting to obtain the motion trajectory equation: ; where 、 、 are the parameters of the trajectory equation, which are three-dimensional vectors; : the trajectory motion equation; Calculate the instantaneous motion direction vector : ; In the global coordinate system, calculate the angle with the vertical axis : ; : the angle between the instantaneous motion direction vector and the vertical axis; : the instantaneous motion direction vector; Yawning: mainly downward vertically (oral cavity expanding), ; Speaking: mainly horizontal swinging, ; Instantaneous velocity and acceleration: Instantaneous velocity : ; : the coordinates of the tip of the tongue at time : the coordinates of the tip of the tongue at time, taking ; Yawning: low speed, , showing a slow rising and falling trend; Speaking: high speed, , accompanied by short-time pulses; Acceleration : ; : the instantaneous velocity of the tip of the tongue at time : the instantaneous velocity of the tip of the tongue at time Yawning: the absolute value of the acceleration is small, ; Speech: The absolute value of acceleration changes violently, ; Complexity of the movement trajectory: Quantify the complexity by calculating the approximate entropy of the tongue movement path: Parameter setting: Embedding dimension , tolerance ; Construct a time series vector : ; : The displacement amplitude (in millimeters) of the tip of the tongue at the th time point; : The displacement amplitude (in millimeters) of the tip of the tongue at the th time point; : Index of the time series, range where is the total data length, is the embedding dimension; Statistical proportion of similar vectors : ; : Threshold for the tolerance to judge whether two time series vectors are similar; : Within the tolerance , the proportion similar to the th vector , is the time series vector; : Total data length; : Embedding dimension; : Index of the time series, range where is the total data length, is the embedding dimension, represents the th vector; Calculate the entropy value : ; : Proportion of similar vectors when the embedding dimension is ; : Complexity of the tongue movement trajectory, i.e., the entropy value ; Yawn: Simple trajectory, low value, ; Speech: Complex trajectory, high value, .

[0032] Step S3, defining an input tensor, the input being time series data containing multimodal physiological indicators, and constructing a one-dimensional convolutional neural network for processing multimodal time series data; Step S3.1, input data definition, The input is time series data containing multimodal physiological indicators, and each time window contains sampling points (e.g. Y=60, corresponding to 5 seconds of data, sampling frequency is 12Hz), feature dimension is 9; Input Tensor ,like Figure 3 As shown: Input Example The matrix: ; : Sampling point The degree of eyelid closure at ; : Sampling point The mouth corner position shifts when ; : Sampling point The head center of gravity shifts when ; : Sampling point Prefrontal blood flow at ; : Sampling point The tongue displacement amplitude at ; : Sampling point The direction of tongue movement, ; : Sampling point The tongue movement speed, ; : Sampling point The tongue movement acceleration at ; : Sampling point The complexity of the tongue movement trajectory at ; Step S3.2, Convolutional Neural Network (CNN) architecture The present invention uses a one-dimensional convolutional neural network (1D-CNN) to process multi-modal time series data. The network architecture starts from the input layer to receive a feature sequence with a length of 60 time steps and a dimension of 9, and sequentially extracts spatio-temporal features through two convolutional and pooling modules. The first layer uses 64 convolutional kernels of size 5 to slide along the time dimension, captures local time patterns (such as the gradual process of eyelid closure or sudden drop in blood flow) through the ReLU activation function, and then performs max pooling with a stride of 2 to compress the feature dimension and enhance translational invariance; the second layer further stacks 128 convolutional kernels of size 3 to fuse cross-modal associations at a higher abstraction level (such as the co-variation of head center of gravity shift and tongue movement), and obtains high-level spatio-temporal features after pooling again. The feature map is flattened and connected to a fully connected layer. First, global feature integration is performed through a 256-dimensional hidden layer, and finally, the 3-neuron output layer combines the Softmax function to generate the probability distributions of wakefulness, mild fatigue, and severe fatigue; the entire network gradually expands the receptive field through hierarchical convolutional operations to extract multi-scale time series patterns, and the pooling operation reduces the impact of local fluctuations, ultimately realizing end-to-end fatigue state mapping as Figure 4 shown in: Among them: Convolutional layer 1, that is: ; : Convolutional layer 1; : The th convolutional kernel at the input channel, the th time step weight; : The bias value of the th convolutional kernel; : Input tensor; : Convolutional kernel index, range , indicating that there are 64 convolutional kernels in the first layer; : Input feature channel index, range , corresponding to 9 input feature dimensions; : Output feature map time step index, the input time window length is 60, and the output dimension is ; Max pooling layer 1: ; : Max pooling layer 1; : The value of the -th channel of the feature map output by convolutional layer 1 at time step ; : The value of the -th channel of the feature map output by convolutional layer 1 at time step ; : The time step index after pooling, with a pooling stride of 2 and an output dimension of ; Convolutional layer 2: ; : Convolutional layer 2; : The -th convolutional kernel's weight at the -th input channel and the -th time step; : The value of the -th channel in the output of pooling layer 1 at time step ; : The convolutional kernel index, ranging from , indicating that there are 128 convolutional kernels in the first layer; : The input feature channel index, ranging from , corresponding to the 64 output channels of the first layer; : The time step index of the output feature map, with an output dimension of ; Max pooling layer 2: ; : Max pooling layer 2; : The value of the -th channel of the feature map output by convolutional layer 2 at time step ; : The value of the -th channel of the feature map output by convolutional layer 2 at time step ; : The time step index after pooling, with a pooling stride of 2 and an output dimension of ; Fully connected layer 1: ; : Fully connected layer 1; : The weight matrix of the fully connected layer 1, with a dimension of ; : The output of the pooling layer 2 The dimension is , and the length of the flattened vector is 1920; : The bias term of the fully connected layer 1, with a dimension of 256; Fully connected layer 2: ; : The fully connected layer 2; : The weight matrix of the fully connected layer 2, with a dimension of ; : The output vector of the fully connected layer 1, with a dimension of 256; : The bias term of the fully connected layer 2, with a dimension of 3.

[0033] Step S4, Output layer design. According to the input tensor and the input layer of the convolutional neural network, design the output layer to obtain the probability vector of the driver's fatigue driving state and judge the degree of the driver's fatigue driving; Step S4.1, Output layer design, The output layer is designed as a fully connected layer containing 3 neurons, corresponding to the three states of the driver, namely awake, mildly fatigued, and severely fatigued. Each neuron uses the Softmax activation function to convert the linear output of the previous layer into a normalized probability value, representing the confidence of the occurrence of this state. Specifically, it includes: after the original output value (logits) of the last layer of the network passes through the Softmax transformation, a three-element probability vector is generated, where each element represents the predicted probability of the corresponding category, and the sum of all probabilities is equal to 1; in practical applications, the system will select the category with the highest probability value as the final judgment result. For example, if the output probability vector is [0.05, 0.25, 0.70], it is determined that the driver is in a severely fatigued state. To further improve the reliability, by setting a probability threshold (such as the highest probability needs to exceed 0.7), if the probabilities of all categories are lower than the set probability threshold, it is marked as an uncertain state and the multi-modal data review mechanism module is started for review; this design not only ensures the intuitiveness of the classification decision but also quantifies the confidence of the judgment through probability, providing an interpretable decision basis for subsequent hierarchical early warnings (such as sound prompts, seat vibrations, or forced stops).

[0034] Output dimension: 3D probability vector , corresponding to three types of fatigue states; : Awake, ; : Mild fatigue, ; : Severe fatigue, ; : Karolinska Sleepiness Scale; Softmax formula: ; , is the original output value of the fully connected layer 2, and e is the natural constant; Step S4.2, Judgment rule, Direct classification: Take the category with the highest probability as the prediction result:

[0035] : The classification result of the current fatigue level of the driver, which is a discrete category label with values of 0, 1, 2; : The traversed variable, with values of 0, 1, 2, used to compare the probability values of the three categories and return the index of the category with the highest probability.

[0036] Step S5. For the driver detected in a fatigue driving state, control their vehicle, and implement corresponding vehicle control strategies according to the obtained fatigue level of the driver; Step S5.1. According to the detected fatigue level of the driver, adopt the following vehicle control strategies: When mild fatigue of the driver is detected, start a low-frequency in-vehicle voice broadcast to remind the driver to take a break, reduce the throttle opening to 4 / 5 of the current opening, and dynamically adjust the throttle response curve according to the fatigue probability value to reduce the acceleration upper limit: ; is the current maximum acceleration of the vehicle, is the original maximum acceleration of the vehicle, : Mild fatigue, ; When severe fatigue of the driver is detected, start a high-frequency in-vehicle voice broadcast to remind the driver to take a break, control the vehicle lights to flash to remind other drivers to pay attention and avoid, reduce the throttle opening to 5 / 6 of the current opening, lock the automatic transmission, prohibit downshifting and accelerating, and directly limit the engine output torque to the safety threshold: ; wherein is the current torque of the engine, is the rated torque of the engine, is the safety threshold of the current engine output torque.

[0037] In the specific implementation process, obtain the 3D model of the driver's head in the initial state through a depth camera and a portable ultrasonic probe, and obtain the subsequent real-time head external contour point cloud data of the driver.

[0038] Calculate the current physiological indexes of the driver according to the real-time head external contour point cloud data of the driver and the set time window and sampling frequency in the neural network.

[0039] After the calculation is completed, input the matrix of the driver's physiological indexes into the neural network, and output the 3D probability vector of the driver's fatigue judgment through the neural network.

[0040] Judge the current fatigue degree of the driver through the 3D probability vector, and adopt corresponding vehicle control strategies according to the current fatigue degree of the driver.

[0041] The above is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for detecting the fatigue driving state of a driver based on head 3D modeling, characterized in that, It includes the following steps: Step S1: The depth camera acquires the point cloud of the external contour of the driver's head, and the portable ultrasonic probe scans the blood vessel positions of the driver to obtain the 3D modeling of the driver's head; Step S2: Based on the 3D modeling of the driver's head, physiological index detections such as the eyelid closure degree, the deviation of the corner of the mouth position, the deviation of the head center of gravity, the blood flow in the prefrontal lobe, and the tongue state of the driver are carried out to obtain the driver's physiological indexes; Step S3: Define the input tensor, the input is the time series data containing multi-modal physiological indexes, and a one-dimensional convolutional neural network for processing multi-modal time series data is constructed; Step S4: Output layer design, according to the input tensor and the input layer of the convolutional neural network CNN architecture, design the output layer to obtain the probability vector of the driver's fatigue driving state and judge the degree of the driver's fatigue driving; Step S5: For the driver detected to be in a fatigue driving state, control his vehicle, and implement the corresponding vehicle control strategy according to the obtained degree of the driver's fatigue driving.

2. The method for detecting the fatigue driving state of a driver based on head 3D modeling according to claim 1, wherein, In step S1, the following steps are also included: Step S1.1: Driver's head data acquisition, Step S1.1.1: External contour modeling: Use the depth camera to obtain the 3D point cloud data on the surface of the driver's head, and its calculation formula is: ； : Head surface point cloud data of the driver; : Number of collected point clouds; : The -th point's three-dimensional coordinates in the global coordinate system; Using the Delaunay triangulation algorithm, convert the point cloud data into a continuous surface : ； : The set of vertices, : The -th vertex, : The number of vertices, : The vertex coordinates; : Set of triangular patches, : The th patch, : Number of patches, from the set of vertices , which are the vertices of the triangular patch . Step S1.1.2: Internal structure mapping: Install the portable ultrasonic probe on the headrest of the driver's seat, perform ultrasonic detection on the driver's head, and obtain the blood vessel distribution of the driver's head; Step S1.1.2.1: Blood vessel centerline extraction: Adopt the three-dimensional Hessian matrix filtering to enhance the blood vessel structure module and extract the centerline coordinates: ； : The set of the midpoint coordinates of the blood vessel centerline and the blood vessel radius; , , : The three-dimensional coordinates of the midpoint of the centerline of the th segment of blood vessel : The radius of the blood vessel in the : Total number of blood vessel segments; Step S1.1.2.2: Blood vessel fractal model construction: Based on fractal dimension Describe the complexity of blood vessel branching: ； : The number of blood vessels with a radius less than ; : Fractal dimension; Step S1.1.2.3: Inertial unit measurement: Measure the head posture through the IMU sensor and represent the rotation using the attitude quaternion: Attitude quaternion : ； : The real part represents the rotation angle; , , : The imaginary part, representing the direction of the rotation axis; : The unit vector along the positive direction of the axis; : The unit vector along the positive direction of the axis; : The unit vector along the positive direction of the axis; Step S1.2: Coordinate system alignment: Step S1.2.1: Global coordinate system definition: With the head center of gravity as the origin, and the vertical direction as the vertical axis axis direction, the positive direction of the axis points to the roof of the vehicle, perpendicular to the axis and pointing directly in front of the driver's face as the front-rear axis the positive direction of the axis, perpendicular to the axis and the plane formed by the axis is the left-right axis axis direction, the positive direction of the axis points to the right, defining a global three-dimensional rectangular coordinate system : ； : The head center of gravity coordinates in the initial state; : Number of collected point clouds; Step S1.2.2: Sensor extrinsic parameter calibration: Align the coordinate systems of the depth camera, the portable ultrasonic probe, and the IMU sensor to the global coordinate system: The form of each transformation matrix is: ； : Rotation matrix; : Translation vector; Step S1.3: Point cloud preprocessing, Step S1.3.1: Outlier filtering: Calculate the distance and standard deviation of all points within the neighborhood of each point. If the distance between a certain point in the neighborhood and this point exceeds the threshold, it is judged as an outlier and removed; Neighborhood radius Average distance of interior points and standard deviation : ； ； : The neighborhood set of the -th point in the point cloud, that is, all points within a radius of ; within millimeters; : The number of points within the neighborhood; : The th three-dimensional point coordinates to be evaluated; : Neighborhood the th three-dimensional point coordinate; The threshold is set as: ； Step S1.3.2: Voxel grid downsampling: Evenly divide the point cloud into voxels; The downsampled point cloud set is as follows: ； : The th three-dimensional point coordinates to be evaluated; : The point set within the : The number of points after downsampling; Step S1.4: Model fusion and final output, Step S1.4.1: Triangular mesh surface and blood vessel fusion: Map the vascular centerline data onto the triangular mesh surface to obtain a 3D head model containing the vascular distribution , and guide the layout of NIRS sensors: ； : The set of the midpoint coordinates of the blood vessel centerline and the blood vessel radius; , , : The three-dimensional coordinates of the center line of the th blood vessel segment; : The radius of the th blood vessel segment; Step S1.4.2: Head dynamic posture compensation: By rotation matrix update the model pose in real time: ； : Quaternion representation, describing the gimbal lock problem of the rotation of the head in three-dimensional space; : The real part represents the rotation angle; , , : The imaginary part, representing the direction of the rotation axis; : The unit vector along the axis direction; : Unit vector along the axis direction; : The unit vector along the axis direction; Step S1.4.3: Model output: The final model includes: Geometric surface, i.e., triangular mesh ; Vascular network, i.e., the classification model ; Dynamic attitude parameters, namely quaternions and the rotation matrix .

3. A method for detecting the fatigue driving state of a driver based on 3D head modeling according to claim 1, characterized in that, In step S2, the following steps are also included: Step S2.1, eyelid closure degree Detection 3D key point detection: Locate the upper eyelid points and the lower eyelid points which are the intersection points of the straight line passing through the pupil center and perpendicular to the axis with the upper and lower eyelids; Calculation of closure degree : ； Reference distance when eyes are open : ； Reference distance when eyes are closed : ； Among them i.e., the upper eyelid point , i.e., the lower eyelid point ; : Real-time three-dimensional coordinates of the upper eyelid point; : Real-time three-dimensional coordinates of the lower eyelid point; : Coordinates of the upper eyelid point when the eyes are fully open ; : Lower eyelid point when eyes are fully open Coordinate : Upper eyelid point when eyes are fully closed z Coordinates : Lower eyelid point when eyes are fully closed z Coordinate : Reference spacing when eyes are open; : Reference spacing when eyes are closed; Step S2.2, offset of the corner of the mouth Detection 3D key point extraction: Locate the left and right corner points of the mouth and , and the left mouth corner reference position in the resting state and the right mouth corner reference position in the resting state ; Dynamic offset : ； : Left corner of the mouth point ; : Right corner of the mouth point ; : Real-time three-dimensional coordinates of the left corner of the mouth; : Real-time three-dimensional coordinates of the right corner of the mouth; : namely , the reference position of the left corner of the mouth at rest; : namely , the reference position of the right corner of the mouth at rest; : Three-dimensional coordinates of the left corner of the mouth at rest; : Three-dimensional coordinates of the right corner of the mouth at rest; Step S2.3, Head center of gravity shift , Real-time center of gravity of the head Calculation: ； : Number of collected point clouds; , , : The real-time coordinates of the th point in the global coordinate system; Offset : ； : Real-time head center-of-gravity coordinates; : The head center of gravity coordinates in the initial state; Step S2.4, Blood flow in the prefrontal lobe Detection NIRS sensor deployment: According to the 3D modeling of the driver's head, prompt the driver to deploy the NIRS sensor in the area with dense blood vessel distribution; Blood flow Estimation: ； : Change in oxyhemoglobin concentration; : Baseline concentration of oxyhemoglobin at rest; : calibration coefficient; Step S2.5, tongue state ( 、 、 、 、 ) detection Perform a separate 3D modeling of the oral region using a near-infrared structured light camera installed above the dashboard with a wavelength of 850 mm to penetrate the lip surface and collect lip dynamic point cloud data at 30 frames per second. : ； : Coordinates of the th point; : The total number of lip dynamic point cloud data collected; Use the U-Net network to perform semantic segmentation on the internal oral cavity area and extract the coordinates of the tip of the tongue body ; Three-dimensional displacement trajectory analysis: Displacement amplitude is the instantaneous displacement relative to the reference position of the tip of the tongue body as follows: ； : Displacement amplitude of the tip of the tongue body; : Co-ordinates of the tip of the tongue at a given moment; : Reference position of the tip of the tongue body; The amplitude is large and continuously increasing when yawning, while the amplitude is small and periodically fluctuating when speaking, ; Motion direction: For consecutive K coordinates of the tip of the tongue body in frames perform polynomial fitting to obtain the motion trajectory equation: ； Among them 、 、 are the parameters of the trajectory equation and is a three-dimensional vector; : Locus motion equation; Calculate the instantaneous motion direction vector : ； In the global coordinate system, calculate the angle with the vertical axis of : ； : The angle between the instantaneous motion direction vector and the vertical axis; : Instantaneous motion direction vector; Yawning: mainly downward vertically, ; Speech: mainly horizontal swinging, ; Instantaneous velocity and acceleration: Instantaneous velocity : ； : The coordinates of the tip of the tongue at a given moment; : The coordinates of the tip of the tongue at a given moment, taking ; Yawning: low speed, , showing a slow rising and falling trend; Speech: high speed, , accompanied by short pulses; Acceleration : ； : The instantaneous velocity of the tip of the tongue at a given moment; : The instantaneous velocity of the tip of the tongue at a moment Yawning: The absolute value of acceleration is small, ; Speech: The absolute value of acceleration changes violently, ; Motion trajectory complexity: Quantify the complexity by calculating the approximate entropy of the tongue movement path: Parameter setting: Embedding dimension , tolerance ; Construct a time series vector : ； : The displacement amplitude of the tip of the tongue body at the th time point; : The displacement amplitude of the tip of the tongue body at the th time point; : Index of the time series, in the range where is the total data length, is the embedding dimension; Statistical proportion of similar vectors : ； : A threshold for tolerance to determine whether two time series vectors are similar; : within the tolerance , the ratio similar to the th vector is the time series vector; is the time series vector; : Total data length; : Embedding dimension; : Index of the time series, range where is the total data length, is the embedding dimension, represents the th vector; Calculate the entropy value : ； : Proportion of similar vectors when the embedding dimension is ; : Complexity of tongue movement trajectory, i.e., entropy value ; Yawning: The trajectory is simple and low value, ; Speech: Complex trajectory, high value, .

4. A method for detecting the fatigue driving state of a driver based on head 3D modeling according to claim 1, characterized in that, In step S3, the following steps are also included: Step S3.1: Input data definition, The input is time series data containing multimodal physiological indicators, and each time window contains sampling points, and the feature dimension is 9; Input tensor ; Input example matrix: ； : Eyelid closure degree at the sampling point, ; : Sampling point The mouth corner position shifts when ; : Head center of gravity offset at the sampling point; ; : Prefrontal blood flow at the sampling point, ; : Tongue displacement amplitude at the sampling point, ; : Tongue movement direction at the sampling point; ; : Tongue movement speed at the sampling point, ; : Tongue movement acceleration at the sampling point, ; : Complexity of tongue movement trajectory at the sampling point, ; Step S3.2, Convolutional Neural Network (CNN) architecture A one-dimensional convolutional neural network is used to process multi-modal time series data. The network architecture receives a feature sequence with a length of 60 time steps and a dimension of 9 from the input layer, and sequentially extracts spatio-temporal features through two layers of convolutional and pooling modules. The first layer uses 64 convolutional kernels of size 5 to slide along the time dimension, captures local time patterns through the ReLU activation function, and then performs max pooling with a stride of 2 to compress the feature dimension and enhance translation invariance. The second layer further stacks 128 convolutional kernels of size 3 to fuse cross-modal associations at a higher abstraction level. After pooling again, high-level spatio-temporal features are obtained. The feature map is flattened and connected to a fully connected layer. First, global feature integration is performed through a 256-dimensional hidden layer, and finally, a 3-neuron output layer combines the Softmax function to generate the probability distributions of being awake, mildly fatigued, and severely fatigued. Among them, Convolutional Layer 1, that is: ； : Convolution layer 1; : The weight of the th convolutional kernel at the th input channel and the th time step; : The bias value of the th convolutional kernel; : Input tensor; : Convolution kernel index, range , indicating that there are 64 convolution kernels in the first layer; : Input feature channel index, range , corresponding to 9 input feature dimensions; : The time step index of the output feature map, with the input time window length of 60 and the output dimension of ; Max Pooling Layer 1: ； : Max pooling layer 1; : The value of the -th channel of the feature map output by the first convolutional layer at time step ; : The value of the -th channel of the output feature map of convolutional layer 1 at time step . : The time step index after pooling, with a pooling stride of 2 and an output dimension of ; Convolutional Layer 2: ； : Convolutional layer 2; : The weight of the th convolutional kernel at the input channel and the th time step; : The value of the -th channel in the output of pooling layer 1 at time step ; : Convolution kernel index, range , indicating that there are 128 convolution kernels in the first layer; : Input feature channel index, range , corresponding to 64 output channels of layer 1; : The time step index of the output feature map, with an output dimension of ; Max Pooling Layer 2: ； : Max pooling layer 2; : The value of the -th channel of the feature map output by the convolutional layer 2 at the time step ; : The value of the -th channel of the feature map output by the convolutional layer 2 at the time step ; : The time step index after pooling, with a pooling stride of 2 and an output dimension of ; Fully Connected Layer 1: ； : Fully connected layer 1; : The weight matrix of the fully connected layer 1, with a dimension of ; : Output of pooling layer 2 The dimension is , and the length of the flattened vector is 1920; : The bias term of the fully connected layer 1, with a dimension of 256; Fully Connected Layer 2: ； : Fully connected layer 2; : The weight matrix of the fully connected two layers, with a dimension of ; : The output vector of the fully connected layer 1, with a dimension of 256; : The bias term of the fully connected layer 2, with a dimension of 3.

5. A method for detecting the fatigue driving state of a driver based on 3D head modeling according to claim 1, characterized in that Step S4 also includes the following steps: Step S4.1, Output layer design The output layer is designed as a fully connected layer containing 3 neurons, corresponding to the three states of the driver, namely being awake, mildly fatigued, and severely fatigued. Each neuron uses the Softmax activation function to convert the linear output of the previous layer into a normalized probability value, representing the confidence of the occurrence of that state. Specifically, after the original output value of the last layer of the network undergoes the Softmax transformation, a three-element probability vector is generated, where each element represents the prediction probability of the corresponding category, and the sum of all probabilities is equal to 1. By setting a probability threshold, if all category probabilities are lower than the set probability threshold, it is marked as an uncertain state and the multi-modal data review mechanism module is activated for review. Output dimension: 3D probability vector , corresponding to three types of fatigue states; : Awake, ; : Mild fatigue, ; : Severe fatigue, ; : Karolinska Sleepiness Scale; Softmax formula: ； , is the original output value of the fully connected layer 2, and e is the natural constant; Step S4.2, Judgment rule Direct classification: Take the category with the highest probability as the prediction result: （ ）； : The classification result of the current fatigue level of the driver, which is a discrete category label with values of 0, 1, and 2; : The variable for traversal, with values of 0, 1, 2, used to compare the probability values of three categories and return the index of the category with the maximum probability.

6. The method for detecting the fatigue driving state of a driver based on head 3D modeling according to claim 1, wherein Step S5 also includes the following steps: Step S5.1, According to the detected fatigue level of the driver, adopt the following vehicle control strategies: When mild driver fatigue is detected, start a low-frequency in-vehicle voice broadcast to remind the driver to stop and rest, reduce the throttle opening to 4 / 5 of the current opening, and dynamically adjust the throttle response curve and reduce the acceleration upper limit: ； is the current maximum acceleration of the vehicle, is the original maximum acceleration of the vehicle, : mild fatigue, ; When it is detected that the driver is severely fatigued, start a high-frequency in-vehicle voice broadcast to remind the driver to stop and rest, control the vehicle lights to flash to remind other drivers to pay attention and avoid, reduce the throttle opening to 5 / 6 of the current opening, lock the automatic transmission, prohibit downshifting and accelerating, and directly limit the engine output torque to the safety threshold: ； wherein is the current torque of the engine, is the rated torque of the engine, is the safety threshold of the current engine output torque.

Images