Method and system for detecting a driver's fatigue driving state

Abstract

This invention discloses a method and system for detecting driver fatigue. The invention relates to the technical field of driving state detection methods. It determines a dynamic diagram of the driver's driving state based on multiple past driving states, corresponding combinations of driver driving images, and corresponding time points. Multiple fatigue driving states are then identified based on this dynamic diagram. A fatigue driving state recognition system is constructed based on these multiple fatigue driving states, the driver's corresponding facial image, and the corresponding fatigue driving actions. Therefore, the driver's fatigue driving state is determined based on the driving state coefficient, the driver's current interaction information, and the corresponding current driving action. A safe driving mode for the truck is determined based on the driver's current facial image and the driver's continuous driving duration. Safe stopping measures are then determined based on the truck's safe driving mode, the remaining driving path, and the driver's eye expression image, improving the accuracy of safe stopping measures.

Description

Technical Field

[0001] This invention relates to the technical field of driving state detection methods, and more particularly to a method and system for detecting driver fatigue driving state. Background Technology

[0002] With the development of technology, trucks serve as corresponding cargo transfer equipment and are operated by drivers. As drivers drive trucks, their driving state gradually adjusts over long periods of time. In existing technologies, multiple driving data are collected from the driver, and fatigue driving status is identified based on the identification of each driving data. However, the overall consideration of fatigue driving actions is not taken into account, which affects the accuracy of the fatigue driving status identification system and thus the accuracy of safe parking measures. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of the prior art. This invention provides a method and system for detecting driver fatigue.

[0004] This invention provides a method for detecting driver fatigue, comprising: determining multiple driving image combinations based on multiple driving images of the driver and corresponding image acquisition times in the driver's past driving events; determining multiple sub-driving state features of the driver based on the recognition of each driving image combination; determining the past driving state corresponding to the driving image combination based on the multiple sub-driving state features, the driver's emotional interaction information, and the driver's continuous driving duration; determining a dynamic diagram of the driver's driving state based on the multiple past driving states, the corresponding driver driving image combinations, and the corresponding time nodes, and determining multiple fatigue driving states based on the dynamic diagram of the driving state; constructing a fatigue driving state recognition system based on the multiple fatigue driving states, the driver's corresponding facial image, and the corresponding fatigue driving actions; determining a corresponding driving state coefficient based on the driver's current facial image and the fatigue driving state recognition system, and determining the driver's fatigue driving state based on the driving state coefficient, the driver's current interaction information, and the corresponding current driving actions, thereby triggering safety control of the truck; when the driver is in a fatigue driving state, determining a safe driving mode for the truck based on the driver's current facial image and the driver's continuous driving duration, and determining safe stopping measures based on the truck's safe driving mode, the remaining driving path, and the driver's eye image.

[0005] This invention provides a driver fatigue driving state detection system, which is applied to the above-described driver fatigue driving state detection method. The driver fatigue driving state detection system includes: The driving image combination module is used to determine multiple driving image combinations based on multiple driving images of the driver and the corresponding image acquisition time in the driver's past driving events; The previous driving state module is used to determine multiple sub-driving state features of the driver based on the recognition of various driving image combinations; and to determine the previous driving state corresponding to the driving image combination based on the multiple sub-driving state features, the driver's emotional interaction information and the driver's continuous driving duration. The recognition system module is used to determine the dynamic map of the driver's driving state based on multiple past driving states, the corresponding combination of driver driving images, and the corresponding time points, and to determine multiple fatigue driving states based on the dynamic map of the driving state; and to construct a fatigue driving state recognition system based on multiple fatigue driving states, the driver's corresponding facial images, and the corresponding fatigue driving actions. The fatigue driving status module is used to determine the corresponding driving status coefficient based on the driver's current facial image and the fatigue driving status recognition system. Based on the driving status coefficient, the driver's current interaction information and the corresponding current driving action, the driver's fatigue driving status is determined to trigger the safety control of the truck. The safe parking measures module is used to determine the safe driving mode of the truck based on the driver's current facial image and the driver's continuous driving time when the driver is in a state of fatigue driving. It then determines safe parking measures based on the truck's safe driving mode, the remaining driving route, and the driver's eye image.

[0006] Compared with the prior art, the beneficial effects of the present invention are: In this embodiment of the invention, the method is used to determine a dynamic diagram of the driver's driving state based on multiple previous driving states, corresponding combinations of driver driving images, and corresponding time points. Multiple fatigue driving states are then determined based on this dynamic diagram. A fatigue driving state recognition system is constructed based on multiple fatigue driving states, the driver's corresponding facial images, and the corresponding fatigue driving actions. The introduction of the dynamic diagram of the driver's driving state, which incorporates a holistic consideration of multiple fatigue driving states, the driver's corresponding facial images, and the corresponding fatigue driving actions, improves the accuracy of the fatigue driving state recognition system.

[0007] Therefore, a driving state coefficient is determined based on the driver's current facial image and fatigue driving state recognition system. This coefficient, along with the driver's current interaction information and corresponding current driving actions, determines the driver's fatigue driving state, triggering safety control measures for the truck. When the driver is fatigued, a safe driving mode for the truck is determined based on the driver's current facial image and continuous driving duration. Safe stopping measures are then determined based on the truck's safe driving mode, the remaining driving path, and the driver's eye contact image. This system introduces driver fatigue driving state recognition and achieves truck safety control. Furthermore, it incorporates a holistic consideration of the truck's safe driving mode, the remaining driving path, and the driver's eye contact image, improving the accuracy of safe stopping measures. Attached Figure Description

[0008] Figure 1 This is a flowchart illustrating the driver fatigue driving state detection method in an embodiment of the present invention. Figure 2 This is a flowchart illustrating step S11 in the driver fatigue driving state detection method in this embodiment of the invention. Figure 3 This is a flowchart illustrating step S12 in the driver fatigue driving state detection method in this embodiment of the invention. Figure 4 This is a flowchart illustrating step S13 in the driver fatigue driving state detection method in this embodiment of the invention. Figure 5 This is a flowchart illustrating step S14 of the driver fatigue driving state detection method in an embodiment of the present invention. Figure 6 This is a flowchart illustrating step S15 of the driver fatigue driving state detection method in this embodiment of the invention. Figure 7 This is a schematic diagram of the structural composition of the driver fatigue driving state detection system in an embodiment of the present invention. Detailed Implementation

[0009] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0010] Please see Figures 1 to 7 A method for detecting driver fatigue driving state, applied in driving state detection scenarios; the method for detecting driver fatigue driving state includes: Step S11: Based on the driver's past driving events, determine multiple driving image combinations according to the driver's multiple driving images and the corresponding image acquisition time; Step S12: Based on the recognition of each driving image combination, determine multiple sub-driving state features of the driver; determine the previous driving state corresponding to the driving image combination according to the multiple sub-driving state features, the driver's emotional interaction information and the driver's continuous driving time; the multiple sub-driving state features include blink frequency, head posture and number of yawns; Step S13: Based on multiple past driving states, the corresponding driver driving image combination, and the corresponding time node, determine the driver's driving state dynamic map, and determine multiple fatigue driving states based on the driving state dynamic map; construct a fatigue driving state recognition system based on multiple fatigue driving states, the driver's corresponding facial image, and the corresponding fatigue driving action; the driving state dynamic map not only monitors the current driving state in real time, but also compares with historical dynamic maps to identify abnormal fatigue patterns. Step S14: Determine the corresponding driving state coefficient based on the driver's current facial image and the fatigue driving state recognition system. Determine the driver's fatigue driving state based on the driving state coefficient, the driver's current interaction information, and the corresponding current driving action to trigger the safety control of the truck. Fatigue driving state includes mild fatigue, moderate fatigue, and severe fatigue. Step S15: When the driver is in a state of fatigued driving, determine the safe driving mode of the truck based on the driver's current facial image and the driver's continuous driving time, and determine safe stopping measures based on the safe driving mode of the truck, the remaining driving route and the driver's eye image; safe stopping measures include mild fatigue stopping measures, moderate fatigue stopping measures and severe fatigue stopping measures.

[0011] refer to Figure 2 In step S11, the specific steps are as follows: S111: Collect the driver's truck driving database, determine the driver's past driving events based on the detection of the truck driving database, determine multiple driving images of the driver based on the tracing of the past driving events, and mark the corresponding image acquisition time. S112: Determine the driver's previous driving time period based on the traceability of image acquisition time, determine the driver's driving signal based on the matching of the previous driving time period with the truck driving database, and determine multiple driving image combinations based on the driver's driving signal, multiple driving images of the driver and the corresponding image acquisition time.

[0012] In the embodiments of this application, the primary task of establishing a fatigue driving detection system is to build a comprehensive truck driver database. This basic data collection phase requires the integration of multi-dimensional information. The database should include basic driver information (age, driving experience, work and rest habits, etc.), vehicle information (model, load, vehicle status, etc.), environmental information (weather conditions, road type, time period, etc.), driving behavior data (steering wheel angle, accelerator and brake operation, vehicle speed changes, etc.), and video data (driver's facial expressions, eye status, head posture, etc.).

[0013] Data collection is typically accomplished through the collaboration of multiple devices: vehicle-mounted cameras (capturing the driver from inside and the road from outside), vehicle-mounted sensors (steering wheel sensors, accelerator and brake sensors, etc.), CAN bus data (vehicle speed, engine speed, etc.), and manual recording (driver's self-assessment of fatigue level, rest status, etc.).

[0014] Meaningful driving events are identified and extracted from a massive database. Driving events refer to driving segments with specific characteristics or importance, mainly including fatigue-related events (such as long-term continuous driving, frequent yawning, increased blinking frequency, etc.), abnormal driving events (such as sudden acceleration / deceleration, lane departure, excessive steering wheel correction, etc.), and specific scenario events (such as night driving, driving in bad weather, driving in congested areas, etc.). At the same time, key image frames are extracted from the identified driving events, and precise timestamps are added to each image frame. This process includes image selection (selecting representative image frames from each driving event), time stamping (adding precise acquisition time to each image frame), image preprocessing (performing necessary preprocessing on the images, such as face detection, alignment, normalization, etc.), and image annotation (performing preliminary annotations on the images, such as facial key points, eye status, etc.).

[0015] Furthermore, discrete image time points are organized into continuous time periods to enable the system to analyze changes in driver state over time; time series analysis is performed, and timestamped images are arranged in chronological order to form a continuous timeline; continuous driving time is divided into multiple meaningful time periods based on factors such as driving characteristics and rest intervals; statistical characteristics are calculated for each time period, such as quantitative indicators like average blink frequency and head posture changes; time period division can be based on various criteria: continuous driving duration (e.g., every 30 minutes is a time period), changes in driving tasks (e.g., highway vs. city roads), or rest intervals (e.g., driving time between two rest periods).

[0016] The predefined time periods are precisely matched with other data in the truck driving database to extract driving signals corresponding to those time periods. Driving signals include various types: operation signals (steering wheel angle, frequency and force of accelerator and brake operation), vehicle signals (vehicle speed changes, lane departure, following distance), environmental signals (weather conditions, road type, traffic flow), and physiological signals (heart rate changes, if relevant equipment is available). The matching process uses time synchronization technology to precisely match driving signals with image timestamps, ensuring that data from different sources are consistent in time, and extracting meaningful features from the raw data.

[0017] Driving signals, driving images, and time information are organically integrated to form driving image assemblies for fatigue analysis. Each assembly contains multi-dimensional information that can comprehensively reflect the driver's state during a specific time period. The assembly construction follows principles such as temporal consistency (ensuring that all data within the assembly correspond in time), information complementarity (different types of data complement and verify each other), and feature representativeness (the assembly should be able to represent the typical state of the driver during that time period). At the same time, each driving image assembly includes a core image sequence (keyframes reflecting the driver's facial state), driving signal features (quantified operational behavior data), temporal context (time period information, continuous driving duration, etc.), and environmental factors (road, weather, traffic conditions, etc.).

[0018] refer to Figure 3 In step S12, the specific steps are as follows: S121: Collect various driving image combinations and trigger the recognition of various driving image combinations. Determine the driver's driving state event based on the recognition of various driving image combinations. Determine multiple sub-driving state events based on the detection of the driver's driving state events. Determine the sub-driving state features corresponding to the driver based on the recognition of each sub-driving state event, so as to collect multiple sub-driving state features of the driver. S122: In each driving image combination, the driver's emotional interaction information is determined based on the filtering of the driving image combination, and the driver's continuous driving time is determined based on the tracing of the interaction time corresponding to the driver's emotional interaction information. The previous driving state corresponding to the driving image combination is determined based on multiple sub-driving state features, the driver's emotional interaction information and the driver's continuous driving time.

[0019] In the embodiments of this application, the previously constructed driving image combinations are used as input data to prepare for subsequent state recognition analysis; all constructed driving image combinations are extracted from the database, each combination containing multiple frames of timestamped driving images, while ensuring image quality and integrity, filtering out blurry or severely occluded images; the images are standardized, including size adjustment, brightness normalization, etc., and basic image metadata such as resolution and frame rate are extracted; the image combinations are organized according to time sequence or specific rules, a processing queue is established and parallel processing resources are prepared to improve recognition efficiency.

[0020] The system analyzes image content using computer vision and machine learning algorithms; selects appropriate recognition algorithms based on the application scenario, including face recognition, pose estimation, and expression analysis, and configures algorithm parameters to optimize recognition results; initializes the recognition process by loading pre-trained models or algorithms, setting recognition thresholds and confidence requirements, and initializing the output buffer to store recognition results; performs batch recognition by batch processing all images in each image combination, monitoring recognition progress and quality in real time, and handling any anomalies that occur during the recognition process.

[0021] Meaningful driving state events are extracted from continuous recognition results. These events represent significant changes or specific behaviors in the driver's state. The types of driving state events that need attention are identified, clear judgment criteria are set for each event, and the minimum duration and significance requirements of the event are determined. Event detection is performed by analyzing feature changes between consecutive frames, identifying feature patterns that meet the defined criteria, and recording the time, duration, and intensity of the event. Event classification is performed by categorizing the detected events according to their types, assessing the importance and severity of the events, and establishing the correlation between events.

[0022] The detected driving state events are further subdivided into more specific sub-driving state events to more accurately describe the driver's state changes; each driving state event is decomposed into smaller sub-events, the type, boundaries, and characteristics of the sub-events are determined, and the correlation between the sub-events and the parent event is established; quantitative and qualitative features are extracted for each sub-event, the importance and weight of the features are determined, and the correspondence between the features and the degree of fatigue is established; the sub-events are classified according to a predefined classification system, the severity and scope of impact of the sub-events are assessed, and the temporal and causal relationships between the sub-events are established.

[0023] Quantitative feature parameters are extracted from the identified sub-driving state events to form feature vectors that can be used for fatigue state assessment; the key feature types to be extracted are determined, calculation methods and units are defined for each feature, and a mapping relationship between features and fatigue levels is established; quantitative analysis is performed on each sub-driving state event to calculate the specific values ​​and trends of the features, and to evaluate the statistical properties of the features, such as mean, variance, and extreme values; the calculated features are organized into structured data, and the correlation and hierarchical relationships between features are established to form feature vectors or feature matrices that can be used for subsequent analysis.

[0024] Furthermore, from the multiple combinations of driving images already collected, representative or abnormal combinations are selected based on conditions such as time, scene, and event type for emotion interaction analysis. The system selects driving image combinations that meet specific conditions and uses an expression recognition model (such as a deep learning-based FER model) to identify the driver's facial expressions, including various emotional states such as happiness, anger, surprise, and fatigue. At the same time, it combines a speech recognition system (if there is voice input) to analyze features such as tone of voice, speech rate, and pauses to assist in judging emotional states. In addition, the system can also indirectly infer emotional states by combining the driver's gestures and operational behaviors (such as sudden steering wheel turns and sudden braking). The identified emotional states are classified and recorded, and each emotion record contains metadata such as emotion type, intensity, occurrence time, and duration.

[0025] By leveraging the time information recorded in emotional interaction logs, the system accurately calculates the driver's continuous driving time, providing a crucial temporal dimension for fatigue assessment. The system extracts the timestamps of each emotional interaction from the logs and traces the driver's continuous driving time based on the timing of emotional changes. Regarding continuous driving time calculation, the system calculates the time difference between the last extended break (e.g., exceeding 20 minutes) and the current emotional interaction. Short breaks (e.g., less than 10 minutes) are ignored and still considered continuous driving. When there are multiple emotional interactions, the system calculates the cumulative driving time for each interaction. The continuous driving time is compared with a preset fatigue threshold (e.g., continuous driving exceeding 4 hours is considered high risk) to generate a driving time risk level label (e.g., normal, alert, dangerous).

[0026] By fusing multi-source data, the system comprehensively assesses the driver's fatigue state and generates a comprehensive state assessment result for each combination of driving images. The system integrates multiple sub-driving state features extracted from previous steps (such as blink frequency, head posture, and number of yawns), combines emotional interaction information (such as fatigue expressions and anxious voices) and continuous driving duration, and uses weighted scoring, machine learning models (such as SVM and random forest) or deep learning networks (such as LSTM+Attention) for comprehensive evaluation.

[0027] The state determination model includes an input layer (various feature vectors), a processing layer (the model learns the correlation between different features and fatigue state), and an output layer (driving state labels, such as awake, mild fatigue, moderate fatigue, and severe fatigue). The system generates a comprehensive state assessment result for each driving image combination. This result can be used to build a driver fatigue model and can also be used for historical comparison in a real-time warning system.

[0028] refer to Figure 4 In step S13, the specific steps are as follows: S131: Collect multiple past driving states, determine the corresponding time nodes based on the time detection of multiple past driving states; determine the driver's first-level state change map based on multiple past driving states and corresponding time nodes; S132: Determine the second-level state change map of the driver based on the combination of multiple previous driving states and corresponding driver driving images; determine the dynamic map of the driver's driving state based on the first-level state change map and the second-level state change map; S133: Based on the recognition of the driver's driving state dynamic map, multiple sub-driving state regions are determined, and the corresponding fatigue driving state is determined based on the recognition of multiple sub-driving state regions; at the same time, the driver's corresponding facial image and fatigue driving actions are collected. S134: Determine a first-level recognition combination of fatigue driving states based on multiple fatigue driving states and the corresponding facial images of the driver; determine a second-level recognition combination of fatigue driving states based on multiple fatigue driving states and the corresponding fatigue driving actions of the driver; and construct a fatigue driving state recognition system based on the first-level and second-level recognition combinations of fatigue driving states.

[0029] In the embodiments of this application, multiple past driving states are collected, and the corresponding time nodes are determined based on the time detection of multiple past driving states; a first-level state change map of the driver is determined based on multiple past driving states and the corresponding time nodes, thus introducing a first-level state change map.

[0030] At this point, the primary task in building a driver fatigue state analysis system is to comprehensively collect data on multiple past driving states. This basic data collection phase requires the integration of multi-dimensional information to provide sufficient data support for subsequent state change analysis. The system collects information from four main data sources: driving behavior data (including vehicle speed changes, steering wheel angle, braking frequency, accelerator pedal operation, etc.), physiological characteristic data (such as heart rate, blinking frequency, head posture, eyelid opening, etc.), environmental data (such as weather, lighting, road type, traffic density, etc.), and subjective feedback (driver's self-assessment of fatigue level, etc.). This data is acquired through multiple collection methods: driving behavior data is acquired through the vehicle's CAN bus; physiological characteristic data is acquired through cameras and infrared sensors; and environmental data is acquired through vehicle GPS and map API.

[0031] Specifically, taking the scenario of a driver driving a truck as an example, the system collected multi-dimensional data from his three historical driving experiences. In terms of driving behavior data, the system recorded key indicators such as steering wheel angle, braking frequency, and vehicle speed changes. In terms of physiological characteristics data, the system collected biological characteristics such as blinking frequency, eyelid opening, and head posture. In terms of environmental data, the system recorded information such as weather conditions (sunny / rainy) and road type (highway / national highway). At the same time, the system also collected the driver's subjective feedback of "mild fatigue" after driving for 2 hours. This multi-dimensional and multi-frequency data constitutes the basic dataset for analyzing changes in the driver's fatigue state.

[0032] After completing the collection of multi-dimensional driving status data, the system needs to determine the corresponding time nodes based on the time detection of these data. These time nodes refer to the moments when there are significant changes in state or key events, such as the moment when fatigue begins to worsen, the moment when driving behavior fluctuates abnormally, or the moment when driving is paused for rest. The system adopts three main time node extraction methods: sliding window detection method (using a fixed time window, such as 5 minutes, to analyze the rate of change in state), threshold triggering method (marking a time node when a certain state indicator, such as blinking frequency, exceeds a preset threshold), and event-driven method (automatically marking time nodes based on specific events such as emergency braking or lane departure). Each time node must include a precise timestamp (accurate to milliseconds), state type (such as "fatigue worsening" or "abnormal behavior"), and relevant state values ​​(such as blinking frequency = 28 times / minute).

[0033] Specifically, the system identified several key time points using the sliding window detection method. At 08:30, the system detected that the driver's blinking frequency increased from 15 times / minute to 22 times / minute, exceeding the preset threshold of 20 times / minute, and thus marked it as a time point of increased fatigue. At 09:15, the system found that the steering wheel angle fluctuation increased, and its standard deviation exceeded the preset threshold, marking it as a time point of abnormal behavior. At 10:00, the driver self-reported "mild fatigue" through the App, and the system recorded this subjective feedback time point. At 10:45, the system detected that the driver's eyelid opening decreased by more than 30%, marking it as a time point of abnormal physiological indicators. These time points provided key reference points for the subsequent construction of the state change diagram.

[0034] After identifying key time points, the system constructs a first-level state change chart for the driver based on multiple past driving states and corresponding time points. This chart is a dynamic change curve with time as the horizontal axis and state indicators as the vertical axis, used to intuitively display the trend of driver state changes over time. The construction process includes three main steps: data normalization (normalizing different state indicators to the 0-1 range for easy comparison), time axis alignment (aligning all state data by timestamp to ensure time consistency), and state curve generation (generating smooth curves using linear interpolation or spline interpolation, and using different colors or line types to distinguish different state indicators). At the same time, the system also performs trend analysis, calculates the state change rate (slope), and marks key change points (such as peaks and troughs). The final generated state change chart has clear chart characteristics: the horizontal axis represents time (such as 08:00-12:00), the vertical axis represents state values ​​(normalized 0-1), includes legends explaining different state indicators (such as fatigue index and attention index), and marks key time points and events.

[0035] Specifically, the system integrates the collected status data and determined time points to generate the first-level status change chart. The horizontal axis of this chart covers the driving period from 08:00 to 12:00, and the vertical axis represents the normalized status value (0 = alert, 1 = severely fatigued). The chart contains two main status curves: the fatigue index curve gradually rises from 0.1 at 08:00 to 0.7 at 12:00, showing the trend of fatigue increasing over time; the attention index curve drops from 0.9 at 08:00 to 0.3 at 12:00, reflecting a continuous decline in attention level. The chart also marks four key time points: 08:30 (fatigue index begins to rise), 09:15 (attention index drops significantly), 10:00 (subjective fatigue feedback point), and 10:45 (fatigue index rises sharply). This status change chart intuitively shows the driver's fatigue status changes during driving, providing an important reference for subsequent fatigue warning systems.

[0036] Furthermore, a second-level state change map of the driver is determined based on a combination of multiple previous driving states and corresponding driver driving images; a dynamic map of the driver's driving state is determined based on the first-level and second-level state change maps, realizing a holistic consideration of the first-level and second-level state change maps and improving the accuracy of the dynamic map of the driver's driving state.

[0037] At this point, by analyzing the driver's driving image combination, a second-level state change map based on image features is constructed, providing an important basis for subsequent comprehensive fatigue state assessment. The system obtains information from two main data sources: past driving states (including time-series driving behavior data, such as steering wheel angle, braking frequency, vehicle speed fluctuations, etc.) and driver driving image combinations (including driver's face images, eye images, head posture images, etc., over continuous time periods). In the image combination recognition stage, the system extracts features from the images, including blinking frequency (unit: times / minute) and eyelid opening (unit: blinking frequency). Key indicators include percentage, number of yawns (unit: times / minute), head pitch angle (unit: degrees), and frequency of gaze deviation (unit: times / minute). Based on these characteristics, the system constructs a second-level state change graph, with the horizontal axis representing time (e.g., 08:00-12:00) and the vertical axis representing the image feature fatigue index (0-1, 0 for alertness and 1 for severe fatigue). The graph plots multiple feature curves, including blink frequency curve, eyelid opening curve, yawn frequency curve, and head posture curve. By comprehensively calculating the image feature fatigue index, a complete second-level state change graph is formed.

[0038] Specifically, during the driving period from 08:00 to 12:00, the system collects image data every 5 seconds. The collected image feature data shows that the driver's blinking frequency increased from 12 times / minute at 08:00 to 28 times / minute at 12:00; eyelid opening decreased from 85% at 08:00 to 55% at 12:00; yawning frequency increased from 1 time / minute at 08:00 to 5 times / minute at 12:00; and head pitch angle increased from 5 degrees at 08:00 to 15 degrees at 12:00. Based on a preset model, the system calculates the image fatigue index for each time point: 08:00 = 0.15 (awake state), 09:00 = 0.25 (mild fatigue), and 10:00 = 0.45 (mild fatigue). At 11:00, the fatigue index was 0.65 (moderate fatigue), and at 12:00, it was 0.75 (moderate fatigue). Based on these data, the system plotted a second-level state change graph, with the horizontal axis representing 08:00-12:00 and the vertical axis representing the image fatigue index (0-1). The curve characteristics show that the fatigue index rises slowly from 08:00-09:00, accelerates from 09:00-10:00, and remains high from 10:00-12:00. The graph also marks two key nodes: 09:30 (yawning frequency exceeds the threshold of 3 times / minute) and 11:00 (eyelid opening is less than 60%). This second-level state change graph intuitively demonstrates the fatigue state change trend based on image features, providing an important reference for subsequent comprehensive evaluation.

[0039] By fusing a first-level state change map based on behavioral data and a second-level state change map based on image features, a more comprehensive and accurate dynamic map of the driver's driving state is constructed. The system employs a weighted fusion algorithm to combine the two state change maps from different sources, with behavioral data (e.g., steering wheel angle, braking frequency) accounting for 40% and image data (e.g., blinking frequency, eyelid opening) accounting for 60%. The fusion formula is: Comprehensive Fatigue Index = 0.4 × Behavioral Fatigue Index + 0.6 × Image Fatigue Index. Based on this formula, the system constructs a dynamic map of the driving state, with the horizontal axis representing time (e.g., 08:00-12:00) and the vertical axis representing the comprehensive fatigue index (0-1). The chart plots a comprehensive fatigue curve and marks key nodes, including the fatigue onset point, fatigue aggravation point, and severe fatigue point. Simultaneously, the system sets warning thresholds: 0.3 for mild fatigue, 0.6 for moderate fatigue, and 0.8 for severe fatigue. The dynamic map of the driving state not only monitors the current driving state in real time but also compares it with historical dynamic maps to identify abnormal fatigue patterns and provide personalized fatigue warnings, such as voice prompts and vibration alerts.

[0040] Specifically, the system merges the behavioral fatigue index data from the first-level state change map (08:00: 0.1, 09:00: 0.2, 10:00: 0.4, 11:00: 0.6, 12:00: 0.7) and the image fatigue index data from the second-level state change map (08:00: 0.15, 09:00: 0.25, 10:00: 0.45, 11:00: 0.65, 12:00: 0.75); through weighted calculation, the system obtains the comprehensive fatigue index for each time point. The values ​​are as follows: 08:00 is 0.13 (0.4×0.1+0.6×0.15), 09:00 is 0.23 (0.4×0.2+0.6×0.25), 10:00 is 0.43 (0.4×0.4+0.6×0.45), 11:00 is 0.63 (0.4×0.6+0.6×0.65), and 12:00 is 0.73 (0.4×0.7+0.6×0.75). Based on these data, the system generates a dynamic graph of the driving status, with the horizontal axis representing the time interval from 08:00 to 12:00. The vertical axis represents the comprehensive fatigue index (0-1). The curve shows that the comprehensive fatigue index rises slowly from 08:00 to 09:00 (0.13→0.23), accelerates from 09:00 to 10:00 (0.23→0.43), and remains high from 10:00 to 12:00 (0.43→0.73). The chart also marks warning thresholds: the mild fatigue line (0.3) is breached at 09:30, the moderate fatigue line (0.6) is breached at 10:45, and the severe fatigue line... The line (0.8) was not breached; based on this dynamic graph, the system detected that the comprehensive fatigue index exceeded 0.3 at 09:30, triggering a mild fatigue warning; at 10:45, it detected that the comprehensive fatigue index exceeded 0.6, triggering a moderate fatigue warning, and advised the driver to rest; at the same time, the system recorded the driver's fatigue pattern for subsequent personalized model optimization. This fused dynamic graph combines the advantages of behavioral data and image features, has higher accuracy and reliability, and provides a more accurate judgment basis for the fatigue driving warning system.

[0041] Furthermore, by analyzing the dynamic driving state graph, the continuous fatigue index change curve is divided into multiple sub-driving state regions with relatively stable characteristics, providing accurate time intervals and state labels for subsequent fatigue state identification. The system reviews the driving state dynamic graph constructed in S132, where the horizontal axis represents time and the vertical axis represents the comprehensive fatigue index (0-1), and the curve represents the change in driver fatigue level over time. The dynamic graph typically contains multiple fluctuation segments, which correspond to different driving state regions. During the sub-driving state region division process, the system divides the dynamic graph into multiple "sub-regions" based on the changing trend of the fatigue index, with each region representing a relatively stable driving state. The division criteria mainly include changes in the fatigue index. The system defines four main fatigue zones based on the following criteria: rate (e.g., slope abrupt change point), duration (each zone lasts at least several minutes), and threshold boundaries (e.g., mild fatigue 0.3, moderate fatigue 0.6, severe fatigue 0.8). The system generates a series of time intervals and corresponding fatigue state labels, such as 08:00–09:15: Awake Zone; 09:15–10:30: Mild Fatigue Zone; 10:30–11:45: Moderate Fatigue Zone; 11:45–12:00: Severe Fatigue Zone.

[0042] Specifically, the system analyzed his driving status dynamics from 08:00 to 12:00; the data showed that from 08:00 to 09:15, the fatigue index slowly increased from 0.1 to 0.28, which was within the alert zone; from 09:15 to 10:30, the fatigue index increased from 0.28 to 0.58, entering the mild fatigue zone; from 10:30 to 11:45, the fatigue index increased from 0.58 to 0.77, reaching the moderate fatigue zone; at 11:45... During the period from 08:00 to 12:00, the fatigue index rose from 0.77 to 0.82, entering the severe fatigue zone. Based on this data, the system identified four sub-driving state zones: 08:00–09:15 as the alert zone, 09:15–10:30 as the mild fatigue zone, 10:30–11:45 as the moderate fatigue zone, and 11:45–12:00 as the severe fatigue zone. This zone division provides an accurate time frame and state benchmark for subsequent fatigue state identification.

[0043] After completing the sub-driving state area division, the system enters the fatigue driving state identification stage, performing a detailed fatigue state judgment for each sub-area, comprehensively considering multiple factors such as duration, trend of change, and fluctuation amplitude. The core of the state identification logic is: if an area is continuously in the moderate or severe fatigue zone, it is judged as a "high-risk fatigue state". The system sets specific judgment rules: if an area is in the moderate fatigue zone for more than 15 consecutive minutes, it is judged as "continuous moderate fatigue"; if an area experiences brief severe fatigue (even if it is only for a few minutes), it is judged as "instantaneous severe fatigue" and immediate intervention is required. Through this refined judgment method, the system assigns a corresponding fatigue state label to each sub-area, such as 08:00–09:15: awake state; 09:15–10:30: mild fatigue state; 10:30–11:45: moderate fatigue state; 11:45–12:00: severe fatigue state.

[0044] Specifically, the system performed a detailed fatigue assessment based on the previously determined sub-driving state zones. The results showed that from 08:00 to 09:15, the driver was alert and at no risk of fatigue; from 09:15 to 10:30, the driver entered a state of mild fatigue, requiring monitoring; from 10:30 to 11:45, the driver reached a state of moderate fatigue, and rest was recommended; from 11:45 to 12:00, the driver entered a state of severe fatigue, requiring immediate intervention. This time-zone-based state assessment method not only accurately identifies... The system can not only identify the driver's current level of fatigue but also predict the trend of fatigue development, providing a scientific basis for timely intervention. The system pays special attention to the moderate fatigue state between 10:30 and 11:45, because this state lasted for 75 minutes, far exceeding the 15-minute threshold, indicating that the driver was in a state of sustained moderate fatigue and needed to rest as soon as possible. Although the severe fatigue state between 11:45 and 12:00 lasted for a shorter period of time, the fatigue index was close to the critical value of 0.8, and the system judged it as instantaneous severe fatigue, requiring immediate intervention.

[0045] The system verifies and supplements the fatigue state determination results based on dynamic images by acquiring multimodal data. Within each sub-driving state area, the system periodically acquires facial images of the driver (e.g., every 30 seconds), focusing on capturing key features such as eye status (frequent blinking, decreased eyelid opening), facial expressions (yawning, frowning, blank expression), and head posture (frequent nodding, head tilting). Simultaneously, the system captures the driver's fatigue driving actions through in-vehicle sensors or cameras, including self-stimulation actions such as rubbing eyes and slapping the face, frequent adjustments to seating posture, and slow or irregular movements of the steering wheel or pedals. The system fuses and analyzes image features and action features to verify the accuracy of the fatigue state determination. If both image and action data support fatigue determination, the area is confirmed as a valid fatigue state.

[0046] Specifically, the system collected detailed facial images and fatigue-related movements. In the mild fatigue zone (09:15–10:30), the collected image data showed a blinking frequency of 15 times / minute (normal is 10-12 times / minute), 3 yawns, and a slight forward head tilt. In the moderate fatigue zone (10:30–11:45), the image characteristics worsened further: the blinking frequency increased to 22 times / minute, the yawning frequency increased to 8 times, and the head posture showed frequent nodding. In the severe fatigue zone (11:45–12:00), the image characteristics reached a dangerous level: the blinking frequency reached as high as 28 times / minute, the yawning frequency reached 12 times, the head posture was significantly drooping, and even signs of microsleep were observed. The fatigue action data collection results also verified the deterioration of fatigue: after 10:30, the driver began to frequently rub his eyes and adjust his seating position; after 11:45, the steering wheel operation became noticeably sluggish, and there were several instances of lane departure. These multimodal data were highly consistent with the fatigue state judgment based on dynamic images, confirming the accuracy of the system's judgment. In particular, during the period from 11:45 to 12:00, the micro-sleep signs shown in the images and the dangerous behaviors such as sluggish steering wheel operation and lane departure captured by motion capture provided strong evidence to support the system's judgment of severe fatigue. This multimodal data fusion analysis method not only improved the accuracy of fatigue state identification but also provided a rich data foundation for the optimization of personalized fatigue early warning models.

[0047] Therefore, a first-level recognition combination of fatigue driving states is determined based on multiple fatigue driving states and the corresponding facial images of the driver. A second-level recognition combination of fatigue driving states is determined based on multiple fatigue driving states and the corresponding fatigue driving actions of the driver. A fatigue driving state recognition system is constructed based on the first-level and second-level recognition combinations, which improves the accuracy of the fatigue driving state recognition system. At the same time, the dynamic image of the driver's driving state is introduced, which takes into account multiple fatigue driving states, the corresponding facial images of the driver, and the corresponding fatigue driving actions, thus improving the accuracy of the fatigue driving state recognition system.

[0048] At this point, by analyzing the driver's facial image features, a fatigue state recognition combination based on physiological manifestations is constructed. The system integrates input data, including multiple fatigue driving states from S133 (such as alert, mild fatigue, moderate fatigue, and severe fatigue) and driver facial images collected during each fatigue state time period. During processing, the system uses advanced computer vision technologies (such as OpenCV, Dlib, or deep learning models) to extract key features from the facial images, mainly including eye features (blink frequency, blink speed, eye closure time PERCLOS), mouth features (yawn frequency, mouth opening degree), and head posture (nodding frequency, head tilt angle). These features are then quantified into numerical indicators, such as blink frequency = blinks per minute, PERCLOS, etc. OS = Percentage of eye closure time out of total time; Yawning frequency = Number of yawns per minute. The system associates these quantitative features with fatigue state to form the first layer of identification: Awake state corresponds to a blinking frequency of 10-12 times / minute, PERCLOS < 10%, and a yawning frequency < 1 time / minute; Mild fatigue corresponds to a blinking frequency of 13-20 times / minute, PERCLOS 10%-20%, and a yawning frequency of 1-3 times / minute; Moderate fatigue corresponds to a blinking frequency of 21-25 times / minute, PERCLOS 20%-30%, and a yawning frequency of 4-6 times / minute; Severe fatigue corresponds to a blinking frequency > 25 times / minute, PERCLOS > 30%, and a yawning frequency > 6 times / minute. The output of this step is the first layer of identification composed of fatigue state and facial feature quantification table.

[0049] Specifically, the system conducted detailed facial image acquisition and analysis between 09:00 and 12:00. During the conscious state from 09:00 to 10:30, the acquired data showed a blinking frequency of 11 times / minute, a PERCLOS of 8%, and a yawning frequency of 0.5 times / minute. During the mild fatigue stage from 10:30 to 11:45, facial features changed significantly: the blinking frequency increased to 18 times / minute, the PERCLOS increased to 15%, and the yawning frequency increased to 2 times / minute. During the moderate fatigue stage from 11:45 to 12:00, facial features further deteriorated: the blinking frequency reached 24 times / minute, the PERCLOS increased to 25%, and the yawning frequency increased to 5 times / minute. These data constitute the driver's first layer of identification, clearly demonstrating the gradual deterioration of facial fatigue characteristics as driving time increases.

[0050] By analyzing driver behavior, a fatigue state recognition system based on action features is constructed. The system also integrates input data, including multiple fatigue driving states and driver behavior collected within each fatigue state time period. During processing, the system extracts action features from onboard sensors (such as steering wheel angle sensors, brake pedal sensors, and lane departure detection systems). These features primarily include steering wheel operation (frequency of angle changes, fine-tuning amplitude), braking behavior (braking frequency, changes in braking force), and lane control (number of lane departures, deviation amplitude). These action features are then quantified into numerical indicators, such as steering wheel adjustment frequency = number of steering wheel fine-tunings per minute, braking frequency = number of brakes per minute, and lane departure frequency = number of lane departures per hour. The system correlates quantitative features with fatigue state to form a second identification combination: Awake state corresponds to steering wheel adjustment frequency <5 times / minute, braking frequency <1 time / minute, and lane departure frequency <1 time / hour; Mild fatigue corresponds to steering wheel adjustment frequency 5-10 times / minute, braking frequency 1-2 times / minute, and lane departure frequency 1-2 times / hour; Moderate fatigue corresponds to steering wheel adjustment frequency 11-15 times / minute, braking frequency 3-4 times / minute, and lane departure frequency 3-4 times / hour; Severe fatigue corresponds to steering wheel adjustment frequency >15 times / minute, braking frequency >4 times / minute, and lane departure frequency >4 times / hour. The output of this step is the second identification combination composed of a fatigue state and motion feature quantification table.

[0051] Specifically, the system synchronously collected the driver's driving action data. During the period of alertness from 9:00 to 10:30, the action data showed: steering wheel adjustment frequency 4 times / minute, braking frequency 0.8 times / minute, and lane departure frequency 0.5 times / hour. During the period of mild fatigue from 10:30 to 11:45, the action characteristics began to change: steering wheel adjustment frequency increased to 8 times / minute, braking frequency increased to 1.5 times / minute, and lane departure frequency rose to 1.5 times / hour. During the period of moderate fatigue from 11:45 to 12:00, the action characteristics deteriorated significantly: steering wheel adjustment frequency reached 13 times / minute, braking frequency increased to 3 times / minute, and lane departure frequency rose to 3 times / hour. These data constituted the driver's second layer of identification, corroborating the facial feature data and jointly reflecting the deterioration of the driver's fatigue state.

[0052] By integrating the recognition combinations obtained from the first two steps, a comprehensive and accurate fatigue driving state recognition system is constructed. The system employs a weighted fusion method to combine the first-level (facial image) and second-level (driving action) recognition combinations. Considering that facial features more directly reflect physiological fatigue, they are assigned a weight of 60%, while action features reflect behavioral fatigue and are assigned a weight of 40%. The comprehensive fatigue index is calculated using the formula "Comprehensive Fatigue Index = (Facial Feature Score × 0.6) + (Action Feature Score × 0.4)". Subsequently, the system establishes fatigue state judgment rules: conscious corresponds to a comprehensive fatigue index < 0.3; mild fatigue corresponds to a comprehensive fatigue index < 0.6; moderate fatigue corresponds to a comprehensive fatigue index < 0.8; and severe fatigue corresponds to a comprehensive fatigue index 0.8. In addition, the system is designed with a dynamic update mechanism to continuously optimize the weights and thresholds based on historical data to adapt to individual differences. The output of this step is a complete fatigue driving state recognition system.

[0053] Specifically, the system underwent detailed data fusion and recognition system construction. During the mild fatigue stage (10:30-11:45), the facial feature score was 0.4, the action feature score was 0.35, and the comprehensive fatigue index was calculated as (0.4×0.6)+(0.35×0.4)=0.38. During the moderate fatigue stage (11:45-12:00), the facial feature score was 0.7, the action feature score was 0.65, and the comprehensive fatigue index was calculated as (0.7×0.6)+(0.65×0.4)=0.68. The recognition system outputs a timeline of the driver's fatigue state: 09: 00:00-10:30 is considered a conscious state (overall fatigue index 0.2); 10:30-11:45 is considered a mild fatigue state (overall fatigue index 0.38); and 11:45-12:00 is considered a moderate fatigue state (overall fatigue index 0.68). This multimodal fusion identification system not only improves the accuracy of fatigue state identification but also provides a scientific basis and technical support for the realization of personalized fatigue early warning systems. Through this comprehensive analysis method, the system can more comprehensively and accurately identify the driver's fatigue state, providing a reliable technical means for timely intervention and ensuring driving safety.

[0054] refer to Figure 5 In step S14, the specific steps are as follows: S141: Real-time monitoring of the driver's truck driving process, and collection of the driver's current facial image. Based on the recognition of the driver's current facial image, multiple facial features are determined. Based on the characteristic shape of multiple facial features and the recognition system of fatigue driving state, the corresponding driving state coefficient is determined. S142: Collect the driver's current interaction information, determine multiple sub-interaction contents of different dimensions based on the identification of the driver's current interaction information, and determine the first fatigue driving coefficient based on the multiple sub-interaction contents and driving state coefficient. S143: Determine the second fatigue driving coefficient based on the driver's current driving action and driving state coefficient, and determine the driver's fatigue driving state based on the mapping relationship between the first fatigue driving coefficient, the second fatigue driving coefficient and fatigue driving state.

[0055] In the embodiments of this application, a real-time monitoring mechanism for the driver is established, which continuously acquires facial images to provide a real-time data basis for subsequent fatigue state recognition. The system continuously monitors the driver through a camera installed in the driver's cab (usually located above the dashboard or near the rearview mirror), with a monitoring frequency of 5 to 10 frames per second to ensure that subtle changes in facial expressions and movement features are captured. The monitoring range includes the driver's entire facial area, and may also include upper body movements when necessary. In the image acquisition process, the image stream captured by the camera is preprocessed, including image denoising, illumination compensation, and face alignment. The aligned image is then fed into a face detection model (such as MTCNN, RetinaFace, etc.) to extract the face region and crop it to a standard size (such as 224×224 pixels). The processed image sequence is stored in a cache for subsequent feature extraction.

[0056] Multiple key features are extracted from preprocessed facial images to provide quantitative basis for fatigue state recognition. The system uses deep learning models (such as OpenFace, MediaPipe, DeepFace, etc.) to extract multiple key features from aligned facial images. The main extracted features include eye features (blinking frequency, PERCLOS duration of eye closure, pupil diameter change), mouth features (number of yawns, degree of mouth opening), facial expression features (frequency of frowning, degree of drooping corners of the mouth), and head posture (nodding frequency, head tilt angle). Each feature is then quantified into a numerical value, such as blinking frequency in times / minute, eye closure duration in seconds, number of yawns in times / minute, and head tilt angle in degrees.

[0057] The system extracts multiple facial features and transforms them into a comprehensive driving state coefficient to determine the driver's current fatigue state. It employs a multi-classification or regression model trained on historical data as the fatigue driving state recognition system. The input is multiple facial features, and the output is the driving state coefficient. The driving state coefficient is a value between 0 and 1, representing the driver's fatigue level: 0-0.3 indicates alertness, 0.3-0.6 indicates mild fatigue, 0.6-0.8 indicates moderate fatigue, and 0.8-1.0 indicates severe fatigue. The coefficient is calculated using methods including weighted average, machine learning models (such as SVM, random forest), or deep learning. Multimodal learning models (such as LSTM and Transformer) can be used. For example, different weights can be assigned to different features, and a weighted average can be calculated: Driving state coefficient = w1 × blink frequency + w2 × eye closure duration + w3 × number of yawns + ... Through this real-time monitoring and recognition process, the system can accurately capture changes in the driver's fatigue state, providing reliable technical support for timely early warning measures. This recognition method based on multimodal feature fusion not only improves the accuracy of fatigue state recognition but also adapts to individual differences among drivers, laying a solid foundation for building a personalized fatigue driving early warning system.

[0058] Furthermore, by collecting real-time driver interaction information through multiple channels, the system provides behavioral data support for fatigue assessment. The system obtains interaction information from four main sources: in-vehicle system interaction (including GPS navigation operation, air conditioning adjustment, radio channel switching, etc.), vehicle control interaction (such as turn signal usage frequency, windshield wiper operation, window control, etc.), communication device interaction (such as in-vehicle phone use, walkie-talkie calls, mobile phone operation, etc.), and driver assistance system interaction (such as lane keeping system intervention, adaptive cruise control adjustment, etc.). The collection methods are diverse, including obtaining vehicle control signals through the CAN bus, obtaining human-machine interaction data through the in-vehicle system API, collecting voice interaction information through the in-vehicle microphone, and capturing manual operation behavior through the camera.

[0059] The system performs multi-dimensional analysis on the collected interaction information to extract key sub-interaction content that reflects the driver's state. The system classifies the interaction information from four dimensions: interaction frequency (number of interactions per unit time), interaction complexity (complexity of operation steps), interaction initiative (active initiation vs. passive response), and interaction effectiveness (whether the operation achieves the expected effect). The identification methods include time sequence analysis (analyzing the time distribution characteristics of interaction behavior), pattern recognition (identifying normal vs. abnormal interaction patterns), and context association (judging the rationality of the interaction in conjunction with the driving environment).

[0060] By integrating multi-dimensional interaction features with driving state coefficients, a more comprehensive and accurate first fatigue driving coefficient is calculated. The system adopts a weight allocation mechanism, assigning different weights to the correlation between sub-interaction content and fatigue in different dimensions: the interaction frequency dimension has the highest weight (0.4), followed by interaction effectiveness (0.3), initiative (0.2) and complexity (0.1). The calculation method normalizes the sub-interaction content of each dimension, performs a weighted average according to the weights, and combines the driving state coefficient for comprehensive calculation.

[0061] Specifically, assuming the driving state coefficient (from S141) is 0.659, the system calculates the first fatigue driving coefficient according to the following steps: the interaction frequency dimension is (0.07 / 0.2+0 / 0.1+0 / 0.05) / 3=0.12, the interaction complexity dimension is (0.9+0.2) / 2=0.55, the interaction initiative dimension is (0.1+(1-2.5 / 1)) / 2=0.05, and the interaction effectiveness dimension is (0.6+0.7) / 2=0.65; the weighted interaction fatigue index is calculated as: 0.4×0.12+0.1×0.55+0.2×0.05+0.3×0.65=0.278; combined with the driving state coefficient, the first fatigue driving coefficient is calculated as: 0.6×0.659+0.4×0.278=0.5086.

[0062] Therefore, the second fatigue driving coefficient is determined based on the driver's current driving actions and driving state coefficient, and the driver's fatigue driving state is determined based on the mapping relationship between the first fatigue driving coefficient, the second fatigue driving coefficient and fatigue driving state, thus providing accuracy in determining the driver's fatigue driving state.

[0063] At this point, by analyzing the driver's real-time driving action data and combining it with the driving state coefficient, a second fatigue driving coefficient is calculated to more comprehensively assess the driver's fatigue level. The system collects multi-dimensional driving action data, including steering wheel operation (steering angle, steering frequency, fine adjustment range, etc.), pedal operation (accelerator pedal change rate, brake pedal usage frequency, force change, etc.), vehicle speed control (vehicle speed fluctuation range, cruise control usage, etc.), and lane keeping (lane departure frequency, departure range, correction reaction time, etc.). Key features are extracted from these raw data, such as steering wheel stability (calculating the standard deviation of the steering wheel angle), operation reaction time (the time interval from the occurrence of the stimulus to the driver's operation), action smoothness (assessing the smoothness of operation through the rate of change of acceleration), and action consistency (comparing the consistency between the current action and the driver's normal driving mode).

[0064] The calculation process of the second fatigue driving coefficient includes normalizing each driving action feature (range 0-1), calculating the fatigue weight of each feature according to the fatigue driving feature database, and performing a weighted calculation in combination with the driving state coefficient: Second fatigue coefficient = w1 × action feature 1 + w2 × action feature 2 + ...; and fusing with the driving state coefficient: The final second fatigue coefficient = α × driving state coefficient + (1-α) × action fatigue index, where α is the weight coefficient (usually taken as 0.6-0.8).

[0065] By integrating the first fatigue driving coefficient (based on interactive information) and the second fatigue driving coefficient (based on driving actions), and combining the mapping relationship of fatigue driving states, the system ultimately determines the driver's fatigue driving state. The system uses a weighted average method to integrate the two fatigue coefficients: comprehensive fatigue coefficient = β × first fatigue coefficient + (1-β) × second fatigue coefficient, where β is the weighting coefficient (usually taken as 0.5-0.7). The system determines the state based on the preset fatigue state mapping relationship: conscious state (comprehensive fatigue coefficient < 0.4), mild fatigue (0.4 ≤ comprehensive fatigue coefficient < 0.6), moderate fatigue (0.6 ≤ comprehensive fatigue coefficient < 0.8), and severe fatigue (comprehensive fatigue coefficient ≥ 0.8). The state determination logic also includes calculating the comprehensive fatigue coefficient, determining the fatigue level based on the mapping relationship, and confirming the state by combining time trend analysis (such as the fatigue coefficient change rate), and finally outputting the fatigue driving state determination result.

[0066] refer to Figure 6 In step S15, the specific steps are as follows: S151: The fatigue driving state recognition system is embedded in the truck's control system. The fatigue driving state recognition system outputs a corresponding fatigue driving signal, and the fatigue driving duration is determined based on the fatigue driving signal, the driver's fatigue driving state, and the driver's continuous driving time. S152: Collect the driver's current facial image, and determine the safe driving mode of the truck based on the driver's current facial image, the driver's decompression method and the corresponding fatigue driving duration. S153: In the safe driving mode of the truck, multiple driving control items are determined based on the recognition of the safe driving mode of the truck. The current safe driving section of the truck is determined based on the multiple driving control items and the remaining driving path. Safe stopping measures are determined based on the current safe driving section of the truck, the driver's fatigue driving state and the driver's eye image.

[0067] In the embodiments of this application, the fatigue driving recognition system is deeply integrated into the truck control system to achieve real-time and efficient fatigue state monitoring. In terms of system architecture design, the fatigue driving recognition system is integrated as an independent module in the truck ECU (electronic control unit), connected to the vehicle CAN bus, and can acquire vehicle data such as vehicle speed, steering angle, and throttle opening, and communicate with hardware devices such as cameras and sensors through a dedicated interface. Regarding the real-time monitoring mechanism, the system adopts a multi-threaded processing architecture: the main thread is responsible for processing the real-time video stream (30fps), the auxiliary thread analyzes historical data trends, and the monitoring thread checks the system's operating status; the data processing flow follows a complete chain of "raw data → preprocessing → feature extraction → status recognition → signal output"; to ensure system reliability, a fault safety mechanism is also designed. When the identification system malfunctions, it automatically switches to the basic safety mode and records the fault code, which is then uploaded to the cloud diagnostic system.

[0068] After the system completes fatigue state identification, the result needs to be converted into a standardized fatigue driving signal for output. The signal generation has strict conditions: the signal output is triggered when the fatigue state confidence exceeds the threshold (e.g., 85%), and the signal is locked when the same fatigue level is detected 3 times in a row. The signal format adopts the standard CAN signal format, which includes ID and data fields (level, confidence, timestamp, check code).

[0069] By integrating multi-dimensional data, the system accurately calculates the driver's fatigue driving time. The system uses a weighted average method to calculate the comprehensive fatigue index: Comprehensive fatigue index = w1 × fatigue level + w2 × confidence level + w3 × continuous driving coefficient, where the weights can be adjusted according to the actual situation (e.g., w1=0.4, w2=0.3, w3=0.3). The formula for calculating the continuous driving coefficient is continuous driving coefficient min(current continuous driving time / legal maximum driving time, 1), where the legal maximum driving time is set according to local regulations (e.g., 4 hours).

[0070] The fatigue driving duration calculation model adopts a piecewise function: when the comprehensive fatigue index is <0.3, the fatigue driving duration is 0; when 0.3≤comprehensive fatigue index<0.7, the fatigue driving duration is the current time minus the time of the first fatigue detection; when the comprehensive fatigue index is ≥0.7, the fatigue driving duration is the cumulative time plus a multiple of the continuous driving duration (usually taken as 1.2-1.5); the system also has a dynamic adjustment mechanism that recalculates the fatigue driving duration every 5 minutes.

[0071] Furthermore, high-precision image acquisition technology is used to obtain real-time facial images of the driver, providing a high-quality data source for subsequent fatigue state analysis. The system uses an in-vehicle high-definition infrared camera (1080p resolution, 30fps frame rate) and is equipped with active infrared illumination technology to ensure clear imaging even in low-light environments such as nighttime or tunnels. The camera is installed in the center of the dashboard above the steering wheel, about 60-80cm away from the driver's face, to obtain the best shooting angle. The image preprocessing process follows a complete chain of "original image noise reduction, face detection, key point localization, and region cropping".

[0072] In terms of technical implementation, the system uses MTCNN (Multi-task Convolutional Network) based on deep learning for face detection, accurately locating 68 facial key points (including the contours of the eyes, eyebrows, nose, and mouth); the images are compressed in JPEG format (85% quality) and transmitted via in-vehicle Ethernet (100Mbps); in terms of data storage, the system uses a circular storage mechanism to save the facial image data of the most recent 30 minutes, abnormal state images are automatically marked and stored for a long time, while normal state images are sampled and saved at 5-minute intervals, which ensures data integrity and optimizes storage space utilization.

[0073] After acquiring high-quality facial images, the system further analyzes the self-regulation behaviors adopted by drivers to alleviate fatigue, namely "decompression methods." Decompression methods are systematically divided into three categories: active rest (including resting with eyes closed, head support, stretching exercises, etc.), stimulus maintenance (including forced eye opening, facial massage, dietary stimulation, etc.), and environmental regulation (including air conditioning adjustment, window opening, music adjustment, etc.). The recognition algorithm follows the process of "image input, behavior detection, feature extraction, behavior classification, and confidence calculation." The system performs excellently in key technical indicators: behavior detection accuracy ≥92%, processing latency <200ms, and behavior duration threshold adjusted between 2 and 5 seconds according to behavior type.

[0074] Based on the analysis results of fatigue index, continuous driving time, and decompression methods, the system dynamically adjusts the safe driving mode to maximize driving safety. The safe driving modes are divided into four levels: Normal Mode (Mode-N) is suitable for fatigue index <0.3 and continuous driving time <2 hours, with no restrictions on vehicle parameters and only routine monitoring; Warning Mode (Mode-W) is suitable for fatigue index 0.3 ≤ fatigue index <0.6 or 2 hours ≤ continuous driving time <4 hours, with a vehicle speed limit of 90 km / h and an acceleration limit of 1.5 m / s², and the system provides a reminder every 30 minutes and enhanced monitoring; Restriction Mode (Mode-R) is suitable for fatigue index 0.6 ≤ fatigue index <0.8 or 4 hours ≤ continuous driving time <6 hours, with a vehicle speed limit of 70 km / h and an acceleration limit of 1.0 m / s², and the system provides a reminder every 15 minutes and a mandatory rest suggestion; Protection Mode (Mode-P) is suitable for fatigue index ≥0.8 or continuous driving time ≥6 hours, with a vehicle speed limit of 50 km / h and an acceleration limit of 0.5 m / s².

[0075] The system immediately suggests stopping and automatically searches for a service area. The mode decision algorithm comprehensively considers fatigue index, duration weight, and decompression method correction factors, and adopts a dynamic adjustment mechanism: it is re-evaluated every 5 minutes. When an effective decompression method is detected, the mode level can be temporarily lowered by one level; while if an invalid decompression method is detected three times in a row, the mode level will be raised by one level. This multi-factor, dynamic safe driving mode adjustment mechanism can not only provide personalized protection measures according to the driver's actual condition, but also minimize interference with normal driving while ensuring safety through intelligent mode switching, achieving the best balance between safety and practicality.

[0076] Therefore, in the safe driving mode of trucks, multiple driving control items are determined based on the identification of the truck's safe driving mode. The current safe driving section of the truck is determined based on the multiple driving control items and the remaining driving path. Safe stopping measures are determined based on the current safe driving section of the truck, the driver's fatigue driving state, and the driver's eye image, which improves the accuracy of safe stopping measures. At the same time, the identification of the driver's fatigue driving state is introduced, and the safety management of trucks is realized. This further realizes the overall consideration of the truck's safe driving mode, the remaining driving path, and the driver's eye image, and improves the accuracy of safe stopping measures.

[0077] At this time, based on different safe driving modes, the system intelligently identifies and implements corresponding controlled driving items to effectively intervene in fatigued driving. The system divides safe driving modes into three levels: warning mode (mild fatigue) corresponds to the lowest intervention level, restriction mode (moderate fatigue) corresponds to the medium intervention level, and emergency mode (severe fatigue) corresponds to the highest intervention level. The process of identifying controlled driving items follows a complete chain of "safety mode identification → controlled item matching → control parameter adjustment → execution strategy generation".

[0078] In Warning Mode, the system implements five controlled measures: speed limit (maximum speed reduced by 10%), acceleration response (accelerator pedal sensitivity reduced by 15%), cruise control (automatic activation of Intelligent Cruise Control (ACC)), lane keeping assist (enhanced LKA system sensitivity), and reminder frequency (fatigue reminder interval shortened to 5 minutes). In Restriction Mode, the number of controlled items increases to seven, including stricter speed limit (reduced by 20%), acceleration response (reduced by 30%), mandatory activation of ACC and increased following distance, continuous activation of LKA system, more frequent reminders (interval of 2 minutes), and prohibition of active overtaking. Emergency Mode has the most stringent intervention, including nine controlled measures: significantly reduced speed (reduced by 40%), significantly reduced acceleration response (60%), mandatory activation of ACC and setting of minimum following distance, maximum LKA system sensitivity, continuous voice and vibration reminders, complete prohibition of lane changing and overtaking, and automatic deceleration to 60 km / h when there is no vehicle in front.

[0079] In terms of technical implementation, the system adjusts control parameters via the CAN bus, activates ECU software limitation strategies, dynamically configures ADAS system parameters, and displays the system status in real time on the HMI interface. The data update mechanism ensures that the status of the controlled items is updated every 100ms, the response time for control parameter adjustment is <200ms, and the system status synchronization delay is <50ms, thus achieving precise and real-time control of driving behavior.

[0080] After identifying the controlled driving items, the system further combines the remaining driving route information to intelligently assess and determine the current safe driving section; the route analysis process follows a complete chain of "route information acquisition → road segment feature extraction → controlled item matching → safe road segment calculation".

[0081] The system integrates multi-source path data, including real-time path data from navigation systems, high-precision map (HDMap) information, real-time road condition updates, and historical driving behavior data. Road segment feature extraction parameters cover ten key dimensions: road grade (expressway / national highway / provincial highway / county road / township road), number of lanes (single lane / two lane / multi-lane), speed limit information, road curvature (curve radius and angle), slope information (uphill / downhill percentage), traffic density (vehicle density index), weather conditions (sunny / rainy / snowy / foggy), visibility (current visibility distance), road surface conditions (dry / slippery / icy), and service area distribution (distance to the next service area). The safe driving road segment calculation model uses a multi-factor weighted function: Safe Road Segment Index = f(Road Grade, Number of Lanes, Speed ​​Limit Information, Road Curvature, Slope Information, Traffic Density, Weather Conditions, Visibility, Road Surface Conditions, Service Area Distribution). Where f = w1×R + w2×L + w3×S + w4×C + w5×G + w6×T + w7×W + w8×V + w9×P + w 10 ×D, the weighting coefficient is dynamically adjusted according to the current safe driving mode.

[0082] Based on the calculation results, the system divides road segments into four safety levels: Level A road segments (safety index ≥ 0.8) are suitable for the current driving mode; Level B road segments (0.6 ≤ safety index < 0.8) require additional monitoring; Level C road segments (0.4 ≤ safety index < 0.6) require adjustments to the driving strategy; and Level D road segments (safety index < 0.4) are not suitable for continued driving. The system adopts a dynamic adjustment mechanism, reassessing the road segment safety index every 1 kilometer, updating it in real time when encountering sudden road conditions, and adjusting the assessment standards according to changes in the driver's state to ensure the real-time nature and accuracy of road segment safety assessments.

[0083] After completing the road safety assessment, the system further combines the driver's fatigue state and eye characteristics to intelligently determine the most appropriate safe stopping measures. The decision-making process for safe stopping measures follows a complete chain of "road safety assessment → fatigue state analysis → eye characteristic extraction → stopping measure decision". The system establishes a correlation model between fatigue state and eye characteristics by analyzing nine key eye image parameters: blink frequency (number of blinks per minute), blink duration (duration of a single blink), fixation distribution (front / left rearview mirror / right rearview mirror / instrument panel), fixation duration (duration of a single fixation), saccade amplitude (angle of eye movement), pupil diameter (current pupil size), pupil change rate (speed of pupil size change), eyelid opening (distance between upper and lower eyelids), and eyelid opening change rate (speed of eyelid opening change).

[0084] Fatigue severity = g(blink frequency, blink duration, fixation distribution, fixation duration, saccade amplitude, pupil diameter, pupil change rate, eyelid opening, eyelid opening change rate), where g = a1×F + a2×D + a3×G + a4×T + a5×S + a6×P + a7×C + a8×E + a9×R.

[0085] Based on the comprehensive evaluation results, the system categorizes safe stopping measures into three levels: Mild fatigue stopping measures include suggested stopping (voice prompts suggesting rest at the next service area), route planning (automatic navigation to the nearest service area), time suggestion (suggested rest of 15-20 minutes), and wake-up prompts (playing soft music or voice dialogue); Moderate fatigue stopping measures are upgraded to mandatory stopping (the system mandates stopping at a safe location), route planning (automatic navigation to the nearest safe stopping point), time suggestion (mandatory rest of 30-45 minutes), wake-up prompts (enhanced reminders, including vibration + voice + light), and status monitoring (reassessing fatigue status after rest).

[0086] The most stringent measures are for stopping due to severe fatigue, including emergency stopping (immediately finding the nearest safe location for emergency stopping), route planning (prioritizing emergency parking lanes or service areas), time recommendations (mandatory rest of more than 60 minutes), wake-up reminders (comprehensive and high-intensity reminders), status monitoring (multiple assessments to confirm recovery status), and emergency contact (contacting the dispatch center or emergency services when necessary). In selecting stopping points, the system comprehensively considers six key factors: distance (distance to the nearest stopping point), safety (stop point safety level), facility completeness (rest area / food / restroom), accessibility (whether complex operations are required to reach the stop), weather impact (weather conditions at the stopping point), and traffic impact (the degree of impact of stopping on traffic).

[0087] The system establishes a comprehensive implementation monitoring mechanism, including real-time monitoring of the execution status of parking measures, recording of driver response behavior, tracking and assessment of fatigue status after parking, and emergency response plans for abnormal situations. This ensures the effective implementation of safe parking measures and the full recovery of the driver's condition. This multi-dimensional and intelligent safe parking decision-making system can not only provide personalized parking suggestions based on the driver's actual condition and road conditions, but also maximize the safety of the driver and vehicle through precise parking point selection and comprehensive follow-up monitoring.

[0088] Please see Figure 7 , Figure 7 This is a schematic diagram of the structural composition of the driver fatigue driving state detection system in an embodiment of the present invention; the driver fatigue driving state detection system includes: The driving image combination module 21 is used to determine multiple driving image combinations based on multiple driving images of the driver and the corresponding image acquisition time in the driver's previous driving events; The previous driving state module 22 is used to determine multiple sub-driving state features of the driver based on the recognition of various driving image combinations; and to determine the previous driving state corresponding to the driving image combination based on the multiple sub-driving state features, the driver's emotional interaction information and the driver's continuous driving time. The recognition system module 23 is used to determine the dynamic map of the driver's driving state based on multiple previous driving states, the corresponding combination of driver driving images and the corresponding time nodes, and to determine multiple fatigue driving states based on the dynamic map of the driving state; and to construct a fatigue driving state recognition system based on multiple fatigue driving states, the driver's corresponding facial image and the corresponding fatigue driving action. The fatigue driving status module 24 is used to determine the corresponding driving status coefficient based on the driver's current facial image and the fatigue driving status recognition system. Based on the driving status coefficient, the driver's current interaction information and the corresponding current driving action, the driver's fatigue driving status is determined to trigger the safety control of the truck. The safe stopping measures module 25 is used to determine the safe driving mode of the truck based on the driver's current facial image and the driver's continuous driving time when the driver is in a state of fatigue driving, and to determine safe stopping measures based on the safe driving mode of the truck, the remaining driving route and the driver's eye image.

[0089] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A method for detecting driver fatigue, characterized in that, include: In the driver's past driving events, multiple driving image combinations are determined based on multiple driving images of the driver and the corresponding image acquisition time; Based on the recognition of various driving image combinations, multiple sub-driving state features of the driver are determined; the previous driving state corresponding to the driving image combination is determined according to the multiple sub-driving state features, the driver's emotional interaction information and the driver's continuous driving duration; the multiple sub-driving state features include blink frequency, head posture and number of yawns; Based on multiple past driving states, the corresponding driver driving image combination, and the corresponding time points, a dynamic diagram of the driver's driving state is determined, and multiple fatigue driving states are determined based on this dynamic diagram of the driving state. A fatigue driving state recognition system is constructed based on multiple fatigue driving states, the driver's corresponding facial images, and the corresponding fatigue driving actions. The driving status dynamic map not only monitors the current driving status in real time, but also compares it with historical dynamic maps to identify abnormal fatigue patterns; The driver's current facial image and fatigue driving status recognition system determine the corresponding driving status coefficient. Based on this driving status coefficient, the driver's current interaction information and corresponding current driving actions, the driver's fatigue driving status is determined to trigger the truck's safety control. Fatigue driving states include mild fatigue, moderate fatigue, and severe fatigue. The process includes: real-time monitoring of the driver's truck driving process and acquisition of the driver's current facial image; identification of multiple facial features based on the driver's current facial image; determination of corresponding driving state coefficients based on the characteristic shape of these multiple facial features and a fatigue driving state recognition system; acquisition of the driver's current interaction information; identification of multiple sub-interaction contents in different dimensions based on the driver's current interaction information; determination of a first fatigue driving coefficient based on the multiple sub-interaction contents and the driving state coefficient; determination of a second fatigue driving coefficient based on the driver's current driving actions and the driving state coefficient; and determination of the driver's fatigue driving state based on the mapping relationship between the first fatigue driving coefficient, the second fatigue driving coefficient, and the fatigue driving state. The driving state coefficient is a value between 0 and 1, representing the driver's fatigue level: 0-0.3 indicates a conscious state, 0.3-0.6 indicates mild fatigue, 0.6-0.8 indicates moderate fatigue, and 0.8-1.0 indicates severe fatigue. In the first fatigue driving coefficient, different weights are assigned based on the correlation between different dimensions of sub-interaction content and fatigue: interaction frequency has the highest weight, followed by interaction effectiveness, initiative, and complexity. The sub-interaction content of each dimension is normalized, and a weighted average is calculated based on the weights, combined with the driving state coefficient for comprehensive calculation. In the second fatigue driving coefficient, each driving action feature is normalized, and the fatigue weight of each driving action feature is calculated based on the fatigue driving feature database, combined with the driving state coefficient for weighted calculation, and then fused with the driving state coefficient. In the driver's fatigue driving state, a weighted average method is used to combine the first fatigue driving coefficient and the second fatigue driving coefficient to determine the comprehensive fatigue coefficient, which is then judged according to a preset fatigue state mapping relationship. The comprehensive fatigue coefficient is calculated, the fatigue level is determined based on the mapping relationship, and the state is confirmed by combining time trend analysis, finally outputting the fatigue driving state judgment result. When the driver is in a state of fatigue, the safe driving mode of the truck is determined based on the driver's current facial image and the duration of the driver's continuous driving. The safe stopping measures are determined based on the safe driving mode of the truck, the remaining driving route, and the driver's eye image. The safe stopping measures include mild fatigue stopping measures, moderate fatigue stopping measures, and severe fatigue stopping measures.

2. The method for detecting driver fatigue driving state according to claim 1, characterized in that, The method involves determining multiple driving image combinations based on the driver's past driving events and the corresponding image acquisition times, including: Collect the truck driving database of drivers, determine the driver's past driving events based on the detection of the truck driving database, determine multiple driving images of the driver based on the tracing of the past driving events, and mark the corresponding image acquisition time; The driver's past driving time period is determined by tracing the image acquisition time. The driver's driving signal is determined by matching the past driving time period with the truck driving database. Multiple driving image combinations are determined based on the driver's driving signal, multiple driving images of the driver, and the corresponding image acquisition time.

3. The method for detecting driver fatigue driving state according to claim 1, characterized in that, The process involves identifying multiple sub-driving state features of the driver based on the recognition of various driving image combinations; and determining the previous driving state corresponding to the driving image combination based on the multiple sub-driving state features, the driver's emotional interaction information, and the driver's continuous driving duration, including: Collect various driving image combinations and trigger the recognition of each driving image combination. Determine the driver's driving state events based on the recognition of each driving image combination. Determine multiple sub-driving state events based on the detection of the driver's driving state events. Determine the sub-driving state features corresponding to the driver based on the recognition of each sub-driving state event, so as to collect multiple sub-driving state features of the driver. In each combination of driving images, the driver's emotional interaction information is determined based on the selection of the driving image combination, and the driver's continuous driving time is determined based on the tracing of the interaction time corresponding to the driver's emotional interaction information. Based on multiple sub-driving state features, the driver's emotional interaction information and the driver's continuous driving time, the previous driving state corresponding to the driving image combination is determined.

4. The method for detecting driver fatigue driving state according to claim 1, characterized in that, The driver's driving state dynamic map is determined based on multiple past driving states, corresponding driver driving image combinations, and corresponding time nodes, and multiple fatigue driving states are determined based on the driving state dynamic map. A fatigue driving state recognition system is constructed based on multiple fatigue driving states, the driver's corresponding facial images, and the corresponding fatigue driving actions, including: Collect multiple past driving states, determine the corresponding time nodes based on the time detection of multiple past driving states; determine the driver's first state change map based on multiple past driving states and corresponding time nodes; The second level of driver state change map is determined by combining multiple previous driving states and corresponding driver driving images; the dynamic map of driver driving state is determined based on the first level of driver state change map and the second level of driver state change map.

5. The method for detecting driver fatigue driving state according to claim 4, characterized in that, The driver's driving state dynamic map is determined based on multiple past driving states, corresponding driver driving image combinations, and corresponding time nodes, and multiple fatigue driving states are determined based on the driving state dynamic map. A fatigue driving state recognition system is constructed based on multiple fatigue driving states, the driver's corresponding facial images, and the corresponding fatigue driving actions. This system also includes: Multiple sub-driving state regions are determined based on the recognition of the driver's driving state dynamic map, and the corresponding fatigue driving state is determined based on the recognition of multiple sub-driving state regions; at the same time, the driver's facial image and fatigue driving actions are collected. A first-level recognition combination of fatigue driving states is determined based on multiple fatigue driving states and the corresponding facial images of the driver. A second-level recognition combination of fatigue driving states is determined based on multiple fatigue driving states and the corresponding fatigue driving actions of the driver. A fatigue driving state recognition system is constructed based on the first-level and second-level recognition combinations of fatigue driving states.

6. The method for detecting driver fatigue driving state according to claim 1, characterized in that, When the driver is in a state of fatigue, the safe driving mode of the truck is determined based on the driver's current facial image and the duration of continuous driving. Safe stopping measures are then determined based on the safe driving mode, the remaining driving route, and the driver's eye contact image, including: The fatigue driving state recognition system is embedded in the truck's control system. The fatigue driving state recognition system outputs a corresponding fatigue driving signal, and the fatigue driving duration is determined based on the fatigue driving signal, the driver's fatigue driving state, and the driver's continuous driving time.

7. The method for detecting driver fatigue driving state according to claim 6, characterized in that, The method of determining a safe driving mode for the truck based on the driver's current facial image and the duration of continuous driving when the driver is fatigued, and determining safe stopping measures based on the truck's safe driving mode, the remaining driving route, and the driver's eye image, also includes: Collect the driver's current facial image, and determine the safe driving mode of the truck based on the driver's current facial image, the driver's decompression method and the corresponding fatigue driving duration; In the safe driving mode of the truck, multiple driving control items are identified based on the identification of the safe driving mode of the truck. The current safe driving section of the truck is determined based on the multiple driving control items and the remaining driving path. Safe stopping measures are determined based on the current safe driving section of the truck, the driver's fatigue driving state and the driver's eye image.

8. A driver fatigue driving state detection system, characterized in that, The driver fatigue driving state detection system is applied to the driver fatigue driving state detection method as described in any one of claims 1-7, and the driver fatigue driving state detection system includes: The driving image combination module is used to determine multiple driving image combinations based on multiple driving images of the driver and the corresponding image acquisition time in the driver's past driving events; The previous driving state module is used to determine multiple sub-driving state features of the driver based on the recognition of various driving image combinations; and to determine the previous driving state corresponding to the driving image combination based on the multiple sub-driving state features, the driver's emotional interaction information and the driver's continuous driving duration. The recognition system module is used to determine the dynamic map of the driver's driving state based on multiple past driving states, the corresponding combination of driver driving images, and the corresponding time points, and to determine multiple fatigue driving states based on the dynamic map of the driving state; and to construct a fatigue driving state recognition system based on multiple fatigue driving states, the driver's corresponding facial images, and the corresponding fatigue driving actions. The fatigue driving status module is used to determine the corresponding driving status coefficient based on the driver's current facial image and the fatigue driving status recognition system. Based on the driving status coefficient, the driver's current interaction information and the corresponding current driving action, the driver's fatigue driving status is determined to trigger the safety control of the truck. The safe parking measures module is used to determine the safe driving mode of the truck based on the driver's current facial image and the driver's continuous driving time when the driver is in a state of fatigue driving. It then determines safe parking measures based on the truck's safe driving mode, the remaining driving route, and the driver's eye image.

Images