Driver fatigue state intervention method, system, vehicle and readable storage medium

Abstract

This application relates to the field of vehicle control technology and discloses a method, system, vehicle, and readable storage medium for intervening in driver fatigue. The intervention method includes: real-time acquisition of multi-dimensional modal raw data related to the driver and vehicle state, including visual modal data, behavioral modal data, and physiological modal data; feature encoding processing of the visual modal data, behavioral modal data, and physiological modal data to obtain corresponding visual feature vectors, behavioral feature vectors, and physiological feature vectors; acquisition of current driving scenario state parameters; concatenation of the visual feature vectors, behavioral feature vectors, and physiological feature vectors with the scenario state parameters and inputting them into a fatigue model for inference, outputting the fatigue level; comparison of the fatigue level with a preset intervention threshold, and triggering the vehicle to perform an intervention action on the driver when the comparison result meets the triggering condition.

Description

Technical Field

[0001] This application relates to the field of vehicle control technology, and in particular to a method, system, vehicle, and readable storage medium for intervening in driver fatigue. Background Technology

[0002] With the rapid development of smart wearable devices and in-vehicle driving assistance systems, real-time status monitoring technology based on physiological signals or motion characteristics is widely used in scenarios such as fatigue driving warning, health monitoring, and human-computer interaction. In existing technologies, common status recognition schemes mainly rely on single sensors, such as eye-tracking cameras, heart rate sensors, or accelerometers, to collect data and combine it with preset static threshold rules for judgment. For example, drowsiness is determined by blinking frequency falling below a certain fixed threshold, or an alarm is triggered based on head pitch angle exceeding a set angle. While these methods are effective under ideal laboratory conditions, they face significant limitations in practical applications: when users wear sunglasses, visible light eye-tracking cameras struggle to accurately capture pupil and eyelid movements; when vehicles travel on bumpy roads, accelerometer and gyroscope outputs are easily affected by high-frequency vibrations, leading to distorted attitude estimation; furthermore, different individuals exhibit significant differences in physiological parameters such as baseline heart rate, blink amplitude, and natural head posture, and fixed thresholds cannot accommodate this heterogeneity. Therefore, traditional solutions rely on a single sensor or simple rule thresholds, resulting in high false alarm rates and poor robustness in real and complex scenarios such as wearing sunglasses, bumpy roads, and individual physiological differences. Summary of the Invention

[0003] In view of this, embodiments of this application provide a method, system, vehicle, and readable storage medium for intervening in driver fatigue, which can effectively solve the problems of high false alarm rate and poor robustness in real and complex scenarios such as wearing sunglasses, bumpy roads, and individual physiological differences caused by traditional solutions relying on a single sensor or simple rule thresholds.

[0004] In a first aspect, embodiments of this application provide a method for intervening in driver fatigue, the method comprising: According to the preset duration of the sliding analysis window, multidimensional modal raw data related to the driver and vehicle status are acquired in real time from the vehicle sensor system. The multidimensional modal raw data includes visual modal data, behavioral modal data and physiological modal data. The visual modality data, the behavioral modality data, and the physiological modality data are respectively subjected to feature encoding processing to obtain corresponding visual feature vectors, behavioral feature vectors, and physiological feature vectors; Obtain the current driving scenario state parameters, concatenate the visual feature vector, the behavioral feature vector, the physiological feature vector with the scenario state parameters, input them into the fatigue model, perform inference, and output the fatigue level; The fatigue level is compared with a preset intervention threshold, and when the comparison result meets the triggering condition, the vehicle is triggered to perform an intervention action on the driver.

[0005] In some embodiments, after triggering the vehicle to intervene in the driver's actions, the intervention method further includes: Collect explicit and implicit feedback signals from users in response to the intervention actions; Based on the explicit feedback signal and the implicit feedback signal, the model parameters of the fatigue model are dynamically optimized; A new fatigue model is obtained when the current model parameters meet the preset convergence conditions.

[0006] In some embodiments, the preset convergence condition includes at least one of the following: First: Within a second preset number of consecutive effective driving cycles, the maximum relative change of each of the model parameters is lower than the preset convergence accuracy; The second item: The cumulative driving time within the effective driving cycle reaches the preset time and covers typical driving scenarios; The third item is that the overall acceptance rate of the intervention action by the user is not lower than a preset benchmark, the false alarm rate of triggering the intervention action when the fatigue level is lower than the first preset fatigue level is not higher than the preset false alarm rate, and the missed alarm rate of not triggering the intervention action when the fatigue level is higher than the second preset fatigue level is not higher than the preset missed alarm rate.

[0007] In some embodiments, obtaining the current driving scenario state parameters, concatenating the visual feature vector, the behavioral feature vector, the physiological feature vector, and the scenario state parameters, and inputting them into the fatigue model for inference, and outputting the fatigue level includes: The visual feature vector, the behavioral feature vector, the physiological feature vector, and the scene state parameters are concatenated to obtain a joint input vector; The joint input vector is fed into the fatigue model; The fatigue model performs a nonlinear transformation on the joint input, generates attention weights based on the nonlinearly transformed features, and performs a weighted fusion of the visual feature vector, the behavioral feature vector, and the physiological feature vector based on the attention weights. The weighted fusion result is then subjected to a monotonically increasing nonlinear mapping to output the fatigue level.

[0008] In some embodiments, the intervention method further includes: After each first preset number of effective driving cycles, the preset intervention threshold is optimized based on the false alarm rate and false negative rate of the first preset number of effective driving cycles.

[0009] In some embodiments, the preset intervention thresholds include: a first preset threshold, a second preset threshold, and a third preset threshold, ordered from low to high; The step of comparing the fatigue level with a preset intervention threshold and triggering the vehicle to intervene in the driver's actions when the comparison result meets the triggering conditions includes: When the fatigue level is not less than the first preset threshold, a level one light intervention action is triggered. When the fatigue level is not less than the second preset threshold, a secondary active intervention action is triggered; When the fatigue level is not less than the third preset threshold, a level 3 safety intervention action is triggered.

[0010] In some embodiments, it also includes: If, in a third consecutive set number of valid driving cycles, the fatigue level is detected to meet the triggering condition without triggering the intervention action, the current model parameters are restored to the model parameters that most recently met the convergence condition.

[0011] Secondly, embodiments of this application provide a driver fatigue intervention system, comprising: The acquisition module acquires multi-dimensional modal raw data related to the driver and vehicle status in real time from the vehicle sensor system according to a preset time sliding analysis window. The multi-dimensional modal raw data includes visual modal data, behavioral modal data and physiological modal data. The encoding module performs feature encoding processing on the visual modal data, the behavioral modal data, and the physiological modal data respectively to obtain corresponding visual feature vectors, behavioral feature vectors, and physiological feature vectors; The splicing module obtains the current driving scenario state parameters, splices the visual feature vector, the behavioral feature vector, the physiological feature vector and the scenario state parameters together and inputs them into the fatigue model for inference and outputs the fatigue level. The comparison module compares the fatigue level with a preset intervention threshold, and triggers the vehicle to perform an intervention action on the driver when the comparison result meets the triggering conditions.

[0012] Thirdly, embodiments of this application provide a vehicle, the vehicle comprising: a processor and a memory, the memory storing a computer program, the processor being used to execute the computer program to implement at least one of the above-described driver fatigue intervention methods.

[0013] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed on a processor, implements at least one driver fatigue intervention method as described above.

[0014] The embodiments of this application have the following beneficial effects: The technical solution of this application acquires multi-dimensional modal raw data related to the driver and vehicle status in real time from the vehicle sensor system through a sliding analysis window of preset duration. This includes visual modal data, behavioral modal data, and physiological modal data, effectively solving the time mismatch problem caused by asynchronous sampling frequencies of multi-source signals, transmission delays, and frame drops. This achieves continuous and stable coverage of the dynamic state in real driving scenarios. Based on this, feature encoding processing is performed on the three types of modal data to obtain visual feature vectors, behavioral feature vectors, and physiological feature vectors with high discriminative power and semantic expression capabilities, significantly improving the abstract representation ability and anti-interference ability of various raw signals. Furthermore, current driving scenario state parameters are introduced, and... By concatenating three types of feature vectors with scene state parameters and inputting them into the fatigue model for inference, fatigue assessment is no longer isolated and dependent on a single modality or static rules. Instead, it deeply integrates individual user characteristics, real-time vehicle operating conditions, and external environmental context, possessing true scene adaptability and cognitive understanding capabilities. Finally, the fatigue level is compared with a preset intervention threshold, and the vehicle is directly triggered to perform intervention actions when the triggering conditions are met. This closed-loop process from perception, cognition, decision-making to execution achieves proactive, hierarchical, and vehicle-level collaborative intelligent intervention, effectively improving the accuracy of fatigue driving risk identification, the timeliness of response, and the personalization level of service. This strongly supports the strategic upgrade of intelligent vehicles from driving machines to mobile partners. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A first flowchart of the driver fatigue intervention method according to an embodiment of this application is shown; Figure 2 This paper illustrates a second flowchart of a driver fatigue intervention method according to an embodiment of this application. Figure 3 A schematic diagram of the third process of the driver fatigue intervention method according to an embodiment of this application is shown; Figure 4 A schematic diagram of a driver fatigue intervention system according to an embodiment of this application is shown. Detailed Implementation

[0017] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0018] The components of the embodiments of this application described and illustrated in the accompanying drawings can be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of this application provided in the drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0019] In the following text, the terms "comprising," "having," and their cognates, which may be used in various embodiments of this application, are intended only to indicate a particular feature, number, step, operation, element, component, or combination thereof, and should not be construed as primarily excluding the presence of one or more other features, numbers, steps, operations, elements, components, or combinations thereof, or adding the possibility of one or more combinations thereof. Furthermore, the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0020] Unless otherwise specified, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of this application pertain. Terms (such as those defined in commonly used dictionaries) shall be interpreted as having the same meaning as in their contextual meaning in the relevant technical field and shall not be construed as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of this application.

[0021] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0022] Traditional solutions rely on single sensors or simple rule thresholds, leading to high false alarm rates and poor robustness in complex real-world scenarios such as wearing sunglasses, bumpy roads, and individual physiological differences. This application provides a method, system, vehicle, and readable storage medium for intervening in driver fatigue. By constructing an end-to-end closed loop of sliding window heterogeneous coding context-aware fusion dynamic threshold intervention, this application achieves a paradigm shift in fatigue recognition and intervention from static rule-driven to dynamic cognition-driven approaches. On one hand, a sliding analysis window under a unified time reference is used to perform semantic alignment and statistical aggregation on multi-source asynchronous data, effectively overcoming the temporal mismatch problems caused by transmission delays and frame drops due to differences in sampling rates between modalities, significantly improving the spatiotemporal consistency and robustness of the input data. On the other hand, by applying dedicated encoders to extract high-order semantic features from the raw data of each modality, and introducing a scene context-driven attention mechanism during the fusion stage, the model can adaptively adjust the contribution weights of visual, behavioral, and physiological features. For example, it automatically suppresses the weight of visual features at night and weakens the influence of behavioral features on bumpy roads. This enables high-precision, interpretable, and continuous quantitative assessment of fatigue status in real driving environments with significant individual differences and complex and changing conditions. Finally, by combining personalized grading thresholds optimized through online learning with deep collaboration with vehicle actuators, intervention actions such as voice reminders, seat vibration, and air conditioning adjustments can accurately match user sensitivity and risk levels while avoiding false alarms or missed alerts. This truly achieves a proactive, partner-level human-vehicle collaborative safety protection effect that is tailored to the individual, the environment, and the situation.

[0023] The following examples illustrate the method for intervening in driver fatigue.

[0024] Figure 1 A flowchart illustrating a driver fatigue intervention method according to an embodiment of this application is shown. Exemplarily, the driver fatigue intervention method of this application can be applied to an intelligent cockpit system equipped with multi-source vehicle sensors, specifically deployed on a vehicle-side domain controller or high-performance cockpit chip platform. Through deep collaboration with hardware such as cameras, steering wheel-integrated bioelectrodes, CAN bus gateways, and inertial measurement units, it achieves real-time perception, cognitive modeling, and proactive intervention of the driver's state. Exemplarily, this matching method includes S101-S104: S101, according to the preset duration of the sliding analysis window, acquires multi-dimensional modal raw data related to the driver and vehicle status from the vehicle sensor system in real time. The multi-dimensional modal raw data includes visual modal data, behavioral modal data and physiological modal data.

[0025] As an example, the system uses the high-precision hardware clock of the vehicle's CAN bus as a global unified time reference and defines a sliding analysis window with a fixed duration of 60 seconds. All subsequent calculations are strictly anchored within the time range of this window. This ensures the sufficiency of feature statistics while also taking into account the real-time response requirements, avoiding intervention delays caused by excessively long windows, and preventing noise amplification caused by excessively short windows.

[0026] For visual modal data, the system accesses the raw video stream captured by an infrared camera, performs face detection and eye key point tracking within each sliding window, and then calculates the proportion of eyelid closure time, blink frequency per unit time, and head pitch and yaw angles to form structured primary visual features. For behavioral modal data, the system continuously reads signals such as steering wheel angle, lateral acceleration, lane line recognition results, and turn signal status via the CAN bus, performs time-domain statistics and frequency-domain analysis within the window, and extracts indicators characterizing operational stability such as the standard deviation of steering wheel angle, the frequency of lane departure correction actions, and the entropy value of accelerator pedal force. For physiological modal data, the system acquires raw heart rate variability signals through sensors built into the steering wheel, removes power frequency interference and motion artifacts through bandpass filtering, and calculates time-domain and frequency-domain indicators to constitute primary physiological features reflecting the balance state of the autonomic nervous system. All modalities generate corresponding timestamps at the end of the window and are uniformly managed in a buffer.

[0027] Furthermore, when a certain type of data is missing due to transmission delay or momentary sensor failure, the system automatically calls the historical feature values ​​of the previous valid window for interpolation, or actively blocks the modality and enables a downgrade fusion strategy during the fusion stage. At the same time, the abnormal event is recorded in the model health log to provide a basis for subsequent online diagnosis.

[0028] S102, perform feature encoding processing on the visual modal data, behavioral modal data and physiological modal data respectively to obtain the corresponding visual feature vector, behavioral feature vector and physiological feature vector.

[0029] Visual modal data is mapped into visual feature vectors via a three-layer fully connected neural network encoder. Behavioral modal data, as strong temporal signals, are fed into sequence models such as gate control cyclic unit encoders. The encoder takes the temporal sequence within a sliding window as input and outputs a single temporal behavioral feature vector. Physiological modality data can be input into a fully connected encoder, which outputs physiological feature vectors. The three types of feature vectors together form the input basis of the fatigue model, which significantly improves cross-scenario robustness and individual generalization ability compared to directly using the original signal or simple statistics.

[0030] S103: Obtain the current driving scenario state parameters, concatenate the visual feature vector, behavioral feature vector, and physiological feature vector with the scenario state parameters, input them into the fatigue model, perform inference, and output the fatigue level.

[0031] This step is the cognitive core of the entire intervention method. Its essence is to construct a dynamic decision engine with contextual understanding capabilities, rather than relying on static rules or empirical judgments based on single-modal thresholds. For example, in one implementation, such as... Figure 2 As shown, S103 includes the following sub-steps: S201 concatenates the visual feature vector, behavioral feature vector, physiological feature vector, and scene state parameters to obtain a joint input vector.

[0032] During operation, the system collects scene status parameters in real time, including time attributes, weather conditions, road type, continuous driving duration, and historical fatigue trends. and the state parameters of each scenario With the encoded visual feature vector behavioral feature vector Physiological feature vectors Concatenate them into a joint input vector, specifically, a visual feature vector. The behavioral feature vector characterizes the decreasing alertness implied by the driver's facial micro-movements. Physiological feature vectors depict the coupling relationship between steering wheel operation rhythm and vehicle trajectory stability. Scene parameters reflect the intensity of the physiological response of the autonomic nervous system to cognitive load. This serves as an external constraint, injecting prior knowledge to jointly ensure both the interpretability and robustness of the model. The joint input vector enables the system to maintain the integrity and discriminative power of the feature space even when facing typical complex conditions such as sunglasses obstruction, rain and fog interference, or bumpy roads.

[0033] S202 feeds the joint input vector into the fatigue model.

[0034] Exemplary, the fatigue model is trained offline on a large-scale labeled dataset collected in laboratory or real-vehicle road tests. The dataset covers multimodal synchronous recordings of hundreds of subjects in states of wakefulness, mild fatigue, moderate fatigue, and severe fatigue, with each sample accompanied by a true fatigue label based on the Stanford Sleepiness Scale. Training can employ supervised learning algorithms such as neural networks, aiming to minimize the error between the model output and the true fatigue labels. Optimization targets include the feature encoder weights, attention network parameters, and the linear projection matrix, which includes... Its function is to map high-dimensional feature vectors of different modalities to fatigue response space of the same dimension; based on the current joint input vector, it dynamically generates normalized modal attention weights. This mechanism ensures that weight allocation always serves the current judgment task; for example, in a nighttime highway scenario, the system automatically reduces... Weight and increase Weights are used to compensate for the uncertainty caused by the decrease in visual signal-to-noise ratio.

[0035] S203 performs a nonlinear transformation on the joint input using a fatigue model. Based on the features after the nonlinear transformation, it generates attention weights. Based on the attention weights, it performs weighted fusion on the visual feature vector, behavioral feature vector, and physiological feature vector, and performs a monotonically increasing nonlinear mapping on the weighted fusion result to output the fatigue level.

[0036] Fatigue From the formula The weighted aggregation is completed, and the aggregation result is further compressed using a function in a monotonically increasing nonlinear manner to ensure that the fatigue degree F strictly falls within the closed interval [0,1]. This not only conforms to the continuous nature of human fatigue perception, but also provides a precise, smooth, and differentiable quantitative basis for subsequent graded intervention.

[0037] S104, compare the fatigue level with the preset intervention threshold, and trigger the vehicle to perform an intervention action on the driver when the comparison result meets the triggering condition.

[0038] The core value of this step lies not only in identifying fatigue, but also in achieving human-machine collaborative intervention in an interpretable, adjustable, and verifiable manner. Preset intervention thresholds can be set according to actual application conditions. For example, the preset intervention thresholds include: a first preset threshold T1, a second preset threshold T2, and a third preset threshold T3, ordered from low to high; for example, the first preset threshold T1 = 0.3, the second preset threshold T2 = 0.6, and the third preset threshold T3 = 0.8.

[0039] When the fatigue level F is not less than the first preset threshold T1, a first-level light intervention action is triggered. For example, the system only activates HMI layer resources, presenting gentle prompts in a semi-transparent floating layer on the instrument panel or central control screen, supplemented by voice broadcasts with a slower speech rate and higher pitch, without interrupting the current navigation or media playback, ensuring information delivery while maintaining operational continuity. When the fatigue level F is not less than the second preset threshold T2, a second-level active intervention action is triggered. The system simultaneously calls cockpit actuators and intelligent driving domain resources, including the seat vibration motor outputting tactile reminders at a frequency of 0.5Hz, the air conditioning module automatically adjusting the air outlet temperature to 24℃ and switching to face-blowing mode, and the navigation system real-time searching radius of 15. The system identifies service area locations within a kilometer radius and generates alternative routes, forming a multimodal, cross-domain proactive service response. When the fatigue level F is not less than the third preset threshold T3, a level-three safety intervention action is triggered. The system immediately initiates a graded takeover request to the ADAS domain controller via the vehicle Ethernet. After confirming the availability of longitudinal and lateral control redundancy, a gradual deceleration strategy is initiated to smoothly reduce the vehicle speed to 80% of the current road speed limit with a deceleration of no more than 0.3g. At the same time, combined with high-precision maps and visual perception, the system guides the vehicle to the emergency lane or the nearest ramp exit. The entire process is accompanied by graded voice guidance and one-click notification of emergency contacts, building a safety fallback mechanism covering the entire chain of early warning, intervention, and handling.

[0040] Furthermore, the system can continuously optimize the specific values ​​of the first preset threshold T1, the second preset threshold T2, and the third preset threshold T3 based on the user's actual feedback on the interventions at each level. For example, for a user who frequently receives a Level 1 light intervention reminder during the morning rush hour but then manually turns it off and no adverse driving events occur afterward, the system, after learning through 20 consecutive effective driving cycles, gently increases the first preset threshold T1 from 0.3 to 0.35, thereby reducing the frequency of unnecessary disturbances. All threshold adjustments are strictly constrained by safety boundaries. The first preset threshold T1 is always maintained in the range of 0.25 to 0.40, the second preset threshold T2 is limited to between 0.50 and 0.75, and the third preset threshold T3 is anchored in the range of 0.70 to 0.90, ensuring both personalized adaptation space and preventing the loss of safety margin due to excessive accommodation of user preferences.

[0041] Furthermore, after each first preset number of effective driving cycles, the preset intervention threshold is optimized based on the false alarm rate and false negative rate of the first preset number of effective driving cycles.

[0042] For example, an effective driving cycle must simultaneously meet four objective conditions: first, the time range, which is the complete journey from when the vehicle is ignited to when it is completely turned off; second, the minimum duration, requiring an actual driving time of no less than 10 minutes; third, data quality, with effective data coverage of each key sensor within the journey being no less than 70%; and fourth, driving behavior diversity, requiring the journey to cover at least two typical driving modes, or a cumulative mileage of no less than 5 kilometers. Multiple consecutive effective driving cycles are multiple driving cycles that meet all the above conditions arranged chronologically, with the time interval between any two adjacent cycles not exceeding 7 days, to ensure the learning process has temporal continuity and behavioral continuity.

[0043] As an example, an evaluation window is defined as 10 consecutive valid driving cycles. At the end of each window, the system automatically calculates the effectiveness of all intervention events within the valid driving cycle. If the false alarm rate exceeds 5%, it is considered an excessive false alarm rate, and the system will raise the first preset threshold T1, the second preset threshold T2, and the third preset threshold T3 to suppress over-response in low-risk scenarios. If the false alarm rate exceeds 2%, it is considered an excessive false alarm rate, and the system will lower the first preset threshold T1, the second preset threshold T2, and the third preset threshold T3 to enhance sensitivity in high-risk states. When correcting indicators that exceed the evaluation window limits, it is necessary to simultaneously verify whether the adjusted preset thresholds are still within the safety boundaries. If they exceed the boundaries, the update will be terminated.

[0044] Furthermore, after triggering the vehicle's intervention action on the driver, the system simultaneously activates a closed-loop learning mechanism to collect user feedback signals on the intervention action and dynamically optimize the fatigue model parameters based on these feedback signals, thereby achieving continuous evolution from general capabilities to individual adaptation. This mechanism abandons the traditional design paradigm of fixed upon delivery of in-vehicle systems, instead constructing a self-calibrating, growth-oriented cognitive engine. Its core lies in transforming each human-machine interaction into a modelable, quantifiable, and feedback-based training sample, enabling the system to better understand the user and improve safety with repeated use in real-world applications. For example, in one implementation, such as... Figure 3 As shown, it includes S301-S303: S301 collects explicit and implicit feedback signals from users regarding intervention actions.

[0045] Explicit feedback signals refer to explicit actions initiated by the user, including directly captureable discrete events such as manually closing reminder pop-ups, rejecting intervention commands via voice, or clicking the "Do not remind" button on the central control screen. Implicit feedback signals are inferred indirectly through behavioral sequence modeling. For example, if a user does not take rest measures after receiving a Level 1 minor intervention reminder, and their fatigue level F continues to rise over the next 5 minutes, triggering a Level 2 minor intervention reminder, it indicates that the Level 1 minor intervention failed to achieve the expected effect, constituting an implicit challenge to the validity of the current model's criteria. Both types of feedback together constitute multi-granularity supervision signals, covering a complete spectrum from immediate emotional reactions to long-term behavioral preferences, providing a data foundation that is both timely and stable for subsequent parameter optimization.

[0046] S302 dynamically optimizes the model parameters of the fatigue model based on explicit and implicit feedback signals.

[0047] Specifically, the system will use the current state Defined as a multidimensional joint representation, including the current multimodal feature vector. Scene parameters Recent user behavior history and current fatigue model parameters = [T1, T2, T3]; Action space A is a set of [T1, T2, T3]; constrained perturbation vector The constraints are To maintain the weight sum at a constant 1, To ensure that the threshold order remains unchanged, all The value is strictly limited to ±0.02.

[0048] After each feedback event is triggered, the system performs a reward based on a preset function. Calculate instant returns, where +2 points are awarded if fatigue level F decreases after intervention and no dangerous events occur; a penalty is imposed if a dangerous event occurs without intervention. 5 points, intervention but still a dangerous event occurs: -3 points; A reward of +1 point is given for situations where the user does not cancel and fatigue improves; +0.5 points are given for situations where the user actively requests a rest; and a penalty is imposed for behaviors that explicitly refuse intervention. 1 point; A score of +1 is awarded for the match between the intervention level and the fatigue F-score; penalties are imposed for over-intervention or under-intervention. 0.5 points and 1 point; Points are deducted based on the frequency of accidental triggers while the user is conscious. This composite reward design ensures that the model, while improving recognition accuracy, always considers user experience and system credibility. This reward design ensures that model optimization always strikes a balance between safety, comfort, and effectiveness.

[0049] Each effective driving cycle is treated as a learning unit. After collecting complete data, random exploration is performed with probability ε = 0.1, and random exploration is performed with probability 1. ε selects the action with the highest expected return; the reward is calculated immediately after the action is executed, and the model parameters are updated using the policy gradient method; the immediate update uses a microstep size of 0.001 to ensure a fast response for each feedback; when the convergence condition is met, the system determines that the personalized model has converged and locks the current parameters.

[0050] To address the critical negative feedback from users canceling reminders, the system employs a multi-dimensional penalty function. Quantification: ,in For intervention actions, Tolerance bandwidth; ; The value is dynamically adjusted based on the environment. For example, 1.5 for highways or rainy nights, 0.5 for parking lots or idling, and 1.0 for normal environments. For instance, when a user is on a highway... When fatigue level F = 0.7, immediately cancel the secondary active intervention action. = 0.6, = 0.1, then S = 1.0, P = 1, and the total penalty R = 1.35. For situations where low-level interventions are ineffective, the system uses a triple verification mechanism to determine the rate of increase in fatigue level F. Trajectory analysis confirmed a continuous increase in fatigue level F; statistical analysis of the ineffectiveness index of a certain number of recent similar events was conducted. ,when At that time, according to Adjust the corresponding threshold.

[0051] S303: When the current model parameters meet the preset convergence conditions, a new fatigue model is obtained.

[0052] The preset convergence conditions can be set according to the actual application. For example, the preset convergence conditions include the following: First, within a second preset number of effective driving cycles, the maximum relative change of each model parameter is lower than the preset convergence accuracy; the first condition is parameter stability, for example, within 20 consecutive effective driving cycles, the maximum relative change of the model parameter vector is lower than the preset convergence accuracy of 0.001; Second, the cumulative driving time within the effective driving cycle reaches the preset time and covers typical driving scenarios; the second condition is time sufficiency and scenario coverage, with the cumulative driving time reaching 25 hours and covering typical driving scenarios such as highways, cities, and rural areas; Third, the overall acceptance rate of user intervention actions is not lower than the preset benchmark, the false alarm rate of triggering intervention actions when fatigue is lower than the first preset fatigue level is not higher than the preset false alarm rate, and the false alarm rate of not triggering intervention actions when fatigue is higher than the second preset fatigue level is not higher than the preset false alarm rate. The third criterion is performance compliance: the overall acceptance rate of intervention actions by users is no less than 85%, the false alarm rate of triggering intervention actions when the fatigue level F is below 0.25 is no higher than 5%, and the false negative rate of not triggering intervention actions when the fatigue level F is above 0.7 is no higher than 2%. This triple judgment mechanism ensures that the generated new model not only has stable parameters, but also remains reliable in real and complex scenarios.

[0053] To ensure the system remains under control under any operating conditions, in addition to the intervention threshold being strictly constrained by safety boundaries, The range satisfies Furthermore, if fatigue levels meet the trigger conditions for three consecutive preset number of valid driving cycles without triggering any intervention, the current model parameters will be restored to the model parameters that most recently met the convergence conditions. If, after adjustment, there are three consecutive serious false alarms or missed alarms, or if fatigue levels F ≥ 0.8 are detected for three consecutive valid driving cycles without triggering any intervention, the system will immediately and automatically roll back to the model parameters that most recently met the convergence conditions. This fundamental design eliminates safety hazards caused by model failure, ensuring the model truly possesses both intelligence and robustness.

[0054] In summary, this solution not only achieves high robustness in multimodal fatigue recognition, overcoming inherent limitations such as sunglasses occlusion, rough road surface interference, and individual physiological differences, but also constructs an interpretable, verifiable, and reliable in-vehicle intelligent cognitive system through a dynamic attention fusion architecture, multi-granularity feedback modeling, dual-timescale parameter optimization, triple convergence judgment, and dual safety safeguards. This system enables vehicles to transcend the role of passive response tools, truly possessing the ability to understand user intentions, predict potential risks, and adapt to individual differences, continuously evolving in every real-world interaction, ultimately transforming from a driving machine into a mobile partner.

[0055] like Figure 4As shown, based on the method of the above embodiments, this embodiment provides a driver fatigue intervention system. Exemplarily, the driver fatigue intervention system 100 includes: The acquisition module 110 acquires multi-dimensional modal raw data related to the driver and vehicle status in real time from the vehicle sensor system according to the preset time sliding analysis window. The multi-dimensional modal raw data includes visual modal data, behavioral modal data and physiological modal data. The encoding module 120 performs feature encoding processing on the visual modal data, behavioral modal data, and physiological modal data respectively to obtain the corresponding visual feature vector, behavioral feature vector, and physiological feature vector; The splicing module 130 obtains the current driving scenario state parameters, splices the visual feature vector, behavioral feature vector, and physiological feature vector with the scenario state parameters, and inputs them into the fatigue model for inference and outputs the fatigue level. The comparison module 140 compares the fatigue level with a preset intervention threshold, and triggers the vehicle to perform an intervention action on the driver when the comparison result meets the triggering conditions.

[0056] It is understood that the system in this embodiment corresponds to the control method in the above embodiments, and the options in the above embodiments are also applicable to this embodiment, so they will not be described again here.

[0057] This application also provides a vehicle, exemplary in that the vehicle includes a processor and a memory, wherein the memory stores a computer program, and the processor, by running the computer program, causes the device to perform the functions of the various modules in the above-described driver fatigue intervention method or driver fatigue intervention system.

[0058] The processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, including at least one of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Network Processor (NP), Digital Signal Processor (DSP), Application-Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0059] Memory can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), and Electrically Erasable Programmable Read-Only Memory (EEPROM). Memory is used to store computer programs, and the processor can execute these programs upon receiving execution instructions.

[0060] This application also provides a computer-readable storage medium for storing computer programs used in the aforementioned terminal devices. For example, the computer-readable storage medium may include, but is not limited to, various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0061] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that, in alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0062] In addition, the functional modules or units in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0063] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a smartphone, personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0064] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for intervening in driver fatigue, characterized in that, The method includes: According to the preset duration of the sliding analysis window, multidimensional modal raw data related to the driver and vehicle status are acquired in real time from the vehicle sensor system. The multidimensional modal raw data includes visual modal data, behavioral modal data and physiological modal data. The visual modality data, the behavioral modality data, and the physiological modality data are respectively subjected to feature encoding processing to obtain corresponding visual feature vectors, behavioral feature vectors, and physiological feature vectors; Obtain the current driving scenario state parameters, concatenate the visual feature vector, the behavioral feature vector, the physiological feature vector with the scenario state parameters, input them into the fatigue model, perform inference, and output the fatigue level; The fatigue level is compared with a preset intervention threshold, and when the comparison result meets the triggering condition, the vehicle is triggered to perform an intervention action on the driver.

2. The driver fatigue intervention method according to claim 1, characterized in that, After triggering the vehicle to intervene in the driver's actions, the intervention method further includes: Collect explicit and implicit feedback signals from users in response to the intervention actions; Based on the explicit feedback signal and the implicit feedback signal, the model parameters of the fatigue model are dynamically optimized; A new fatigue model is obtained when the current model parameters meet the preset convergence conditions.

3. The driver fatigue intervention method according to claim 2, characterized in that, The preset convergence condition includes at least one of the following: First: Within a second preset number of consecutive effective driving cycles, the maximum relative change of each of the model parameters is lower than the preset convergence accuracy; The second item: The cumulative driving time within the effective driving cycle reaches the preset time and covers typical driving scenarios; The third item is that the overall acceptance rate of the intervention action by the user is not lower than a preset benchmark, the false alarm rate of triggering the intervention action when the fatigue level is lower than the first preset fatigue level is not higher than the preset false alarm rate, and the missed alarm rate of not triggering the intervention action when the fatigue level is higher than the second preset fatigue level is not higher than the preset missed alarm rate.

4. The driver fatigue intervention method according to claim 1, characterized in that, The process of obtaining the current driving scenario state parameters involves concatenating the visual feature vector, the behavioral feature vector, and the physiological feature vector with the scenario state parameters, inputting them into the fatigue model for inference, and outputting the fatigue level, including: The visual feature vector, the behavioral feature vector, the physiological feature vector, and the scene state parameters are concatenated to obtain a joint input vector; The joint input vector is fed into the fatigue model; The fatigue model performs a nonlinear transformation on the joint input, generates attention weights based on the nonlinearly transformed features, and performs a weighted fusion of the visual feature vector, the behavioral feature vector, and the physiological feature vector based on the attention weights. The weighted fusion result is then subjected to a monotonically increasing nonlinear mapping to output the fatigue level.

5. The driver fatigue intervention method according to claim 1, characterized in that, The intervention methods also include: After each first preset number of effective driving cycles, the preset intervention threshold is optimized based on the false alarm rate and false negative rate of the first preset number of effective driving cycles.

6. The driver fatigue intervention method according to claim 1, characterized in that, The preset intervention thresholds include: a first preset threshold, a second preset threshold, and a third preset threshold, sorted from low to high; The step of comparing the fatigue level with a preset intervention threshold and triggering the vehicle to intervene in the driver's actions when the comparison result meets the triggering conditions includes: When the fatigue level is not less than the first preset threshold, a level one light intervention action is triggered. When the fatigue level is not less than the second preset threshold, a secondary active intervention action is triggered; When the fatigue level is not less than the third preset threshold, a level 3 safety intervention action is triggered.

7. The driver fatigue intervention method according to claim 2, characterized in that, Also includes: If, in a third consecutive set number of valid driving cycles, the fatigue level is detected to meet the triggering condition without triggering the intervention action, the current model parameters are restored to the model parameters that most recently met the convergence condition.

8. A driver fatigue intervention system, characterized in that, include: The acquisition module acquires multi-dimensional modal raw data related to the driver and vehicle status in real time from the vehicle sensor system according to a preset time sliding analysis window. The multi-dimensional modal raw data includes visual modal data, behavioral modal data and physiological modal data. The encoding module performs feature encoding processing on the visual modal data, the behavioral modal data, and the physiological modal data respectively to obtain corresponding visual feature vectors, behavioral feature vectors, and physiological feature vectors; The splicing module obtains the current driving scenario state parameters, splices the visual feature vector, the behavioral feature vector, the physiological feature vector and the scenario state parameters together and inputs them into the fatigue model for inference and outputs the fatigue level. The comparison module compares the fatigue level with a preset intervention threshold, and triggers the vehicle to perform an intervention action on the driver when the comparison result meets the triggering conditions.

9. A vehicle, characterized in that, The vehicle includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the driver fatigue intervention method according to any one of claims 1-7.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is run by the processor, it performs the steps of the driver fatigue intervention method according to any one of claims 1-7.

Images