Driver fatigue driving intervention method and device, storage medium and program product

Abstract

The invention provides a driver fatigue driving intervention method and device, a storage medium and a program product. The method comprises the steps that driving environment information, driver preference information and driver state monitoring data are acquired; determining a driving fatigue level matched with the driver state monitoring data; at least inputting the driving environment information, the driver preference information and the driving fatigue level into a large language model to output a corresponding first fatigue intervention strategy; and controlling and calling at least one vehicle-mounted intelligent agent to execute fatigue driving intervention operation according to the first fatigue intervention strategy. Therefore, the driving environment information, the driver preference data and the state monitoring data are integrated, deep semantic understanding and logical reasoning are realized by means of a large language model, and multi-dimensional information is processed in real time, so that the fatigue state of the driver is accurately evaluated, and a personalized intervention strategy is output.

Description

Technical Field

[0001] This application relates to the field of vehicle intelligent information system technology, and in particular to a method, device, storage medium and program product for intervening in driver fatigue driving. Background Technology

[0002] With the continuous development of artificial intelligence technology, vehicle intelligent information systems are playing an increasingly important role in ensuring driving safety. Fatigue driving is one of the main causes of traffic accidents. It not only reduces a driver's reaction time, attention, and judgment, but also delays reaction time, greatly increasing the risk of incorrect driving decisions and seriously threatening road traffic safety.

[0003] Traditional fatigue monitoring methods assess fatigue levels by analyzing drivers' eye movements, facial expressions, head posture, and other behavioral characteristics using cameras or sensors. They then intervene through fixed methods such as alarms or rest suggestions. As a result, drivers receive only the same reminders or repeated prompts or alarms at different fatigue levels (mild, moderate, and severe), leading to poor driver wake-up effects.

[0004] Currently, the industry has not proposed a better solution to the above problems. Summary of the Invention

[0005] This application provides a method, device, storage medium, and program product for intervening in driver fatigue, which at least solves the problem of poor fatigue intervention effect caused by fixed alarms or single prompt modes in the current related technologies.

[0006] In a first aspect, embodiments of this application provide a method for intervening in driver fatigue driving, comprising: acquiring driving environment information, driver preference information, and driver state monitoring data; determining a driver fatigue level that matches the driver state monitoring data; inputting at least the driving environment information, the driver preference information, and the driver fatigue level into a large language model to output a corresponding first fatigue intervention strategy; and controlling and invoking at least one in-vehicle intelligent agent to perform fatigue driving intervention operations according to the first fatigue intervention strategy, wherein the in-vehicle intelligent agent includes any one of the following: a music wake-up intelligent agent, an emotional companion intelligent agent, an environmental adjustment intelligent agent, and a vehicle takeover intelligent agent.

[0007] Secondly, embodiments of this application provide an electronic device comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the driver fatigue driving intervention method of any embodiment of this application.

[0008] Thirdly, embodiments of this application provide a storage medium storing a computer program thereon, characterized in that, when the program is executed by a processor, it implements the steps of the driver fatigue driving intervention method of any embodiment of this application.

[0009] Fourthly, embodiments of this application provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the driver fatigue driving intervention method of any embodiment of this application.

[0010] The beneficial effects of the embodiments of this application are as follows: By integrating driving environment information, driver preference data, and state monitoring data, and leveraging a large language model for deep semantic understanding and logical reasoning, this system processes multi-dimensional information in real time to accurately assess driver fatigue and output personalized intervention strategies. It flexibly deploys various intelligent agent-based collaborative intervention schemes, including music wake-up, emotional support, environmental adjustment, and vehicle takeover, achieving precise grading and dynamic response to different fatigue levels. This ensures drivers receive appropriate prompts and assistance quickly in various driving situations. Consequently, it not only enhances the monitoring and judgment of driving status but also optimizes the human-computer interaction experience, thereby comprehensively improving driving safety while achieving intelligent and personalized fatigue driving intervention. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart illustrating an example of a driver fatigue driving intervention method according to an embodiment of this application is shown; Figure 2 A flowchart illustrating an example of a fatigue driving intervention operation controlled according to a first fatigue intervention strategy is shown. Figure 3 A flowchart is shown as another example of updating a fatigue intervention strategy according to an embodiment of this application; Figure 4 A schematic diagram of the architecture of an example fatigue driving intervention system based on a large model according to an embodiment of this application is shown; Figure 5 A schematic diagram illustrating the effect of an example of large model processing according to an embodiment of this application is shown. Figure 6A system interaction swimlane diagram of an example of a large-model-based fatigue driving intervention system according to an embodiment of this application is shown; Figure 7 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] In current related technologies, some experts and scholars have proposed methods for driver fatigue monitoring and intervention. These methods may involve integrating data such as visual monitoring, heart rate, and lane departure to assess fatigue status, and all have some form of prompting (sound, light, vibration). Some advanced autonomous driving systems can enter a "takeover" or "safe stop" process when they detect that the driver is not paying attention. However, these methods are mostly single or limited interventions such as "detection → alarm" or "detection → autonomous driving".

[0015] In the current development path of related technologies, when encountering problems such as inaccurate fatigue recognition or poor intervention effects, the approach often involves further optimization of individual modules. For example, this might involve strengthening visual recognition algorithms (improving the accuracy of facial key point detection); adding sensors (heart rate belts, breathing belts) to acquire more fatigue indicators; and expanding alarm notification methods (adding voice broadcasts to vibration, sound, and light alerts). While improving monitoring accuracy or enriching alarm methods can alleviate the problem to some extent, they often fail to achieve "diverse and personalized interventions" from a systemic or intelligent collaborative perspective.

[0016] In addition, some experts and scholars have proposed using machine learning, or even simple neural networks, to fuse multimodal data (visual, physiological, and driving behavior) to improve recognition accuracy. However, their intervention components usually still lean towards rule-based or "if-then" logic: fatigue value above a threshold → alarm / reminder; exceeding a certain higher threshold → forced stop or autonomous driving takeover (if this function is available). Although this approach can be called "multimodal detection," it is still insufficient in terms of the diversity, personalization, and multi-round dynamic scheduling of intervention strategies, and it does not involve centralized scheduling of multiple agents at all.

[0017] Some industry experts have also suggested a shallow integration of fatigue intervention with the in-vehicle ecosystem or third-party applications. Specifically, the task of "awakening the driver" could be delegated to the entertainment system: automatically playing music, videos, or radio when fatigue occurs. However, this integration is only a relatively passive, one-time activation, resulting in poor fatigue driving intervention effects due to fixed alarms or single prompt modes.

[0018] It should be understood that the above description of the relevant technologies is intended only to help the public better understand the inventive spirit and motivation of this application, and is not intended to limit this application. Furthermore, the technical solutions described in the above-mentioned relevant technologies are not prior art; they may also be undisclosed technical solutions, such as those under research or in the laboratory stage.

[0019] In view of this, Figure 1 The flowchart illustrates an example of a driver fatigue driving intervention method according to an embodiment of this application, which utilizes a large model for deep scheduling and dialogue of multiple agents, personalized functions such as music recommendation, etc.

[0020] like Figure 1 As shown, in step S110, driving environment information, driver preference information, and driver status monitoring data are acquired.

[0021] In some implementations, driving environment information can be collected based on multiple sensors and in-vehicle systems. For example, GPS modules or road cloud platforms can be used to collect vehicle location information and road conditions; meteorological sensors can monitor temperature, humidity, and weather conditions inside and outside the vehicle; and illuminance sensors can detect light intensity inside and outside the vehicle. Additionally, driver preference information can be obtained from owner profiles or interaction records. For instance, owner profiles can contain driver-set preference parameters, and interaction records can include the driver's past music choices, air conditioning temperature settings, and communication habits with the in-vehicle system. Furthermore, driver status monitoring data can include driver facial information monitored by in-vehicle cameras to capture eye movements, facial expressions, fatigue characteristics such as yawning, and can also be fused with relevant vehicle sensor information (e.g., steering wheel information, vehicle deviation information), as well as driver physiological parameters collected by physiological sensors, such as wearable devices monitoring heart rate, blood oxygen, and skin conductance. Through multi-source data fusion, the system ensures a comprehensive understanding of the current driving environment, the driver's physiological and behavioral state, and fully considers the driver's personalized preferences, providing a data foundation for subsequent personalized interventions.

[0022] In step S120, a driver fatigue level is determined that matches the driver condition monitoring data.

[0023] Here, driver fatigue levels are categorized into mild, moderate, and severe fatigue. For example, occasional inattention or slightly increased blinking frequency is defined as mild fatigue; frequent blinking, head shaking, and increased yawning are defined as moderate fatigue; and dangerous signs such as the driver looking down and repeated lane departure warnings are defined as severe fatigue. Furthermore, the fatigue level determination can be dynamically adjusted based on driving environment factors (highway or urban streets) to match the fatigue requirements of various driving scenarios. For instance, drivers are more likely to be identified as fatigued on highways, thus ensuring driving safety.

[0024] In some implementations, multimodal feature fusion algorithms can be used to fuse various types of data, and pre-trained machine learning models or rule-based algorithms can be used to comprehensively assess the driver's fatigue state. For example, key indicators such as blinking frequency, eye closure duration, head movements, and physiological fluctuations can be focused on, and these indicators can be matched with standard fatigue characteristics to accurately classify the driver's fatigue state (e.g., mild, moderate, severe). This not only enables rapid assessment of the driver's condition but also provides a clear and quantitative basis for subsequent intervention measures.

[0025] In step S130, at least driving environment information, driver preference information, and driver fatigue level are input into the large language model to output the corresponding first fatigue intervention strategy.

[0026] Here, the large language model can be diverse, such as the GPT and Qwen series. Combined with its powerful reasoning and generation capabilities, it performs scene semantic analysis on driving environment information, driver preference information, and driver fatigue level to generate the most suitable fatigue intervention strategy for the current situation. The introduction of the large language model enhances the intelligence of the intervention strategy, enabling dynamic reasoning and personalized intervention based on multi-dimensional information, effectively mitigating the poor effectiveness of pre-set single-prompt methods. Furthermore, through comprehensive analysis of the driving environment, preferences, and fatigue level, the system can more accurately match the most suitable intervention strategy for the current situation, improving intervention effectiveness and enhancing driver acceptance.

[0027] In step S140, at least one in-vehicle intelligent agent is invoked to perform fatigue driving intervention operation according to the first fatigue intervention strategy. The in-vehicle intelligent agent includes any one of the following: music wake-up intelligent agent, emotional companion intelligent agent, environmental adjustment intelligent agent, and vehicle takeover intelligent agent.

[0028] Here, the output of the large model is converted into control commands for invoking in-vehicle intelligent agents, enabling scheduling and control of different in-vehicle intelligent agents based on the large model. For example, the music wake-up intelligent agent can control the in-vehicle media to play specific music and adjust the volume; the emotional companion intelligent agent can activate the in-vehicle voice interaction system to chat with the driver in a specific chat style; the environmental adjustment intelligent agent can adjust the in-vehicle temperature, airflow, lighting brightness, etc., according to the current environment; and the vehicle takeover intelligent agent can trigger the vehicle takeover function when the driver is severely fatigued or has lost the ability to drive, realizing operations such as automatic parking, deceleration, or changing lanes to a safe area.

[0029] In one example of this application's embodiments, based on the control commands transformed from the first fatigue intervention strategy, a specific vehicle-mounted intelligent agent can be scheduled and controlled to perform corresponding fatigue intervention operations. In another example, based on the control commands transformed from the first fatigue intervention strategy, multiple vehicle-mounted intelligent agents can be scheduled and controlled to jointly perform fatigue intervention operations, such as the combined action of an environmental adjustment intelligent agent and a music wake-up intelligent agent. In this way, the large model acts as the central management hub, coordinating and scheduling multiple intelligent agents through the control commands transformed from the output fatigue intervention strategy, achieving diverse and personalized intervention methods, which are more flexible and targeted.

[0030] Through the embodiments of this application, by integrating multi-dimensional data fusion, intelligent reasoning based on large language models, personalized intervention strategies, and diversified in-vehicle intelligent agents, not only is the personalization and adaptability of intervention measures significantly improved, but the driver's acceptance and cooperation with the intervention are also greatly enhanced, thereby effectively reducing the risk of fatigued driving and improving road traffic safety.

[0031] Regarding the implementation details of step S120, in some examples, a time-series analysis is performed on the multimodal monitoring data contained in the driver state monitoring data based on a time-series analysis model to determine a matching level of driver fatigue. The multimodal monitoring data includes at least one of the following: driver facial information, driver physiological parameters, steering wheel information, and lane departure information.

[0032] This system integrates visual (facial expressions, blink frequency, head posture), physiological (heart rate, skin conductance), and vehicle operation (steering wheel, lane departure) data, and uses a deep learning / fusion algorithm to output a "fatigue level." Based on multimodal fusion and a grading algorithm, it can synthesize multiple features, complement each other, and effectively reduce errors from single sensors. The grading algorithm allows the system to select corresponding intervention strategies for different levels of fatigue.

[0033] Here, the types of time-series analysis models can be diverse, such as LSTM models and Transformer models. By analyzing the temporal continuity and behavioral trends of driver state changes, they can output a more accurate driver fatigue level. Therefore, introducing time-series analysis models effectively utilizes the trend information of driver state changes over time, avoids misjudgments caused by single data anomalies, and improves the accuracy of fatigue level judgment.

[0034] Preferably, the time series analysis model can also adopt a scenario-adaptive weighting mechanism to dynamically adjust feature weights according to the driving environment. For example, when driving at high speed, steering wheel deviation and lane departure information have higher weights; when driving at low speed, facial features and physiological parameters have higher weights. This is to adjust feature weights according to the characteristics of fatigue driving under different driving environments, thereby improving the accuracy of identifying driver fatigue levels.

[0035] Regarding the implementation details of step S130, in some examples, driver emotional information is obtained, and driving environment information, driver emotional information, driver preference information, and driver fatigue level are input into a large language model to output the corresponding first fatigue intervention strategy.

[0036] Here, driver emotional information can be positive (e.g., pleasant, relaxed), neutral (e.g., focused, calm), or negative (e.g., anxious, irritable, angry, depressed), and can be obtained through facial expression analysis or voice emotion analysis, without limitation. By selecting driver emotional information as input parameters for the large model, the generated fatigue intervention strategy can be dynamically adjusted according to the driver's real-time emotional state. For example, when the driver is in a state of "moderate fatigue + focused / relaxed emotion," the corresponding intervention strategy could be: moderately increasing the intensity of the intervention, such as playing music with a strong rhythm or using in-car lighting to enhance the stimulation; on the other hand, when the driver is in a state of "moderate fatigue + anxious / irritable emotion," the corresponding intervention strategy could be: avoiding further stimulation of the driver's emotions, prioritizing soothing interventions (e.g., emotional support) or environmental adjustments (e.g., lowering the in-car temperature or increasing air circulation) to alleviate the driver's tension.

[0037] Through the embodiments of this application, by combining driver emotional information, the system can more accurately understand the driver's true state and avoid mistakenly triggering inappropriate interventions due to simply judging fatigue levels. In addition, by analyzing driver preferences and emotional information, the system can select intervention methods that are more suitable for the driver's personality, avoiding the discomfort and intervention resistance caused by the "single alarm" in traditional fatigue monitoring.

[0038] Figure 2A flowchart illustrating an example of executing fatigue driving intervention operations according to a first fatigue intervention strategy is shown. Here, the first fatigue intervention strategy defines the invocation priority of multiple in-vehicle agents, thereby supporting the sequential invocation of different in-vehicle agents.

[0039] like Figure 2 As shown, in step S210, the control calls the first vehicle-mounted intelligent agent to perform fatigue driving intervention operations and obtains the corresponding first fatigue level update feedback.

[0040] In step S220, based on the feedback of the first fatigue level update, it is determined whether to switch control to call the second vehicle-mounted intelligent agent to perform fatigue driving intervention operation, wherein the calling priority of the first vehicle-mounted intelligent agent defined in the first fatigue intervention strategy is higher than the calling priority of the second vehicle-mounted intelligent agent.

[0041] Regarding the selection of the first in-vehicle intelligent agent, for example, it prioritizes calling intelligent agents with lighter intervention intensity and higher driver acceptance, such as music wake-up intelligent agents or emotional companion intelligent agents. During the intervention operation performed by the first in-vehicle intelligent agent, the system continuously monitors the driver's physiological and behavioral state and updates the fatigue level.

[0042] Specifically, if the intervention of the first in-vehicle intelligent agent has been detected as reducing the driver's fatigue level (e.g., from "moderate fatigue" to "mild fatigue" or "normal state"), the current intervention strategy will be maintained or the fatigue intervention will be terminated without switching to the second in-vehicle intelligent agent. Furthermore, if the intervention of the first in-vehicle intelligent agent is detected as ineffective, and the driver's fatigue state has not improved or has even worsened, the second in-vehicle intelligent agent will be switched to for further intervention based on the calling priority defined by the first fatigue intervention strategy.

[0043] Regarding the selection of the second in-vehicle intelligent agent, its priority is lower than that of the first in-vehicle intelligent agent. Measures with stronger intervention intensity and more obvious stimulation effects can be selected, such as stimulating the driver's senses through an environmental regulation intelligent agent, or calling the vehicle takeover intelligent agent in the case of severe fatigue.

[0044] In this way, by dynamically monitoring the feedback from fatigue level updates, the system ensures that it intervenes with the most suitable intervention method at each stage, avoiding ineffective or excessive stimulation. If the first intervention measure is ineffective, it quickly switches to a stronger intervention method, achieving a "gradual intervention" that gradually increases the intensity of intervention, reducing unnecessary interference to the driver. By progressively escalating intervention measures, driver acceptance is improved while maximizing driving safety.

[0045] Figure 3 A flowchart illustrating another example of updating a fatigue intervention strategy according to an embodiment of this application is shown.

[0046] In step S310, the second fatigue level update feedback for the first fatigue intervention strategy is obtained.

[0047] In step S320, if the second fatigue level update feedback is detected to not meet the preset fatigue relief conditions, at least the first fatigue intervention strategy, the second fatigue level update feedback, driving environment information and driver preference information are input into the large language model to output the corresponding second fatigue intervention strategy.

[0048] In some examples, after detecting that the various in-vehicle agents in the first fatigue intervention strategy have been invoked and are operating according to the corresponding control commands, it is collected whether the driver's fatigue level has decreased. If no significant decrease in fatigue level is detected (e.g., the driver remains in a state of moderate or severe fatigue), or if the fatigue level actually worsens (e.g., from "moderate fatigue" to "severe fatigue") or abnormal driving behavior (e.g., frequent lane departures, braking delays, abnormal steering wheel operation, etc.), then a more effective intervention strategy is needed to wake up the driver to ensure safe driving.

[0049] Specifically, the system integrates a first fatigue intervention strategy used to record the previously used intervention methods and their intensity, a second fatigue level update feedback used to record the latest changes in driver state, driving environment information, and driver preference information. The integrated multidimensional data is then input into a large language model to trigger the large language model to dynamically and adaptively update the intervention strategy, outputting a more accurate and efficient second fatigue intervention strategy.

[0050] In some examples, if the initial intervention strategy is too strong or causes driver discomfort, the large model can be triggered to update the intervention strategy, downgrading it to a milder intervention. If resistance to the current intervention is detected, the system switches to another intervention that better suits the driver's preferences. Thus, after each intervention strategy adjustment, the system continues to monitor the driver's condition and reassess, forming a closed-loop intervention mechanism of "upgrade → downgrade → switch → reassessment".

[0051] Through a central-level large-scale model, the system performs deep semantic understanding of the driver's current emotions, dialogue context, and vehicle status, and dynamically schedules various functional modules such as music wake-up agent, voice chat agent, environmental adjustment agent, and autonomous driving takeover agent. This enables more flexible and personalized intervention methods: combined interventions can be implemented based on different fatigue levels and driver preferences. For example, music or voice chat can be played first at low intensity, and if ineffective, the intervention can be gradually escalated to autonomous driving takeover, greatly improving intervention efficiency and user acceptance.

[0052] In some examples of embodiments of this application, the large model can also be fine-tuned and pre-trained to make its output fatigue intervention strategy more in line with expectations. Specifically, for at least one preset basic fatigue intervention strategy, optimized information on the driver's driving fatigue level is obtained. Then, based on the optimized information of each basic fatigue intervention strategy and the corresponding driving fatigue level, the large model is fine-tuned and pre-trained.

[0053] In some implementations, the basic intervention strategy can be pre-set in the system based on industry experience, driving behavior analysis models, recommendations from traffic management departments, and other information. Preferably, the fatigue basic intervention strategy can be determined based on user-customized input information, such as users pre-customizing corresponding fatigue basic intervention strategies according to their own personalized needs (e.g., preferences for music genres, acceptance of social interaction).

[0054] Specifically, after implementing each basic fatigue intervention strategy, the system needs to collect driver status feedback data to evaluate the actual effect of the intervention strategy and generate optimized information on driver fatigue levels (such as intervention success signals or intervention failure signals). Using the aforementioned basic fatigue intervention strategies and their corresponding optimized information, a dataset required for model fine-tuning is constructed. This dataset is then used to fine-tune the large language model, improving its ability to generate fatigue intervention strategies.

[0055] It should be noted that during the system's cold start phase (the first / first few times a user uses the function), due to the lack of user data, it can execute basic fatigue intervention strategies and fine-tune the large model based on the feedback data from this phase. After fine-tuning, the large model, acting as a scheduling hub, can update the priority and selection weight of intervention strategies based on user feedback (for example, if music intervention is more effective than chat intervention for mild fatigue, the subsequent strategy update prioritizes music adjustment). Therefore, by incorporating driver feedback on basic fatigue intervention strategies, the fine-tuning process of the large model can better integrate the driver's personalized preferences, making the intervention strategies output by the large model more in line with the driver's habits and acceptance, thus improving driver comfort and cooperation.

[0056] It should be noted that, in the process of implementing this application, the inventors conducted research on existing "driver fatigue monitoring and intervention" technologies and found that most solutions focus on fatigue detection itself. For example, they use in-vehicle cameras to recognize the driver's facial expressions and eye closure frequency; they also use steering wheel signals and lane departure warnings as auxiliary detection methods; a few solutions also use physiological signals such as heart rate and respiration. However, after detecting fatigue, these systems typically only issue an "alarm or reminder," and the methods are relatively simple: visual alerts, dashboard indicator lights, or simple seat vibrations.

[0057] As automotive functions have evolved, more in-car senses and functions can be activated (such as music entertainment systems, air conditioning, fragrance systems, and autonomous driving control). However, current technologies largely rely on simple triggering or static rule-based activation, failing to provide deep customization based on individual driver differences and varying levels of fatigue. They also lack truly dynamic scheduling capabilities based on multiple consecutive cycles or specific scenarios. For example, current technologies lack a systematic framework for iterative, closed-loop intervention mechanisms, such as automatically switching to "chatting" when music activation is ineffective, and then prompting for "autonomous driving takeover" if that fails.

[0058] The complete architecture described in this application, consisting of "large model + multimodal perception + unified scheduling of multiple agents (intelligent agents) + continuous learning and adaptation", enables fatigue intervention to have a deeper understanding of the driver's "emotions, context, and personal preferences", thereby improving the driver's awakening effect and the driver's acceptance of fatigue intervention operations.

[0059] Figure 4 A schematic diagram of the architecture of an example fatigue driving intervention system based on a large model according to an embodiment of this application is shown.

[0060] like Figure 4 As shown, the fatigue driving intervention system based on a large model includes a sensor multimodal data acquisition module, an edge-side multimodal fusion and recognition module, a DFM central large model, and a multi-agent module.

[0061] Regarding the details of the sensor multimodal data acquisition module, it can acquire facial expressions, blinking, and yawning through vehicle-mounted cameras; heart rate and skin conductance through physiological sensors; and data such as steering wheel steering and lane departure through vehicle sensors.

[0062] Regarding the details of the edge-side multimodal fusion recognition module, it can use a fusion model (such as CNN-RNN or Transformer) to perform time-series analysis on multimodal data and output fatigue scores, such as 0 = no fatigue, 1 = slight fatigue, 2 = moderate fatigue, 3 = significant fatigue, 4 = severe fatigue, and 5 = extreme fatigue; at the same time, it outputs the current driver's Russell two-dimensional emotional state, such as pleasure level = 1 and arousal level = 1.

[0063] Regarding the details of the DFM central model, in some implementations, it can utilize the Speechocean Dialogue Foundation Model (DUI2.0) platform, for example, using Speechocean's Dongfeng Dialogue Foundation Model as the central model, and customizing a "scene understanding module" and an "intervention scheduling module." Scene understanding integrates fatigue level, emotional state, and driver preferences; intervention scheduling determines which agents (music wake-up, conversational interaction, environmental adjustment, vehicle takeover) to invoke based on the current level and scene, and controls their order or coordinated combination. The large model can act as a "scene understander + instruction distributor," selecting the appropriate intervention method based on the context.

[0064] Figure 5 A schematic diagram illustrating the effect of an example of large model processing according to an embodiment of this application is shown.

[0065] like Figure 5 As shown in the left-hand section, it provides an example of input prompts. In this section, the system provides necessary contextual information to the large model based on information such as the driving environment, driver state, emotional state, and driving preferences. Figure 5 The right side of the diagram provides an example of scenario analysis and pre-instruction generation. Based on input data, the scenario understanding module of the large model analyzes the driving scenario and determines the current driver's state and the required intervention measures. The intervention plan output by the large model describes the system's closed-loop feedback mechanism. Specifically, after each intervention strategy is executed, the system monitors the driver's latest state (such as fatigue score, emotional state, driving behavior, etc.). If the intervention is effective, the system can gradually reduce the intervention intensity to avoid unnecessary interference to the driver. If the intervention is ineffective or the driver's state deteriorates, the system will automatically escalate the intervention intensity, triggering stronger intervention measures (such as vehicle takeover). Thus, by continuously monitoring the driver's state, the system forms a "monitoring → intervention → feedback → optimization" loop mechanism, ensuring that the intervention strategy always maintains optimal effectiveness.

[0066] (4) Multi-Agent Module: Includes a music wake-up agent, an emotional companion agent, an environmental adjustment agent, and a vehicle takeover agent. The key designs of the four agents are as follows: 1) Music Wake-up Agent: ① Personalized Music Library: Collect or analyze music that drivers often listen to that brings pleasure and excitement, or their listening history; ② Different Types of Wake-up Songs: Play songs with moderate tempos and comfortable driving conditions when the driver is mildly fatigued; play music with strong tempos and high energy levels when the driver is severely fatigued; ③ Side Effects and Monitoring: The music volume should not be too loud to avoid affecting driving safety; if the driver looks down at the playback device or expresses aversion, the program should be switched quickly.

[0067] 2) Emotional Chat Agent: ① Dialogue Topic Library: It can automatically select appropriate conversation topics based on the driver's interest tags, such as sports, movies, travel, food, etc.; ② Dynamic Topic Generation: The large model continues to explore new topics based on the driver's real-time reactions and status, maintaining the natural flow of the conversation; ③ Emotion Recognition: If the driver's low mood or boredom is detected, the chat mode can be switched or the conversation pace can be shortened to avoid interfering with the driver's attention.

[0068] 3) Environmental regulation agent: ① Air conditioning temperature and airflow: Increase airflow and lower temperature according to fatigue level and personal preference to keep the driver awake; ② Fragrance / scent: Appropriately release a refreshing or invigorating fragrance in the car to stimulate the senses to a certain extent; ③ Seat / steering wheel vibration: Physical stimulation can often quickly awaken short-term attention.

[0069] 4) Vehicle Takeover Agent: ① Takeover Logic: When the system detects that the driver's fatigue level has reached a dangerous level (such as level 4 or 5), and the vehicle has L2+ / L3 / L4 level autonomous driving functions, the large model initiates a "takeover request" according to a preset strategy; ② Safety Redundancy: An in-vehicle alarm or prompt sound alerts the driver to the fact that the vehicle is about to start autonomous driving; if it is safe and feasible, the vehicle gradually takes over, and the driver enters the assisted monitoring mode; ③ Emergency Stop: If the autonomous driving function is limited or the environment does not support it, the system can actively find a safe area to prompt the driver to stop and rest.

[0070] Table 1 Examples of preset fatigue intervention strategies in this system (users can customize them on the online platform).

[0071] grade fatigue level Possible manifestations Intervention strategy examples 1 Slight fatigue / slightly low mood Slightly decreased attention / minor yawning • Light conversation (entertainment topics) • Playing soft music (to enhance the atmosphere) • Fine-tuning the seat or air conditioning temperature 2 Moderate fatigue / significant mood swings Increased blinking frequency / fatigue tone / heart rate fluctuations • Wake-up music (music with high wake-up rate and familiarity) • Moderate voice interaction: guiding conversation + reminding attention 3 Significant fatigue / significant decrease in attention Nodding off / Vehicle driving becomes unstable • Strong wake-up music + audible alerts • Seat vibration or steering wheel vibration • Emphasis on reminders via dialogue 4 Severe fatigue / difficulty maintaining normal driving Multiple lane departures / physiological warnings • Emergency music + strong vibration in the seat or steering wheel • Voice prompts to rest immediately • Reminder to take over autonomous driving 5 extreme fatigue / disability Multiple warnings / Unable to maintain vehicle stability • Autonomous driving takeover / forced stop recommendation • Emergency call function (if required) Figure 6 A system interaction swimlane diagram of an example of a fatigue driving intervention system based on a large model according to an embodiment of this application is shown.

[0072] like Figure 6 As shown, the interaction process of the fatigue driving intervention system based on a large model mainly includes the following steps: 1) Driver Start-up and Data Acquisition: When the driver enters the vehicle, the system initializes, the vehicle-mounted camera / physiological sensor begins data acquisition, and the ASR monitors the voice.

[0073] 2) Multimodal data-driven driver state monitoring: A lightweight algorithm pre-installed on the vehicle first performs fatigue and sentiment analysis on the data. If the detected fatigue does not exceed the set threshold, the system continues routine monitoring. If intervention is deemed necessary, the "intervention needed" signal is uploaded to the DFM central big model.

[0074] 3) The DFM central big model analyzes the driver's fatigue level, emotional state, current driving scenario, and dialogue context (if there is voice interaction) to provide global information for subsequent intervention decisions.

[0075] 4) The DFM central model generates intervention strategies based on fatigue level, emotional state, personality preferences, and the vehicle's location (e.g., highway, city, nighttime) (examples of predefined intervention strategies in the initial stage are shown in Table 1). In some implementations, some intervention schemes or chat topics can be configured on the Speechocean dialogue customization platform, allowing enterprises or users to customize strategies corresponding to different fatigue levels. When the central model is dispatched to an Agent, it will refer to the scenario scripts or skill rules already defined on the platform.

[0076] Therefore, by deeply fusing multimodal fatigue grading with a large model, multimodal data such as visual, physiological, and vehicle control data are integrated to obtain fatigue grading. Simultaneously, by combining the semantic understanding capabilities of the large model, contextual scene awareness is achieved, and intervention strategies are dynamically generated based on this.

[0077] 5) The central big model refines the decision results into executable instructions, such as "play a certain playlist," "the chat agent tries to talk to the driver about their interests," and "turn on seat vibration." The instructions are then sent to the corresponding agents (music, chat, air conditioning, vehicle control, etc.) to execute the specific functions.

[0078] 6) Agent Command Processing and Response: Music, chat, and other agents receive commands and return execution results or playable resource addresses to the central big model. For example, the "music wake-up agent" receives the command "play familiar music with a strong rhythm," or the "conversation agent" receives the command "start a casual chat, topic: travel." The corresponding agent executes functions, such as the music agent calling the car audio system and selecting a suitable music library, the chat agent conducting a TTS conversation with the driver, the environment agent adjusting the temperature / fragrance, and the vehicle takeover agent controlling the vehicle or reminding the driver to stop.

[0079] Therefore, by using a central big model and a distributed intelligent agent architecture, a large model is used as the "central command" to uniformly manage and call various agents such as music, chat, environmental adjustment, and autonomous driving, forming a flexible "multi-agent fatigue management system"; based on fatigue level and real-time driver feedback, it supports multiple rounds of iteration and upgrade / downgrade intervention methods.

[0080] 7) The central big model returns resource links, TTS audio, etc., to the in-vehicle controls to start music playback or voice output. When multiple agents return execution results or feedback, the big model can integrate them and encapsulate them into TTS (text-to-speech) or in-vehicle screen information to present to the driver, making the entire interaction more consistent and natural.

[0081] 8) The in-vehicle interactive device responds to actions such as seat vibration, air conditioning adjustment, in-vehicle screen prompts, and voice broadcasts. The driver sees or hears the prompts and takes further action (such as responding verbally or adjusting the seat position).

[0082] 9) The onboard sensors continuously monitor the driver's condition to see if fatigue has been relieved; if it has not been relieved, the data will continue to be reported to the central big model for further intervention or autonomous driving takeover.

[0083] This application's embodiments upgrade the fatigue intervention approach from "one-way alarm" to "multi-agent collaboration." It provides several selectable agents, with a large model controlling "when to invoke which agent and how to collaborate." Furthermore, during fatigue monitoring, the large model continuously listens to the driver's voice, identifies emotions, and judges intentions, offering companionship, casual conversation, and emotional reassurance. In implementation, the conversation topic can be dynamically adjusted: if the driver is interested in football, talk about football; if the driver is unwilling to chat or shows signs of agitation, quickly switch methods to avoid excessive interference.

[0084] Fatigue intervention based on a large model supports personalization and continuous learning, enabling learning at the intervention strategy level. For example, as the number of times a driver uses the system increases, the large model can record "what music best wakes them up" and "when should autonomous driving be suggested to take over," adaptively optimizing for the next intervention after a period of time. Furthermore, after intervention, fatigue changes are continuously monitored via onboard sensors, and effective / ineffective results are continuously fed back to the large model, iterating and optimizing the intervention strategy to achieve a personalized experience that becomes more accurate with use, effectively improving driving safety and user satisfaction.

[0085] By using a more refined logic to trigger autonomous driving takeover in a graded manner, under the fatigue grading system, the decision to activate autonomous driving or emergency stop is only made when several interventions (music, chatting, air conditioning stimulation, etc.) are ineffective and the fatigue index remains high. After takeover, the system continues to monitor whether the driver has the ability to resume driving.

[0086] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of combined actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application. In the above embodiments, the descriptions of each embodiment have their own emphasis; for parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0087] In some embodiments, this application provides a non-volatile computer-readable storage medium storing one or more programs including execution instructions, which can be read and executed by electronic devices (including but not limited to computers, servers, or network devices) to perform any of the driver fatigue driving intervention methods described above.

[0088] In some embodiments, this application also provides a computer program product, the computer program product including a computer program stored on a non-volatile computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform any of the above-described driver fatigue driving intervention methods.

[0089] In some embodiments, this application also provides an electronic device comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a driver fatigue driving intervention method.

[0090] Figure 7 This is a schematic diagram of the hardware structure of an electronic device for implementing a driver fatigue intervention method according to another embodiment of this application, as shown below. Figure 7 As shown, the device includes: One or more processors 710 and memory 720, Figure 7 Take the 710 processor as an example.

[0091] The device for implementing driver fatigue intervention methods may also include an input device 730 and an output device 740.

[0092] The processor 710, memory 720, input device 730, and output device 740 can be connected via a bus or other means. Figure 7 Taking the example of a connection between China and Israel via a bus.

[0093] The memory 720, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the driver fatigue driving intervention method in the embodiments of this application. The processor 710 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 720, thereby implementing the driver fatigue driving intervention method in the above-described method embodiments.

[0094] The memory 720 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device. Furthermore, the memory 720 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 720 may optionally include memory remotely located relative to the processor 710, and these remote memories can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0095] Input device 730 can receive input digital or character information and generate signals related to user settings and function control of the electronic device. Output device 740 may include display devices such as a display screen.

[0096] The one or more modules are stored in the memory 720, and when executed by the one or more processors 710, they execute the driver fatigue driving intervention method in any of the above method embodiments.

[0097] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0098] The electronic devices in this application embodiments exist in various forms, including but not limited to: (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include smartphones, multimedia phones, feature phones, and low-end phones.

[0099] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include: PDAs, MIDs, and UMPCs, etc.

[0100] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players, handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0101] (4) Other airborne electronic devices with data interaction capabilities, such as vehicle-mounted systems installed on vehicles.

[0102] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0103] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A driver fatigue driving intervention method, the method comprising: Acquire driving environment information, driver preference information and driver status monitoring data; determining a driving fatigue level that matches the driver status monitoring data; inputting at least the driving environment information, the driver preference information and the driving fatigue level into a large language model to output a corresponding first fatigue intervention strategy; According to the first fatigue intervention strategy control, at least one on-board intelligent agent is called to perform fatigue driving intervention operations, and the on-board intelligent agent includes any one of the following: a music wake-up intelligent agent, an emotional chatting intelligent agent, an environmental adjustment intelligent agent, and a vehicle takeover intelligent agent.

2. The method according to claim 1, wherein: The first fatigue intervention strategy defines a calling priority for multiple on-board intelligent agents; The controlling and calling at least one vehicle-mounted intelligent agent to perform a fatigue driving intervention operation according to the first fatigue intervention strategy includes: Controlling and calling the first vehicle-mounted intelligent agent to perform fatigue driving intervention operations and obtain corresponding first fatigue level update feedback; According to the first fatigue level update feedback, determine whether to switch control to call the second on-board intelligent agent to perform fatigue driving intervention operation, wherein the calling priority of the first on-board intelligent agent defined in the first fatigue intervention strategy is higher than the calling priority of the second on-board intelligent agent.

3. The method according to claim 1 or 2, wherein: After calling at least one vehicle-mounted intelligent agent to perform a fatigue driving intervention operation according to the first fatigue intervention strategy control, the method further includes: Obtaining updated feedback of a second fatigue level for the first fatigue intervention strategy; When it is detected that the second fatigue level update feedback does not meet the preset fatigue relief conditions, at least the first fatigue intervention strategy, the second fatigue level update feedback, the driving environment information and the driver preference information are input into the large language model to output the corresponding second fatigue intervention strategy.

4. The method according to claim 1, wherein: The step of inputting at least the driving environment information, the driver preference information and the driving fatigue level into a large language model to output a corresponding first fatigue intervention strategy includes: Obtain driver emotion information; The driving environment information, the driver emotion information, the driver preference information and the driving fatigue level are input into a large language model to output a corresponding first fatigue intervention strategy.

5. The method according to claim 1, wherein: Fine-tuning pre-training for the large language model includes: For at least one preset fatigue-based intervention strategy, obtaining optimized information of the driver's driving fatigue level; The large model is fine-tuned and pre-trained according to the optimization information of each fatigue-based intervention strategy and the corresponding driving fatigue level.

6. The method according to claim 5, wherein: The fatigue-based intervention strategy is determined based on user customized input information.

7. The method according to claim 1, wherein: The driver status monitoring data includes multimodal monitoring data, and the multimodal monitoring data includes at least one of the following: driver facial information, driver physiological parameters, steering wheel steering information, and lane departure information; The determining of the driving fatigue level matching the driver status monitoring data comprises: The multimodal monitoring data is subjected to time series analysis based on a time series analysis model to determine a matching driving fatigue level.

8. A storage medium having a computer program stored thereon, wherein: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

9. An electronic device, comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images