Control method and device, vehicle, medium and program product

Abstract

The invention relates to a control method and device, a vehicle, a medium and a program product in the technical field of intelligent driving, and the method comprises the steps that driver state information is determined according to driver monitoring information of multiple sensing modes, and the driver monitoring information of the multiple sensing modes is obtained through sensing equipment of different modes; and executing control of target equipment according to the driver state information. Thus, the driver state information is determined through the driver monitoring information of different sensing modes, inaccurate driver state information determination caused by the driver monitoring information of a single sensing mode can be avoided, the accuracy of driver state information determination is improved, and then the safety of vehicle driving is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of intelligent driving technology, and in particular to a control method, device, vehicle, medium, and program product. Background Art

[0002] Driver state monitoring can assess the driver's driving status through real-time monitoring and issue warnings when necessary. This is particularly important in scenarios such as long-distance driving and commuting, where it can alert drivers of fatigue, helping to reduce traffic accidents. Driver monitoring information from a single sensing modality can result in inaccurate driver state information, leading to lower vehicle control accuracy. Summary of the Invention

[0003] To overcome the problems existing in the related art, the present disclosure provides a control method, device, vehicle, medium and program product.

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided a control method, including: determining driver status information based on driver monitoring information of multiple sensing modalities, wherein the driver monitoring information of the multiple sensing modalities is obtained through sensing devices of different modalities; Control of a target device is performed based on the driver state information.

[0005] The above technical solution determines the driver status information through driver monitoring information of different perception modalities, which can avoid the inaccurate driver status information determination caused by driver monitoring information of a single perception modality, improve the accuracy of driver status information determination, and thus improve the safety of vehicle driving.

[0006] In some possible implementations, determining driver status information based on driver monitoring information of multiple perception modalities includes: determining a driver state feature corresponding to each of the sensing modalities based on the driver monitoring information of the plurality of sensing modalities; Determining, based on the state information of the sensing device of each modality, the weight of the driver state feature corresponding to each sensing modality; The driver state features corresponding to multiple perception modalities are fused according to the weight corresponding to each perception modality to determine the driver state information.

[0007] This technical solution utilizes monitoring information collected by different modal sensing devices to determine the driver's state characteristics. It then dynamically assigns weights to each modal characteristic based on the sensing device's state, effectively avoiding information bias caused by single-modal failures or environmental interference. By weightedly fusing multimodal features, it comprehensively and accurately reflects the driver's true state, providing a more reliable basis for vehicle control decisions and significantly improving driving safety.

[0008] In some possible implementations, determining the weight of the driver status feature corresponding to each perception modality based on the status information of the perception device of each modality includes: The weight of the driver status feature corresponding to each of the perception modalities is determined based on the status information of the perception device of each modality within the same time period, wherein the status of the perception device represented by the status information is different, and the corresponding weight is different.

[0009] The above-mentioned technical solution uses the status information of the perception devices of each modality within the same time period to eliminate the state feature misalignment caused by the asynchronous data acquisition timing, and ensure the consistency of multi-modal features on the time axis; at the same time, the weights of each modal feature are dynamically adjusted according to the real-time status information of the perception device, so that the perception modality with good status occupies a dominant position in the fusion, effectively suppressing the negative impact of faulty or interfering modalities, and improving the anti-interference ability and adaptability of the driver's status judgment.

[0010] In some possible implementations, the sensing devices corresponding to the multiple sensing modalities include at least two of the following: a wearable device, an in-vehicle sensing device, and a mobile device.

[0011] The above technical solution realizes multi-scenario and multi-dimensional data complementarity through at least two of wearable devices, vehicle-mounted sensing devices and mobile devices, effectively covers the status monitoring blind spots in different driving environments, and significantly enhances the comprehensiveness and real-time nature of the driver's status perception.

[0012] In some possible implementations, the target device includes an in-vehicle execution device, and executing control of the target device according to the driver state information includes: When the driver status information indicates that there is a need for driving intervention, determining a control strategy of the on-vehicle execution device according to the driver status information; The control of the vehicle-mounted execution device is performed according to the control strategy.

[0013] When the above technical solution determines that a driver's condition requires intervention, it dynamically generates a hierarchical control strategy based on the specific type and severity of the condition information, directly driving the onboard actuators to respond quickly. This seamless transition from condition monitoring to safety intervention effectively shortens risk response time and provides more reliable safety for drivers and passengers.

[0014] In some possible implementations, determining a control strategy for the vehicle-mounted execution device based on the driver status information includes: determining target intervention information based on the driver status information and historical driving information; A control strategy of the vehicle-mounted execution device is determined according to the target intervention information.

[0015] This technical solution not only generates immediate intervention information based on the current driver state but also dynamically modifies this information based on historical driving behavior patterns, creating a targeted intervention plan that better suits the driver's characteristics. The resulting control strategy effectively corrects the current dangerous state while avoiding excessive intervention that could negatively impact the driving experience. This ensures safety while enhancing the comfort and adaptability of human-machine collaboration.

[0016] In some possible implementations, determining target intervention information based on the driver state information and historical driving information includes: determining an intervention threshold for the driver based on the driver status information and historical driving information; The target intervention information for the driver is determined according to the intervention threshold, wherein different intervention thresholds correspond to different preset intervention information.

[0017] This technical solution combines the driver's current state with historical driving behavior patterns to intelligently generate tiered intervention thresholds, avoiding the over-intervention or delayed response caused by a single fixed threshold. This dynamic threshold adjustment mechanism, based on individual behavioral characteristics, allows safety interventions to better meet the driver's actual needs.

[0018] In some possible implementations, the target device includes a home device, and controlling the target device according to the driver status information includes: determining target control parameters of the home appliance according to the driver status information; The target control parameter is sent to the home device, where the target control parameter is used to execute control of the home device.

[0019] This technical solution automatically generates adaptive home environment control parameters based on real-time information such as the driver's fatigue level, emotional state, or health indicators, and sends precise control commands to home devices via the connected car. This cross-scenario proactive service mechanism not only creates a personalized home environment for users, but also enhances convenience by proactively adjusting device status. It also avoids the safety risks of manual operation by the driver, expanding the service boundaries and user experience value of smart cars.

[0020] In some possible implementations, the household appliances include at least one of the following: curtains, beds, lights, stereos, air conditioners, doors and windows.

[0021] The above technical solution can achieve refined control such as lighting atmosphere adjustment, air conditioning temperature and humidity preset, automatic opening and closing of doors and windows based on the driver's status, creating a personalized home environment for users and significantly improving living comfort and intelligent experience.

[0022] According to a second aspect of an embodiment of the present disclosure, there is provided a control device, including: a determination module configured to determine driver status information based on driver monitoring information of multiple sensing modalities, wherein the driver monitoring information of the multiple sensing modalities is obtained through sensing devices of different modalities; The control module is configured to control the target device according to the driver status information.

[0023] In some possible implementations, the determining module is configured to: determining a driver state feature corresponding to each of the sensing modalities based on the driver monitoring information of the plurality of sensing modalities; Determining, based on the state information of the sensing device of each modality, the weight of the driver state feature corresponding to each sensing modality; The driver state features corresponding to multiple perception modalities are fused according to the weight corresponding to each perception modality to determine the driver state information.

[0024] In some possible implementations, the determining module is configured to: The weight of the driver status feature corresponding to each of the perception modalities is determined based on the status information of the perception device of each modality within the same time period, wherein the status of the perception device represented by the status information is different, and the corresponding weight is different.

[0025] In some possible implementations, the sensing devices corresponding to the multiple sensing modalities include at least two of the following: a wearable device, an in-vehicle sensing device, and a mobile device.

[0026] In some possible implementations, the target device includes an in-vehicle execution device, and the control module is configured to: When the driver status information indicates that there is a need for driving intervention, determining a control strategy of the on-vehicle execution device according to the driver status information; The control of the vehicle-mounted execution device is performed according to the control strategy.

[0027] In some possible implementations, the target device includes a household device, and the control module is configured to: determining target control parameters of the home appliance according to the driver status information; The target control parameter is sent to the home device, where the target control parameter is used to execute control of the home device.

[0028] According to a third aspect of an embodiment of the present disclosure, there is provided a vehicle, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to execute the executable instructions stored in the memory to implement the method according to any one of the first aspects.

[0029] According to a fourth aspect of an embodiment of the present disclosure, there is provided a computer-readable storage medium having a computer program stored thereon, which implements the steps of any one of the methods described in the first aspect when executed by a processor.

[0030] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, which implements the steps of any one of the methods in the first aspect when executed by a processor.

[0031] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0033] Figure 1 is a flowchart showing a control method according to an exemplary embodiment.

[0034] Figure 2 An implementation according to an exemplary embodiment is shown Figure 1 Flowchart of step S11 in FIG.

[0035] Figure 3An implementation according to an exemplary embodiment is shown Figure 2 Flowchart of step S112 in FIG.

[0036] Figure 4 An implementation according to an exemplary embodiment is shown Figure 1 Flowchart of step S12 in FIG.

[0037] Figure 5 is a block diagram of a control device according to an exemplary embodiment.

[0038] Figure 6 is a block diagram of a vehicle according to an exemplary embodiment. DETAILED DESCRIPTION

[0039] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

[0040] It should be noted that all actions of acquiring signals, information or data in the present disclosure are carried out in compliance with the corresponding data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0041] Before introducing a monitoring method provided by an embodiment of the present disclosure, the technical solutions in related scenarios are first introduced. Driver health and behavior monitoring mainly monitors the driver's status through physiological data such as the driver's heart rate and blood pressure, or recognizes the driver's facial expressions through the Driver Monitoring System (DMS) to analyze the driver's fatigue and health status, and then provide early warning or intervention in driving behavior.

[0042] However, whether through physiological signals or facial expressions, driver monitoring is performed through a single data source. For example, DMS mainly relies on the driver's facial expressions captured by the camera to monitor whether the driver is in a state of fatigue such as closing his eyes or yawning. That is, it is limited to monitoring the driver through the single data source of the camera, resulting in low accuracy of driver status monitoring. In addition, after the abnormal state of the driver is detected, it is usually limited to adjusting the in-car environment. There is a lack of linkage with home appliances, and it fails to form an integrated "people-car-home" full-ecological linkage control. In addition, when monitoring the driver's status through physiological data such as heart rate, there is a lack of advance prediction capabilities. In addition, for driving behavior warning or intervention, a unified approach is adopted, lacking the ability to personalize warning and intervention, and unable to achieve accurate warning and intervention effects.

[0043] In view of this, the control method provided in the embodiment of the present disclosure is intended to avoid inaccurate determination of driver status information caused by single perception modality driver monitoring information, improve the accuracy of driver status information determination, and thereby enhance vehicle driving safety.

[0044] Figure 1 is a flow chart of a control method according to an exemplary embodiment. Figure 1 As shown, the method can be applied to a vehicle, for example, the method provided by the embodiment of the present disclosure is executed by the vehicle's body domain controller (BDC) or intelligent cockpit domain controller (IDC), including the following steps.

[0045] In step S11, driver status information is determined based on driver monitoring information of multiple sensing modalities, wherein the driver monitoring information of the multiple sensing modalities is obtained through sensing devices of different modalities; Driver monitoring information in different perception modalities may include, for example, driver monitoring information in multiple different modalities, such as visual modality, voice modality, and physiological signal modality. Driver monitoring information in visual modality may be first information such as facial expressions, eye movements, and head posture perceived by a camera configured on the vehicle or a camera configured on a mobile terminal; driver monitoring information in voice modality may be second information such as yawning perceived by a voice acquisition device configured on the vehicle or a voice acquisition device configured on a mobile terminal; and physiological signal modality may be third information such as heart rate, blood pressure, body temperature, and sleep duration perceived by a wearable device. The wearable device may transmit the third information to the vehicle via Bluetooth or WiFi, for example, a smart ring, smart bracelet, or smart watch worn by the driver may perceive heart rate, blood pressure, body temperature, steering wheel grip strength, and the like.

[0046] Driver status information may include behavioral and physiological information. Physiological information may include, for example, fatigue and emotional information. Fatigue information may include the duration and frequency of eye closure and yawning frequency. Emotional information may include facial expression information and speech information. Facial expression information may include anger, anxiety, and other expressions. Speech information may include rapid speech or high-pitched voice. Behavioral information may include, for example, gaze information and hand movement information.

[0047] In the embodiment of the present disclosure, driver monitoring information can be collected synchronously by sensors of different perception modalities, and then the driver monitoring information corresponding to different perception modalities at the same time can be kept synchronized on the timeline according to the collection time of the driver monitoring information corresponding to the sensors of each perception modality. The driver monitoring information of multiple perception modalities can be integrated to determine the driver status information, rather than using the driver monitoring information of a single perception modality to determine the driver status information.

[0048] For example, a convolutional neural network extracts eye features from the first information, uses Mel-frequency cepstral coefficients to analyze fatigue characteristics from the second information, and analyzes features such as heart rate variability, skin conductivity, and breathing patterns from the third information to determine the driver's fatigue and health characteristics. Driver status information is then determined using decision-level fusion (such as weighted voting) or feature-level fusion (such as a deep learning model). For example, if visual detection indicates eye closure for longer than two seconds, voice detection indicates a deep voice tone, and the breathing pattern indicates heavy breathing, driver status information can be output based on a rule engine or machine learning classifier, such as a support vector machine (SVM) or a long short-term memory (LSTM) network. For example, driver monitoring information from multiple sensory modalities can be input and, after passing through an LSTM model, output driver status information such as "Fatigue Level: High."

[0049] In step S12, control of the target device is performed according to the driver state information.

[0050] Target devices can include controlled vehicle subsystems or onboard devices, such as speakers, dashboards, seats, and air conditioners. These can trigger actions such as playing reminders on the speakers, displaying alarms on the dashboard, or vibrating seats, or lowering the air conditioner temperature. They can also include non-onboard devices, such as home appliances logged into the same account as the vehicle. If fatigue levels are high, these can trigger actions such as increasing the air conditioner or controlling the lights at home to ensure a comfortable rest with appropriate lighting and temperature upon arrival.

[0051] In the embodiment of the present disclosure, a preset control logic is mapped according to the driver status information, and then the target device is determined. For example, when the driver status information indicates that the driver's fatigue level is high and the distraction level is medium, the control logic can be determined to start AEB (Autonomous Emergency Braking) + double flash alarm + seat vibration, and trigger LKA (Lane Keeping Assist) + voice prompt "Please focus on driving", and then send control instructions to target devices such as AEB, cockpit domain controller and speakers through the vehicle bus for execution. Furthermore, the driver's response (such as whether to take over the steering wheel) is monitored in real time, and the control intensity is dynamically adjusted. Example: If the driver continues to be unresponsive, gradually upgrade to pull over.

[0052] The above technical solution determines driver status information based on driver monitoring information from multiple sensing modalities, obtained through sensing devices of different modalities; and controls a target device based on this driver status information. This determination of driver status information based on driver monitoring information from different sensing modalities avoids inaccurate driver status information determinations based on driver monitoring information from a single sensing modality, improving the accuracy of driver status information determination and, in turn, enhancing vehicle driving safety.

[0053] In some possible implementations, see Figure 2 As shown, in step S11, determining the driver status information based on the driver monitoring information of multiple perception modalities includes: In step S111, based on the driver monitoring information of the plurality of sensing modalities, a driver state feature corresponding to each of the sensing modalities is determined; Among them, feature extraction can be the process of extracting key information that can characterize the driver's status from the original data, such as facial expression features, heart rate variability, steering wheel operation mode, etc.

[0054] In the disclosed embodiments, high-dimensional raw data (such as images and signals) can be converted into low-dimensional feature vectors, reducing computational complexity and improving classification performance. For example, for driver monitoring information in the visual modality, the HOG algorithm is used to extract edge gradient features of facial images and generate feature vectors. For driver monitoring information in the physiological signal modality, the frequency domain features of the heart rate signal are analyzed using a fast Fourier transform, and the ratio of low-frequency to high-frequency power is extracted as a feature. For driver monitoring information in the behavioral modality, the sliding window method can be used to analyze the steering wheel pressure signal and extract statistical features such as the mean and standard deviation.

[0055] In step S112, based on the state information of the sensing device of each modality, the weight of the driver state feature corresponding to each sensing modality is determined; The weights represent the relative importance of each modal feature during fusion, reflecting the impact of device status on feature reliability. Status information can include sensor accuracy (such as camera resolution), signal strength (such as heart rate sensor noise level), and device failure rate.

[0056] In the disclosed embodiments, feature weights can be dynamically adjusted based on device status to improve the robustness of the fusion results. For example, if the camera is blocked (status = fault), the visual feature weight is set to 0.1; if the heart rate sensor noise is less than 10% (status = good), the physiological feature weight is increased to 0.7; and if the microphone detects wind noise greater than 50% (status = poor), the voice modality feature weight is reduced to 0.2.

[0057] In step S113 , the driver state features corresponding to the plurality of perception modalities are fused according to the weight corresponding to each perception modality to determine the driver state information.

[0058] In the disclosed embodiments, multimodal features can be combined according to weights to generate a comprehensive feature vector for driver status classification. The weights corresponding to the sensory modalities are then combined using a weighted sum method, decision-level fusion, or neural network fusion (e.g., a multi-layer perceptron) to determine the driver status information. For example, multimodal features can be mapped to a unified feature space, and a comprehensive status score can be generated by weighted fusion of the weights corresponding to each sensory modality to determine the driver status information as high fatigue level and medium distraction level.

[0059] In an embodiment of the present disclosure, the driver monitoring information of the multiple perception modalities may be input into a pre-trained model, and then the steps of the above embodiment are executed through the model to obtain the driver status information.

[0060] This technical solution utilizes monitoring information collected by different modal sensing devices to determine the driver's state characteristics. It then dynamically assigns weights to each modal characteristic based on the sensing device's state, effectively avoiding information bias caused by single-modal failures or environmental interference. By weightedly fusing multimodal features, it comprehensively and accurately reflects the driver's true state, providing a more reliable basis for vehicle control decisions and significantly improving driving safety.

[0061] In some possible implementations, see Figure 3 As shown, in step S112, the weight of the driver state feature corresponding to each of the perception modalities is determined based on the state information of the perception device of each of the modalities, including: The weight of the driver status feature corresponding to each of the perception modalities is determined based on the status information of the perception device of each modality within the same time period, wherein the status of the perception device represented by the status information is different, and the corresponding weight is different.

[0062] In the disclosed embodiment, based on the status information of the perception device of each of the modalities within the same time period, the features of different perception modalities can be synchronized in the time or space dimension to ensure that the multi-source features correspond to the same time period or the same driving behavior stage.

[0063] In this disclosed embodiment, the weights of driver status features at the same moment can be assigned based on threshold weights. For example, if a device status indicator (such as noise) exceeds a threshold, the corresponding modal weight is reduced. Reliability-based weighting can also be used, for example, by using the device's historical failure rate to calculate the weight coefficient. For example, with a 10% failure rate, weight = 1 - failure rate = 0.9. This quantifies the impact of device status on feature credibility and improves the robustness of the fusion results.

[0064] The above technical solution aligns the time dimension based on the sampling rate differences of different modalities, eliminates the state feature misalignment caused by the asynchronous data acquisition timing, and ensures the consistency of multimodal features on the time axis; at the same time, it dynamically adjusts the weight of each modal feature according to the real-time status information of the perception device, so that the perception modality with good status occupies a dominant position in the fusion, effectively suppressing the negative impact of faulty or interfering modalities, and improving the anti-interference ability and adaptability of the driver's status judgment.

[0065] In some possible implementations, the sensing devices corresponding to the multiple sensing modalities include at least two of the following: a wearable device, an in-vehicle sensing device, and a mobile device.

[0066] Among them, wearable devices may include smart watches, smart bracelets, etc. that are wirelessly connected to the vehicle. When worn, wearable devices can collect physiological signals such as sleep duration, heart rate, body temperature, etc., and then be used to determine the user's fatigue, health and other status information.

[0067] Among them, the on-board sensing devices may include steering wheel angle sensors, acceleration sensors, brake sensors, etc. inside the vehicle, and then monitor the driver's driving behavior information in real time through the on-board sensing devices. The driving behavior information may include, for example, sudden acceleration, sudden braking, sharp turns, etc., and then transmit the driving behavior information to the body domain controller BDC or the intelligent cockpit domain controller IDC through the CAN bus.

[0068] Among them, the on-board sensing equipment can also include an on-board microphone, which can collect the driver's breathing rate, voice changes, snoring and other sound signals in real time to further analyze his fatigue and health status.

[0069] Mobile devices, such as cell phones, can be wirelessly connected to the vehicle. Mobile devices can participate in determining driver status information if they can capture facial images or voice recordings of the user. This fusion of multi-sensory modalities—including driver monitoring information from wearable devices, mobile devices, and in-vehicle sensing devices—can provide more accurate health status analysis and driving behavior scoring than determining driver status information using a single sensory modality, thereby improving driving safety.

[0070] The above technical solution realizes multi-scenario and multi-dimensional data complementarity through at least two of wearable devices, vehicle-mounted sensing devices and mobile devices, effectively covers the status monitoring blind spots in different driving environments, and significantly enhances the comprehensiveness and real-time nature of the driver's status perception.

[0071] In some possible implementations, the target device includes an in-vehicle execution device. In step S12, executing control of the target device according to the driver state information includes: When the driver status information indicates that there is a need for driving intervention, determining a control strategy of the on-vehicle execution device according to the driver status information; Driving intervention requirements are defined as driver status information (such as fatigue, distraction, or dangerous driving behavior) that triggers system intervention to avoid accident risks. Control strategies are device operation plans tailored to different driver states, including intervention methods (such as alarms and assisted steering), intensity (such as deceleration), and priority (such as braking or avoidance).

[0072] In the disclosed embodiments, the correspondence between driver status and control strategies can be predefined. For example, fatigue (yawning frequency >3 times / minute) corresponds to the control strategy of activating Lane Keeping Assist (LKA) and gradual deceleration. Distraction (eye deviation >3 seconds) corresponds to the control strategy of triggering steering wheel vibration and voice reminders. Furthermore, driver status information corresponding to multiple sensory modalities, such as visual (fatigue), biological (heart rate), and behavioral (steering wheel operation), is integrated to generate a combined control strategy. For example, in the case of fatigue and distraction, LKA and gradual deceleration are activated simultaneously, rather than a single intervention.

[0073] The control of the vehicle-mounted execution device is performed according to the control strategy.

[0074] Among them, the on-board execution equipment is the controllable hardware in the vehicle, such as the steering system, braking system, display device, acoustic reminder device, etc.

[0075] In the disclosed embodiments, multiple onboard actuators can be included, allowing them to work in concert, such as coordinating the braking and steering systems to achieve avoidance maneuvers. For example, a control strategy can be generated that includes LKA and emergency braking. The strategy is broken down into device commands (e.g., "steering angle = 5°," "brake pressure = 8 MPa"), which then execute LKA and braking system controls.

[0076] When the above technical solution determines that a driver's condition requires intervention, it dynamically generates a hierarchical control strategy based on the specific type and severity of the condition information, directly driving the onboard actuators to respond quickly. This seamless transition from condition monitoring to safety intervention effectively shortens risk response time and provides more reliable safety for drivers and passengers.

[0077] In some possible implementations, determining a control strategy for the vehicle-mounted execution device based on the driver status information includes: determining target intervention information based on the driver status information and historical driving information; Among them, historical driving information is the driver's past driving behavior data corresponding to different driver status information, for example, the frequency of sudden braking, speeding, and lane departure when the fatigue level is medium, or the number or frequency of sudden acceleration when the distraction level is low.

[0078] In the disclosed embodiments, the same driving intervention information can be matched with different historical driving information to determine different target intervention information. Thus, the same driving intervention information obtained for different drivers can be matched with the driver's corresponding historical driving information to obtain target intervention information for each driver.

[0079] A control strategy of the vehicle-mounted execution device is determined according to the target intervention information.

[0080] In the disclosed embodiment, each intervention information corresponds to a preset vehicle-mounted execution device to be controlled and a control strategy for the vehicle-mounted device. When the target intervention information is determined, the control strategy corresponding to the intervention information can be directly called to control the vehicle-mounted execution device.

[0081] For example, if an abnormal health condition (such as fatigue or deteriorating health) is detected, the control strategy can issue a warning through the speaker and display a recommendation for the driver to rest on the central control screen and instrument panel. Furthermore, if the fatigue level is high, the control strategy can also automatically activate driving assistance (or intelligent driving) mode or reduce acceleration response sensitivity to ensure driving safety. For another example, if abnormal breathing or snoring is detected through the microphone, the control strategy can provide driving warnings and interventions. Furthermore, if minor health issues are identified, the control strategy can be implemented as health interventions, such as providing sleep duration recommendations.

[0082] This technical solution not only generates immediate intervention information based on the current driver state but also dynamically modifies this information based on historical driving behavior patterns, creating a targeted intervention plan that better suits the driver's characteristics. The resulting control strategy effectively corrects the current dangerous state while avoiding excessive intervention that could negatively impact the driving experience. This ensures safety while enhancing the comfort and adaptability of human-machine collaboration.

[0083] In some possible implementations, determining target intervention information based on the driver state information and historical driving information includes: determining an intervention threshold for the driver based on the driver status information and historical driving information; Among them, driving intervention information can generate preliminary intervention instructions based on the current driver status information (such as fatigue, distraction, dangerous behavior) and vehicle environment (such as road conditions, obstacles). The preliminary intervention instructions may include intervention type (alarm, auxiliary control), intensity (light / medium / heavy), priority and other parameters.

[0084] Among them, historical driving information is the driver's past behavior data (such as frequency of sudden braking, number of speeding, number of lane departures) and vehicle historical status (such as fault records, maintenance cycles), which are used to evaluate driver habits and vehicle performance.

[0085] The intervention threshold is the critical value that triggers driving intervention measures, used to determine whether the current driver state requires system intervention. The threshold can be set based on physiological signals (such as heart rate and blink rate), behavioral characteristics (such as number of lane departures and steering wheel grip), or environmental parameters (such as distance to the vehicle ahead). The threshold is dynamically adjusted based on historical driver data to adapt to individual differences and behavioral changes.

[0086] In the disclosed embodiments, real-time features (such as fatigue level and distraction duration) can be extracted from driving intervention information. Historical features (such as average frequency of sudden braking and number of historical distractions) can then be extracted from historical driving information. The real-time and historical features can then be mapped to a unified dimension (such as time, space, and behavior category) to ensure comparability.

[0087] Then, by training a classifier (such as a random forest or neural network) to predict intervention priorities, the system takes real-time driving intervention information and historical driving data as input, and outputs an intervention level (low / medium / high). For example, for user A, the intervention threshold is dynamically adjusted based on driving intervention information and historical data, such as "when the number of historical fatigue driving incidents exceeds three, the fatigue intervention threshold is lowered," making driving intervention easier. Meanwhile, for user B, the intervention threshold is dynamically adjusted based on driving intervention information and historical data, such as "when the number of historical fatigue driving incidents is less than three, the fatigue intervention threshold is raised," making driving intervention more difficult.

[0088] The target intervention information for the driver is determined according to the intervention threshold, wherein different intervention thresholds correspond to different preset intervention information.

[0089] The intervention threshold can be linked to a specific intervention measure. A higher intervention threshold indicates a lower intervention intensity, while a lower intervention threshold indicates a higher intervention intensity. Intervention strategies can also be graded based on the intervention threshold. For example, intervention levels (e.g., low, medium, high) can be assigned to different intervention measures (e.g., reminders, auxiliary control, emergency braking) based on the intervention threshold range.

[0090] For example, the relationship between predefined threshold ranges and intervention information. For example: Threshold range [0, 50]: Alarm only (such as the voice prompt "Please pay attention"). Threshold range (50, 80]: Alarm + mild assistance (such as steering wheel vibration). Threshold range (80, 100]: Alarm + emergency control (such as automatic braking).

[0091] In the disclosed embodiments, driving intervention information can include sleep and rest recommendations for the driver, such as recommended rest and sleep times. This intelligent generation of tiered intervention thresholds combines the driver's current state (e.g., fatigue level, distraction level) with historical driving behavior patterns (e.g., response speed to warnings, risk aversion habits, etc.), avoiding excessive intervention or delayed response caused by a single fixed threshold. For example, for drivers who are accustomed to responding quickly to warnings, the system can appropriately relax the early warning threshold; while for users who frequently ignore prompts, stronger intervention can be triggered in advance. This dynamic threshold adjustment mechanism, based on individual behavioral characteristics, ensures that safety interventions are more tailored to the driver's actual needs.

[0092] This technical solution generates a driver-specific health trend model through long-term data tracking and analysis, providing personalized long-term health improvement recommendations. This provides each driver with personalized driving behavior analysis and health feedback, continuously optimizing the driving experience and health status, thereby helping drivers improve their health. By integrating with smart home devices at home, the solution optimizes the driver's home environment, achieving a closed-loop "person-car-home" experience and enhancing quality of life and the driving experience.

[0093] In some possible implementations, the target device includes a home device, see Figure 4 As shown, in step S12, executing the control of the target device according to the driver status information includes: In step S121, target control parameters of the home appliance are determined according to the driver status information; Home appliances are controllable devices in a smart home system, such as air conditioners, lights, curtains, audio equipment, and security devices. Target control parameters are specific control command parameters for home appliances, such as the air conditioner temperature setting (26°C), light brightness percentage (50%), and curtain opening / closing ratio (30%).

[0094] In this disclosed embodiment, a mapping between driver status information and home appliance control parameters can be defined, such as "When the driver is fatigued, reduce light brightness to 30%." This allows the driver's status information to be converted into specific control parameters for home appliances, enabling a linkage between the driver's status and the home environment.

[0095] In the disclosed embodiments, multi-device collaboration can be achieved, allowing for the coordinated control of multiple home appliances. For example, when a driver is fatigued, the air conditioning temperature, light brightness, and curtain opening / closing ratio can be adjusted simultaneously. Priorities can also be defined for different control parameters, such as giving temperature control a higher priority than light control.

[0096] In step S122, the target control parameter is sent to the home appliance, where the target control parameter is used to control the home appliance.

[0097] In the disclosed embodiment, the target control parameters can be sent to corresponding home appliances through the Internet of Vehicles system, thereby controlling the home appliances to make adjustments in advance. Furthermore, feedback information after the home appliances execute the target control parameters can be received, and the current status of the home appliances can be fed back to the vehicle side to achieve state synchronization.

[0098] This technical solution automatically generates adaptive home environment control parameters based on real-time information such as the driver's fatigue level, emotional state, or health indicators, and sends precise control commands to home devices via the connected car. This cross-scenario proactive service mechanism not only creates a personalized home environment for users, but also enhances convenience by proactively adjusting device status. It also avoids the safety risks of manual operation by the driver, expanding the service boundaries and user experience value of smart cars.

[0099] In this way, combining multi-sensory modal driver monitoring information to determine driver status information can not only realize the control of on-board execution equipment, but also link home appliances to provide personalized in-car experience and smart home experience.

[0100] In some possible implementations, the household appliances include at least one of the following: curtains, beds, lights, stereos, air conditioners, doors and windows.

[0101] In the disclosed embodiment, home appliances can be adjusted in advance based on the driver's status information, such as optimizing the sleeping environment through smart curtains and lighting control; the air-conditioning temperature in the home can be adjusted in advance when the driver is tired to provide a comfortable environment for him when he returns home.

[0102] In this way, if the driver's status information indicates a need for driver intervention, such as fatigue, the system can synchronize control of home appliances and interact with home appliances (such as smart beds and smart air purifiers) to automatically adjust the home environment based on the driver's status information, ensuring the driver gets a good rest upon returning home. At the same time, the driver's status information can be synchronized and transmitted to home appliances, which can then determine control parameters based on the driver's status information to optimize the home environment. Based on the driver's status, refined control can be achieved, such as adjusting the lighting atmosphere, presetting the air conditioning temperature and humidity, and automatically opening and closing doors and windows, creating a personalized home environment for the user, significantly improving living comfort and the intelligent experience.

[0103] According to an embodiment of the present disclosure, a control device is also provided. Figure 5 Shown, including: a determination module 510 configured to determine driver status information based on driver monitoring information of multiple sensing modalities, wherein the driver monitoring information of the multiple sensing modalities is obtained through sensing devices of different modalities; The control module 520 is configured to control the target device according to the driver status information.

[0104] In some possible implementations, the determining module 510 is configured to: determining a driver state feature corresponding to each of the sensing modalities based on the driver monitoring information of the plurality of sensing modalities; Determining, based on the state information of the sensing device of each modality, the weight of the driver state feature corresponding to each sensing modality; The driver state features corresponding to multiple perception modalities are fused according to the weight corresponding to each perception modality to determine the driver state information.

[0105] In some possible implementations, the determining module 510 is configured to: The weight of the driver status feature corresponding to each of the perception modalities is determined based on the status information of the perception device of each modality within the same time period, wherein the status of the perception device represented by the status information is different, and the corresponding weight is different.

[0106] In some possible implementations, the sensing devices corresponding to the multiple sensing modalities include at least two of the following: a wearable device, an in-vehicle sensing device, and a mobile device.

[0107] In some possible implementations, the target device includes an in-vehicle execution device, and the control module 520 is configured to: When the driver status information indicates that there is a need for driving intervention, determining a control strategy of the on-vehicle execution device according to the driver status information; The control of the vehicle-mounted execution device is performed according to the control strategy.

[0108] In some possible implementations, the control module 520 is configured to: determining target intervention information based on the driver status information and historical driving information; A control strategy of the vehicle-mounted execution device is determined according to the target intervention information.

[0109] In some possible implementations, the control module 520 is configured to: determining an intervention threshold for the driver based on the driver status information and historical driving information; The target intervention information for the driver is determined according to the intervention threshold, wherein different intervention thresholds correspond to different preset intervention information.

[0110] In some possible implementations, the target device includes a home device, and the control module 520 is configured to: determining target control parameters of the home appliance according to the driver status information; The target control parameter is sent to the home device, where the target control parameter is used to execute control of the home device.

[0111] In some possible implementations, the household appliances include at least one of the following: curtains, beds, lights, stereos, air conditioners, doors and windows.

[0112] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0113] According to an embodiment of the present disclosure, a vehicle is further provided, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to execute the executable instructions stored in the memory to implement the method described in any one of the aforementioned embodiments.

[0114] According to an embodiment of the present disclosure, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in any one of the aforementioned embodiments are implemented.

[0115] According to an embodiment of the present disclosure, a computer program product is further provided, including a computer program, which implements the steps of any one of the methods in the aforementioned embodiments when executed by a processor.

[0116] Figure 6 FIG6 is a block diagram illustrating a vehicle 600 according to an exemplary embodiment. For example, vehicle 600 may be a hybrid vehicle, a non-hybrid vehicle, an electric vehicle, a fuel cell vehicle, or another type of vehicle. Vehicle 600 may be an autonomous vehicle, a semi-autonomous vehicle, or a non-autonomous vehicle.

[0117] Reference Figure 6 Vehicle 600 may include various subsystems, such as an infotainment system 610, a perception system 620, a decision control system 630, a drive system 640, and a computing platform 650. Vehicle 600 may also include more or fewer subsystems, and each subsystem may include multiple components. Furthermore, each subsystem and each component of vehicle 600 may be interconnected via wired or wireless means.

[0118] In some embodiments, the infotainment system 610 may include a communication system, an entertainment system, a navigation system, and the like.

[0119] The perception system 620 may include several sensors for sensing information about the environment surrounding the vehicle 600. For example, the perception system 620 may include a global positioning system (which may be a GPS system, a BeiDou system, or another positioning system), an inertial measurement unit (IMU), a laser radar, a millimeter-wave radar, an ultrasonic radar, and a camera.

[0120] The decision control system 630 may include a computing system, a vehicle controller, a steering system, a throttle, and a braking system.

[0121] The drive system 640 may include components that provide power to the vehicle 600. In one embodiment, the drive system 640 may include an engine, an energy source, a transmission system, and wheels. The engine may be an internal combustion engine, an electric motor, an air compression engine, or a combination thereof. The engine is capable of converting energy provided by the energy source into mechanical energy.

[0122] Some or all functions of the vehicle 600 are controlled by a computing platform 650. The computing platform 650 may include at least one processor 651 and a memory 652. The processor 651 may execute instructions 653 stored in the memory 652.

[0123] The processor 651 can be any conventional processor, such as a commercially available CPU. The processor can also include a graphics processor (GPU), a field programmable gate array (FPGA), a system on chip (SOC), an application specific integrated circuit (ASIC), or a combination thereof.

[0124] The memory 652 may be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0125] In addition to instructions 653 , memory 652 may also store data, such as road maps, route information, and vehicle location, direction, speed, etc. The data stored in memory 652 may be used by computing platform 650 .

[0126] In the embodiment of the present disclosure, the processor 651 may execute the instruction 653 to complete all or part of the steps of the above-mentioned control method.

[0127] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the appended claims.

[0128] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A control method, characterized in that: include: determining driver status information based on driver monitoring information of multiple sensing modalities, wherein the driver monitoring information of the multiple sensing modalities is obtained through sensing devices of different modalities; Control of a target device is performed based on the driver state information.

2. The method according to claim 1, characterized in that Determining the driver status information based on the driver monitoring information of the multiple sensing modalities includes: determining a driver state feature corresponding to each of the sensing modalities based on the driver monitoring information of the plurality of sensing modalities; Determining, based on the state information of the sensing device of each modality, the weight of the driver state feature corresponding to each sensing modality; The driver state features corresponding to multiple perception modalities are fused according to the weight corresponding to each perception modality to determine the driver state information.

3. The method according to claim 2, characterized in that Determining the weight of the driver status feature corresponding to each perception modality based on the status information of the perception device of each modality includes: The weight of the driver status feature corresponding to each of the perception modalities is determined based on the status information of the perception device of each modality within the same time period, wherein the status of the perception device represented by the status information is different, and the corresponding weight is different.

4. The method according to claim 1, wherein The sensing devices corresponding to the multiple sensing modalities include at least two of the following: wearable devices, vehicle-mounted sensing devices, and mobile devices.

5. The method according to any one of claims 1 to 4, characterized in that The target device includes an in-vehicle execution device, and the controlling of the target device according to the driver state information includes: When the driver status information indicates that there is a need for driving intervention, determining a control strategy of the on-vehicle execution device according to the driver status information; The control of the vehicle-mounted execution device is performed according to the control strategy.

6. The method according to claim 5, characterized in that Determining the control strategy of the vehicle-mounted execution device according to the driver status information includes: determining target intervention information based on the driver status information and historical driving information; A control strategy of the vehicle-mounted execution device is determined according to the target intervention information.

7. The method according to claim 6, characterized in that The determining target intervention information according to the driver state information and historical driving information includes: determining an intervention threshold for the driver based on the driver status information and historical driving information; The target intervention information for the driver is determined according to the intervention threshold, wherein different intervention thresholds correspond to different preset intervention information.

8. The method according to any one of claims 1 to 4, characterized in that The target device includes a home device, and the controlling of the target device according to the driver status information includes: determining target control parameters of the home appliance according to the driver status information; The target control parameter is sent to the home device, where the target control parameter is used to execute control of the home device.

9. The method according to claim 8, characterized in that The household appliances include at least one of the following: curtains, beds, electric lights, stereos, air conditioners, doors and windows.

10. A control device, characterized in that: include: a determination module configured to determine driver status information based on driver monitoring information of multiple sensing modalities, wherein the driver monitoring information of the multiple sensing modalities is obtained through sensing devices of different modalities; The control module is configured to control the target device according to the driver status information.

11. The device according to claim 10, characterized in that The determining module is configured to: determining a driver state feature corresponding to each of the sensing modalities based on the driver monitoring information of the plurality of sensing modalities; Determining, based on the state information of the sensing device of each modality, the weight of the driver state feature corresponding to each sensing modality; The driver state features corresponding to multiple perception modalities are fused according to the weight corresponding to each perception modality to determine the driver state information.

12. The device according to claim 11, characterized in that The determining module is configured to: The weight of the driver status feature corresponding to each of the perception modalities is determined based on the status information of the perception device of each modality within the same time period, wherein the status of the perception device represented by the status information is different, and the corresponding weight is different.

13. The device according to claim 10, characterized in that The sensing devices corresponding to the multiple sensing modalities include at least two of the following: wearable devices, vehicle-mounted sensing devices, and mobile devices.

14. The device according to any one of claims 10 to 13, characterized in that The target device includes an in-vehicle execution device, and the control module is configured to: When the driver status information indicates that there is a need for driving intervention, determining a control strategy of the on-vehicle execution device according to the driver status information; The control of the vehicle-mounted execution device is performed according to the control strategy.

15. The device according to any one of claims 10 to 13, characterized in that The target device includes a home device, and the control module is configured to: determining target control parameters of the home appliance according to the driver status information; The target control parameter is sent to the home device, where the target control parameter is used to execute control of the home device.

16. A vehicle, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute the executable instructions stored in the memory to implement the method according to any one of claims 1 to 9.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

18. A computer program product, characterized in that The invention comprises a computer program, which implements the steps of the method according to any one of claims 1 to 9 when the computer program is executed by a processor.

Images