Methods, apparatus, systems, and vehicles for sensing a driver state

Abstract

The present disclosure provides methods, devices, systems and vehicles for perceiving a driver state. In perceiving the driver state, at least two modalities of driver state observations perceived via driver state sensors are obtained; for each modality of the at least two modalities of driver state observations, a driver state perception result corresponding to the modality of driver state observations is determined based on the modality of driver state observations and a software recognition capability of a corresponding driver state sensor, the determined driver state perception result being indicative of state occurrence probabilities of various driver states under the modality of driver state observations. Then, the driver state perception results corresponding to the at least two modalities of driver state observations are fused to determine the driver state. With the driver state perception method, the accuracy of driver state perception can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of vehicle driving, and more specifically, to methods, devices, systems, and vehicles for sensing the state of a driver. Background Technology

[0002] With the continuous development of vehicle intelligence technology, advanced driver assistance systems (ADAS) are gradually becoming standard features in intelligent vehicles. While vehicles equipped with systems such as Level 3 (L3) can achieve autonomous driving in limited scenarios, they still require the driver to remain focused at all times to take over promptly in case of system prompts or emergencies, thereby preventing accidents. The driver's level of focus directly determines the timeliness and effectiveness of takeover, necessitating the deployment of a driver state perception system in the vehicle to monitor and ensure the accuracy of this perception during driving.

[0003] Currently, driver state perception systems primarily rely on single-modal perception, such as visual signal perception, mechanical signal perception, and physiological signal perception. Visual signal perception, for example, uses in-vehicle cameras to detect the driver's face, eyes, and head posture to determine whether they are looking ahead or exhibiting signs of fatigue such as closed eyes or yawning. Mechanical signal perception uses mechanical signals such as steering wheel grip force, steering operation, and pedal pressure to infer whether the driver is in a state of control. Physiological signal perception uses physiological data such as the driver's heart rate and respiratory rate to infer the driver's state.

[0004] However, visual signal perception, primarily based on the driver's physical posture, struggles to accurately reflect the driver's true cognitive state, thus failing to effectively identify states such as driver inattention or wandering thoughts. Mechanical signal perception often only assesses the physical contact between the driver and the vehicle, failing to distinguish between the driver's active control intentions and passive contact behaviors, thereby failing to fully reflect the driver's actual takeover ability. Furthermore, in complex driving environments, physiological signal data is susceptible to motion artifacts and environmental noise interference, leading to insufficient reliability and impacting the accuracy of driver state perception. Summary of the Invention

[0005] This disclosure aims to provide a method, apparatus, system, and vehicle for sensing driver state. Using this method, the accuracy of driver state sensing can be improved.

[0006] According to a first aspect of this disclosure, a method for sensing a driver's state is provided, comprising: acquiring driver state observation results of at least two modalities sensed by a driver state sensor; for each modal of driver state observation results, determining a driver state perception result corresponding to that modal of driver state observation results based on the driver state observation results of that modality and the software recognition capability of the corresponding driver state sensor, wherein the driver state perception result is used to indicate the probability of occurrence of various driver states under the driver state observation results of that modality; and performing perceptual fusion on the driver state perception results corresponding to the at least two modalities of driver state observation results to determine the driver state.

[0007] Optionally, in one example of the above aspects, for each of the at least two modalities of driver state observation results, determining the driver state perception result corresponding to the driver state observation result of that modality based on the driver state observation result of that modality and the software recognition capability of the corresponding driver state sensor may include: for each of the at least two modalities of driver state observation results, determining the driver state perception result corresponding to the driver state observation result of that modality based on the vehicle driving scenario and the software recognition capability of the driver state observation result of that modality and the corresponding driver state sensor.

[0008] Optionally, in one example of the above aspects, for each of the at least two modalities of driver state observation results, determining the driver state perception result corresponding to that modality of driver state observation results, based on the vehicle driving scenario and the software recognition capability of the driver state observation results and the corresponding driver state sensor, may include: for each of the at least two modalities of driver state observation results, obtaining the driver state perception result corresponding to that modality of driver state observation results from a driver state observation result-driver state perception result relationship table, wherein the driver state observation result-driver state perception result relationship table is constructed based on the vehicle driving scenario and the software recognition capability of the driver state sensor.

[0009] Optionally, in one example of the above aspects, the software recognition capability of the driver state sensor corresponding to the driver state observation results of each modality includes the algorithm processing capability of the driver state sensor corresponding to the driver state observation results of that modality when sensing the driver state observation results of that modality.

[0010] Optionally, in one example of the above aspects, the software recognition capability of the driver state sensor corresponding to the driver state observation results of each modality is characterized as a state detection capability factor indicating the perception probability of various driver state observation results given by the driver state sensor under a known real driver state, and the vehicle driving scenario is characterized as a driving scenario factor indicating the probability of occurrence of various driver states under the vehicle driving scenario. Accordingly, the driver state observation result-driver state perception result relationship table can be constructed based on the following construction process: for each modality of driver state observation results, the probability of occurrence of various driver states under the driver state observation results of that modality is determined based on the state detection factor and driving scenario factor corresponding to the driver state observation results of that modality, as the driver state perception result corresponding to the driver state observation results of that modality; and the driver state observation result-driver state perception result relationship table is constructed according to the driver state observation results of each modality and the corresponding driver state perception result.

[0011] Optionally, in one example of the above aspects, determining the driver's state by performing perceptual fusion on the driver's state perception results corresponding to the driver's state observation results of the at least two modalities may include: performing perceptual fusion using the driver's state perception results corresponding to the driver's state observation results of the at least two modalities and the fusion weights corresponding to the driver's state observation results of each modality to determine the driver's state.

[0012] Optionally, in one example of the above aspects, the fusion weight corresponding to the driver state observation results of each modality may include at least one of the following fusion weights: a fusion weight reflecting the hardware sensing capability of the driver state sensor corresponding to the driver state observation results of that modality under ideal conditions; a fusion weight reflecting the driver's modal perception adaptation to the driver state observation results of that modality; a fusion weight reflecting the data reliability of the driver state sensing data sensed by the driver state sensor corresponding to the driver state observation results of that modality; and a fusion weight reflecting the influence of the driving environment on the sensing capability of the driver state sensor corresponding to the driver state observation results of that modality.

[0013] Optionally, in one example of the above aspects, the fusion weights include at least two fusion weights. The method may further include: performing weight fusion on the fusion weights. Accordingly, determining the driver's state using the driver's state perception results corresponding to the at least two modalities of driving state observations and the fusion weights corresponding to the driver's state observation results of each modality may include: determining the driver's state using the driver's state perception results corresponding to the at least two modalities of driving state observations and the fusion weights corresponding to the driver's state observation results of each modality after weight fusion.

[0014] Optionally, in one example of the above aspects, weight fusion of the fusion weights may include: normalizing the weight fusion of the fusion weights.

[0015] Optionally, in one example of the above aspects, the driver state observations of the at least two modalities have perceptual synergy in driver state perception.

[0016] Alternatively, in one example of the above aspects, the driver state sensor may be selected from a given set of driver state sensors based on at least one of sensor cost and engineering feasibility, as well as the perceptual synergy of the driver state observations sensed by the driver state sensor in driver state perception.

[0017] According to another aspect of the embodiments of this disclosure, an apparatus for sensing a driver's state is provided, comprising: a state observation result acquisition unit configured to acquire driver state observation results of at least two modalities sensed by a driver state sensor; a state perception result determination unit configured to, for each of the at least two modalities of driver state observation results, determine a driver state perception result corresponding to that modality of driver state observation results based on the driver state observation result of that modality and the software recognition capability of the corresponding driver state sensor, wherein the driver state perception result is used to indicate the probability of occurrence of various driver states under the driver state observation result of that modality; and a state perception result fusion unit configured to perform perception fusion on the driver state perception results corresponding to the at least two modalities of driver state observation results to determine the driver state.

[0018] According to another aspect of the embodiments of the present disclosure, a system for sensing a driver's state is provided, comprising: a driver state sensor; and the means for sensing a driver's state as described above.

[0019] According to another aspect of the embodiments of this disclosure, a vehicle is provided, including: a system for sensing the driver's state as described above.

[0020] According to another aspect of the embodiments of the present disclosure, an apparatus for sensing a driver's state is provided, comprising: at least one processor; a memory coupled to the at least one processor; and a computer program stored in the memory, wherein the at least one processor executes the computer program to implement the method for sensing a driver's state as described above.

[0021] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided that stores executable instructions, which, when executed, cause a processor to perform the method for sensing a driver's state as described above.

[0022] According to another aspect of the embodiments of this disclosure, a computer program product is provided, including a computer program that, when run by a processor, performs the method for sensing a driver's state as described above. Attached Figure Description

[0023] A further understanding of the nature and advantages of this specification can be achieved by referring to the following figures. In the figures, similar components or features may have the same reference numerals.

[0024] Figure 1 A schematic diagram of an example architecture of a driver state perception system according to an embodiment of the present disclosure is shown.

[0025] Figure 2 An example flowchart of a method for sensing a driver's state according to an embodiment of the present disclosure is shown.

[0026] Figure 3 An example schematic diagram of a state detection capability factor according to an embodiment of the present disclosure is shown.

[0027] Figure 4 An example flowchart is shown illustrating the process of constructing a driver state observation-driver state perception result relationship table according to an embodiment of the present disclosure.

[0028] Figure 5 An example schematic diagram of a driver state observation-driver state perception result relationship table according to an embodiment of the present disclosure is shown.

[0029] Figure 6 An example schematic diagram of a driver state perception process according to an embodiment of the present disclosure is shown.

[0030] Figure 7 An example block diagram of a driver state perception device according to an embodiment of the present disclosure is shown.

[0031] Figure 8An example block diagram of a relation table construction unit according to an embodiment of the present disclosure is shown.

[0032] Figure 9 An example block diagram of a driver state perception device implemented using a computer system according to an embodiment of the present disclosure is shown. Detailed Implementation

[0033] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed merely to enable those skilled in the art to better understand and implement the subject matter described herein, and are not intended to limit the scope, applicability, or examples set forth in the claims. The function and arrangement of the elements discussed may be changed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the various examples. For example, the described methods may be performed in a different order than described, and steps may be added, omitted, or combined. Furthermore, features described in some examples may be combined in other examples.

[0034] As used herein, the term "comprising" and its variations are open terms meaning "including but not limited to". The term "based on" means "at least partially based on". The terms "one embodiment" and "an embodiment" mean "at least one embodiment". The term "another embodiment" means "at least one other embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other definitions, whether explicit or implicit, may be included below. Unless explicitly indicated by the context, the definition of a term shall remain consistent throughout the specification.

[0035] The flowcharts used in this specification illustrate operations implemented according to some embodiments of this specification. It should be clearly understood that the operations in the flowcharts may not be implemented in a sequential order. Instead, the operations may be implemented in reverse order or simultaneously. Furthermore, one or more additional operations may be added to the flowcharts. One or more operations may be removed from the flowcharts.

[0036] The following description, in conjunction with the accompanying drawings, describes a method, apparatus, and system for sensing a driver's state, as well as a vehicle having the system, according to embodiments of the present disclosure.

[0037] Figure 1 An example architecture diagram of a driver state perception system 100 according to an embodiment of the present disclosure is shown.

[0038] like Figure 1As shown, the driver state perception system 100 includes a driver state sensor group and a driver state perception device 120. The driver state sensor group can communicate with the driver state perception device 120 via a network connection, for example, by transmitting driver state observation results perceived by the driver state sensors. In some embodiments, the network can be any one or more of a wired network or a wireless network. Examples of networks may include, but are not limited to, vehicle networks, the Internet, local area networks (LANs), wide area networks (WANs), wireless local area networks (WLANs), metropolitan area networks (MANs), public switched telephone networks (PSTNs), Bluetooth networks, ZigZee networks, near field communication (NFC), controller area network (CAN) buses, device internal buses, device internal wiring, etc., or any combination thereof. In some embodiments, one or more of the driver state sensor group can also be directly connected to the driver state perception device 120 without a network.

[0039] The driver state sensor array may include at least one driver state sensor. Each driver state sensor is used to sense driver state observations in at least one modality. A driver state observation in one modality may be a driver state determined based on driver state data of that modality, that is, a driver state determined based on raw sensing data. For example, suppose the driver state includes a state of focus. and abnormal states (For example, fatigue or distraction), then the driver state observation results of one modality can or In some embodiments, and The value can be a Boolean value of 1 or 0. For example, when the driver's state is determined to be a focused state, and The values ​​are 1 and 0. When the driver's status is determined to be abnormal, and The values ​​are 0 and 1, respectively. In some embodiments, the driver state observation results output by the driver sensor can be vector values ​​( ),in, and These represent the perceived probabilities of a focused state and an abnormal state, respectively. For example, assuming the probability of a driver's state being identified as focused is 80% and the probability of it being identified as abnormal is 20%, then... and The values ​​are 0.8 and 0.2 respectively.

[0040] Driver state data for each modality can include driver state data sensed by different types of state sensors for the same perceived target, or driver state data sensed by different types of state sensors for different perceived targets. Driver state data of various modalities may include, but is not limited to, driver posture data, seat posture data, steering wheel take-off status data, driver physiological characteristic data, etc., and may also include driver state data in various data formats, such as video data, image data, and audio data. For example, regarding driver posture data, if the driver state data sensed by different types of state sensors has different data formats, it can also be referred to as driver posture data of different modalities.

[0041] In some embodiments, there is perceptual synergy among at least two modalities of driver state observations sensed by the driver state sensor group. In this specification, the term "perceptual synergy" refers to the ability of various modalities of driver state observations to mutually compensate for each other's deficiencies or defects in driver state perception capabilities, thereby improving the accuracy of driver state perception. In some embodiments, "perceptual synergy" may refer to the perceptual synergy of different modalities of driver state observations for the same sensing target, for example, the perceptual synergy between driver state observations from a visible light camera and an infrared camera regarding the driver's line of sight. In some embodiments, "perceptual synergy" may refer to the perceptual synergy of different modalities of driver state observations for different sensing targets, for example, the perceptual synergy between driver state observations from a visible light camera regarding the driver's line of sight and driver state observations from a seat sensor regarding the driver's posture.

[0042] In some embodiments, the driver state sensor group may include at least two of the following driver state sensors: driver posture sensor, seat posture sensor, steering wheel sensor, pedal pressure sensor, and physiological sign sensor.

[0043] Driver posture sensors are used to sense the driver's posture data within the cockpit, such as head position, upper body posture, hand movements, and / or gaze direction. Based on this driver posture data, it is possible to analyze whether the driver is in a normal, fatigued, or distracted state. Examples of driver posture sensors may include, but are not limited to, visible light cameras, infrared cameras, depth cameras, infrared thermal imaging sensors, and inertial measurement units.

[0044] Seat posture sensors are used to sense the driver's seat posture, such as seat pressure and seat angle. Seat posture is an important indicator of the driver's state. For example, an excessively large seat angle can reduce the driver's ability to take control. Examples of seat posture sensors may include, but are not limited to, seat pressure sensors, seat angle sensors, weight sensors, tilt sensors, inertial measurement units, capacitive contact sensors, and fiber optic sensors.

[0045] A steering wheel sensor is used for hands-off detection. Examples of steering wheel-related sensors may include, but are not limited to, torque sensors, grip force sensors, pressure distribution sensors, capacitive touch sensors, resistive touch sensors, current sensing sensors, angle sensors, and conductance skin sensors. For example, hands-off detection can be performed by sensing the pressure of the driver's hands on the steering wheel. If the steering wheel pressure is less than a predetermined threshold, it is considered that the driver's hands have left the steering wheel, i.e., the driver is in a hands-off state. If the steering wheel pressure is not lower than the predetermined threshold, it is considered that the driver's hands are on the steering wheel. A hands-off state indicates weak driver control and a lack of driver focus. A hands-on state indicates strong driver control and a focused driver. Examples of steering wheel sensors may include, but are not limited to, steering wheel pressure sensors.

[0046] The pedal pressure sensor is used to sense pedal pressure. If the sensed pedal pressure is less than a predetermined threshold, it indicates that the driver's ability to take over driving is weak, and therefore the driver is not focused. If the sensed pedal pressure is not less than the predetermined threshold, it indicates that the driver's ability to take over driving is strong, and therefore the driver is focused.

[0047] Physiological characteristic sensors are used to sense the driver's physiological characteristic data, such as heart rate and respiratory rate. Physiological characteristic sensors may include, for example, infrared cameras and heart rate monitors. For instance, a near-infrared enhanced cabin camera can be deployed inside the vehicle's cabin, with a near-infrared supplemental light integrated into the camera module, using non-contact detection methods such as rPPG (remote photoplethysmography) to sense the driver's respiratory rate and heart rate.

[0048] It should be noted that each modality of driver state sensor may include a single sensor or multiple sensors deployed at different locations within the vehicle. Additionally, the driver state sensor group may optionally include other types or modalities of driver state sensors.

[0049] In some embodiments, a driver state sensor group can be selected from a given set of driver state sensors based on the perceptual synergy of driver state observations sensed by the driver state sensors in driver state perception. In this specification, the term "perceptual synergy" is used as a measure reflecting the perceptual synergy of various modal driver state observations in driver state perception, and can be characterized, for example, using perceptual synergy efficiency. In some embodiments, perceptual synergy efficiency can be evaluated using experimental data. In some embodiments, experimental data can be used to evaluate the perceptual synergy effect of combinations of various modal driver state observations in driver state perception. In some embodiments, the signal correlation, information redundancy, and information gain between the sensor sensing signals corresponding to various modal driver state observations sensed by the driver state sensors can be determined, and the perceptual synergy of various modal driver state observations in driver state perception can be determined based on the signal correlation, information redundancy, and information gain. Then, driver state sensors are selected based on the determined perceptual synergy.

[0050] In some embodiments, a driver state sensor group can be selected from a given set of driver state sensors based on at least one of sensor cost and engineering feasibility, as well as the perceptual synergy between the driver state observations perceived by the driver state sensors and the driver state perception. For example, for mass-produced vehicles, sensors with advantages in cost and power consumption can be prioritized, while also considering layout space, installation convenience, and compatibility with the vehicle's electrical architecture. Based on this, and combining the perceptual synergy between the driver state observations perceived by each driver state sensor and the driver state perception, a driver state sensor group that achieves optimal perceptual synergy can be selected from a given set of driver state perception sensors.

[0051] Preferably, the selected driver state sensor group covers as many different types of data sources as possible. For example, visual sensors can provide the driver's facial posture and gaze direction, mechanical sensors can provide hand contact status, and physiological characteristic sensors can provide vital signs data such as heart rate and respiration. By combining visual, mechanical, and physiological characteristic sensors, it is possible to ensure more comprehensive and reliable driver state perception.

[0052] In some embodiments, for the driver state sensor group deployed on the vehicle, a dynamic optimization strategy can also be used for driver state sensor enable control. For example, based on changes in the vehicle operating scenario and driving task, some driver state sensors in the driver state sensor group can be activated / disabled, thereby reducing the number of driver state sensors used while ensuring perception accuracy, thus reducing energy consumption and computational load. For example, the driver state perception system may also include a driver perception enabling device. The driver perception enabling device is configured to select driver state sensors for driver state perception from the driver state sensor group based on the perceptual coordination of driver state observations sensed by the driver state sensors in driver state perception, and enable the selected driver state sensors to perform driver state perception.

[0053] In some embodiments, the driver state sensing data (raw state sensing data) collected by the driver state sensor array in various modalities can be processed by data processing such as signal filtering, timestamp alignment, and outlier removal, and the driver state observation results can be determined based on the processed driver state sensing data. By using the above-mentioned data processing on the raw state sensing data, interference factors in the driver state sensing data of various modalities can be eliminated, and data time alignment can be achieved.

[0054] The driver state perception device 120 can be deployed in the vehicle as a hardware module or integrated into the vehicle system as a software module, for example, integrated into a vehicle driver assistance system or a central domain controller. After sensing various modal driver state observations, the driver state sensor array can provide these observations to the driver state perception device 120. The driver state perception device 120 determines the driver state based on the received various modal driver state observations.

[0055] Figure 2 An example flowchart of a method 200 for sensing a driver's state according to an embodiment of the present disclosure is shown.

[0056] like Figure 2 As shown, at 210, driver state observations in at least two modalities perceived by the driver state sensor are acquired.

[0057] The driver state sensor group may include at least one driver state sensor. When the driver state sensor group includes one driver state sensor, that driver state sensor can sense driver state observations in at least two modalities. When the driver state sensor group includes at least two driver state sensors, each driver state sensor can sense driver state observations in at least one modality. After sensing at least two modalities of driver state observations, the driver state sensor group provides the sensed driver state observations to the driver state sensing device 120 via a wired or wireless network, thereby enabling the driver state sensing device 120 to acquire driver state observations in at least two modalities.

[0058] Assume the set of states is defined as follows: Among them, abnormalities include driver fatigue, driver distraction, and driver hands-off driving. The driver state observation results of at least two modes acquired by the driver state perception device 120 constitute the driver state observation result set. ,in, These represent the driver state observations in different modalities sensed by the driver state sensor, used to represent the driver state determined by the sensor based on sensor data from different modalities. For example, The conclusions of the driver's state perception based on the driver's posture, head position, and eye attention are obtained from the visible light band camera inside the cabin. The in-cabin near-infrared camera provides driver state perception conclusions regarding attention and takeover ability based on the driver's breathing rate and heart rate. The driver's state perception conclusions regarding attention and takeover ability provided by the seat sensors based on seat posture (including adjustment status), and The driver's state perception conclusions, based on the steering wheel sensor's assessment of attention and takeover ability, are derived from the steering wheel being out of control (including the application of control torque, etc.).

[0059] Considering the inherent characteristics of different driver state sensors, the meanings of the driver state sensing data sensed by these sensors are not entirely consistent. For ease of discussion, state perception software can be embedded in the sensors, and the sensor state sensing data can be summarized into state perception conclusions based on the state perception software, such as classifying it into a specific state (e.g., "focused" or "abnormal"), thereby achieving consistent processing of driver state sensing data.

[0060] In 220, for each of the at least two modal driver state observation results, based on the driver state observation results of that modality and the software recognition capability of the corresponding driver state sensor, a driver state perception result corresponding to the driver state observation results of that modality is determined, and the determined driver state perception result is used to indicate the probability of occurrence of various driver states under the driver state observation results of that modality.

[0061] In some embodiments, the software recognition capability of the driver state sensor corresponding to each modality of driver state observation results may include the algorithm processing capability of the driver state sensor when sensing the driver state observation results of that modality. For example, the software recognition capability of the driver state sensor corresponding to each modality of driver state observation results can be characterized as a state detection capability factor used to indicate the perception probability of various driver state observation results given by the driver state sensor under known real driver conditions. Each modality of driver state observation results corresponds to a state detection capability factor, which reflects the inherent state detection capability of the driver state sensor when detecting the driver state observation results of that modality, that is, the inherent detection capability of the driver state sensor for the driver state observation results of that modality (e.g., the algorithm processing capability of the driver state sensor for the driver state perception data of that modality). The state detection capability factor can be determined based on historical sensing data. For example, it can be determined based on the statistical results of historical sensing data.

[0062] Figure 3 An example schematic diagram of a state detection capability factor according to an embodiment of the present disclosure is shown.

[0063] exist Figure 3 The diagram illustrates the state detection capability factors for driver state observations across four modalities, characterized using conditional probabilities. Conditional probabilities Indicates the known true state of the driver for At that time, the first Driver state observation results detected by individual driver state sensors for The probability, It indicates the driver's true state, such as focus or abnormality (including distraction, fatigue, and loss of control).

[0064] like Figure 3As shown, the visible light band camera senses the driver's posture data within the cockpit, such as upper body posture, head position, and direction of gaze. When the actual state is "focused," the visible light band camera detects the driver's state as "focused" with a probability of 0.85, and as "abnormal" with a probability of 0.15. When the actual state is "abnormal," the visible light band camera detects the driver's state as "focused" with a probability of 0.2, and as "abnormal" with a probability of 0.8.

[0065] For near-infrared cameras, the data sensed is the driver's physiological characteristics, such as heart rate and respiratory rate. When the actual driver is "focused," the probability of the near-infrared camera detecting a "focused" driver state is 0.9, and the probability of detecting an "abnormal" driver state is 0.1. When the actual driver state is "abnormal," the probability of the near-infrared camera detecting a "focused" driver state is 0.3, and the probability of detecting an "abnormal" driver state is 0.7.

[0066] The seat sensors detect seat pressure or seat angle. When the actual driver is "focused," the probability of the seat sensors detecting "focused" is 0.75, and the probability of detecting "abnormal" is 0.25. When the actual driver is "abnormal," the probability of the seat sensors detecting "focused" is 0.15, and the probability of detecting "abnormal" is 0.85.

[0067] The steering wheel sensor senses information such as steering wheel pressure. When the actual driver is "focused," the probability of the steering wheel sensor detecting "focused" is 0.95, and the probability of detecting "abnormal" is 0.05. When the actual driver is "abnormal," the probability of the steering wheel sensor detecting "focused" is 0.2, and the probability of detecting "abnormal" is 0.8.

[0068] In some embodiments, the state detection capability factor can be dynamically updated based on sensor data collected by the vehicle. Based on the principle of "data not leaving the vehicle," and to ensure that the conditional probability does not drift excessively relative to a baseline, boundaries (e.g., upper and lower limits) can be set for the conditional probability, and limited updates can be made locally in the vehicle using data collected from mass-produced vehicles. For example, the state detection capability factor can be updated online using an update formula based on a learning rate of 0.01. ,in, This represents the updated state detection capability factor. This represents the historical state detection capability factor (the previous state detection capability factor), and This represents the on-site condition detection capability factor (current condition detection capability factor).

[0069] By setting a corresponding state detection capability factor for the driver state observation results of each modality, the difference in state detection capability of the driver state sensor for driver state observation results of different modalities can be taken into account when the driver state sensor perceives the driver state observation results. Based on the difference in state detection capability, adjustments can be made to the driver state observation results of that modality, thereby improving the perception accuracy of driver state observation results of different modalities.

[0070] In some embodiments, the vehicle driving scenario may also be considered when determining the driver state perception result. The vehicle driving scenario is characterized as a driving scenario factor that indicates the probability of occurrence of various driver states under the vehicle driving scenario, and is used to reflect the differences in the probability of state occurrence under different vehicle driving scenarios.

[0071] Considering general experience and statistical data, the likelihood of a driver being in a "focused" or "abnormal" state varies depending on the driving scenario. For example, a driver is slightly more likely to be in an "abnormal" state on a highway than on an urban road. Based on this consideration, different driving scenario factors are assigned to different states under different driving scenarios.

[0072] Driving scenario factors can be represented using prior probabilities determined based on historical data. Below are some examples of driving scenario factors.

[0073] Urban road scene: ,

[0074] High-speed scenarios: .

[0075] In some embodiments, the driving scenario factor can be dynamically adjusted based on the driver's driving time.

[0076] For each of the at least two modalities of driver state observation results, the driver state perception result corresponding to that modality can be determined based on the vehicle driving scenario, the driver state observation result of that modality, and the software recognition capability of the corresponding driver state sensor.

[0077] In some embodiments, a relationship table between driver state observation results and driver state perception results can be constructed based on the vehicle driving scenario and the software recognition capability of the driver state sensor. For example, the relationship table can be constructed based on the driving scenario factor corresponding to the vehicle driving scenario and the state detection capability factor corresponding to the software recognition capability of the driver state sensor. Then, for each mode of driver state observation results among at least two modes of driver state observation results, the driver state perception result corresponding to that mode of driver state observation result is obtained from the relationship table.

[0078] Figure 4 An example flowchart is shown of a process 400 for constructing a driver state observation result-driver state perception result relationship table according to an embodiment of the present disclosure.

[0079] like Figure 4 As shown in section 410, the state detection capability factor corresponding to the driver state observation results for various modalities is determined. For example, statistics can be performed based on historical data, and the driver state observation results for various modalities can be determined based on the statistical results. The corresponding state detection capability factor ,in, This indicates determining the driver state observation results given the driver state S. The probability. If there are three driver states. , and Then the state detection capability factor It can be represented as a vector .

[0080] At 420, determine the driving scenario factors. Driving scenario factors This can be determined based on empirical data or historical statistical data. If there are three driver states... , and Then driving scenario factor It can be represented as a vector .

[0081] In 430, the driver state perception results corresponding to the driver state observation results of various modalities are determined based on the driving scenario factors and the state detection capability factors corresponding to the driver state observation results of various modalities.

[0082] For example, driver state observations for a particular mode can be determined based on formula (1). The corresponding driver state perception results :

[0083] (1),

[0084] in, Represents the set of all driver states. Represents a set One of the driver states, This represents the driver's state observation results for one mode. Corresponding driver status The results of perception Indicates the state of a given driver The driver's status observation results were determined in time. The probability, This indicates that a driving state occurs under a given driving scenario. The probability of.

[0085] At 440, a driver state observation-driver state perception result relationship table is constructed based on the driver state observation results and the corresponding driver state perception results for each modality.

[0086] When there are n possible driver states, the driver state observation results for one mode are... The n probability results can be determined according to formula (1). These n probability results constitute a modality of driver state observation results. The corresponding driver state perception results.

[0087] Figure 5 An example schematic diagram of a driver state observation-driver state perception result relationship table according to an embodiment of the present disclosure is shown.

[0088] exist Figure 5 The example lists the relationship between driver state observation results and driver state perception results for two vehicle driving scenarios. Driver states include three states (state 1, state 2, state 3), and observation results include three observation results (observation result 1, observation result 2, observation result 3). Each observation result corresponds to three probability values ​​for the driver state perception result, representing the probability of state 1, state 2, and state 3 occurring under that observation result. The driver state result for each observation result may differ in different vehicle driving scenarios.

[0089] In some embodiments, the driving state observations for each modality can provide the probability that the driver is in each of the various states. For example, suppose there are three driver states. , and The driving state observation results for each modality can include the driver state as the driver state. probability The driver status is the driver status. probability And the driver status is the driver status. probability .

[0090] In this scenario, the driving state observations for each modality can be corrected using software recognition capabilities based on driver state sensors to determine the corresponding driver state perception result. For example, for each modality's driver observations, the state detection capability factor corresponding to that modality's driver observations can be used for modification, thereby determining the corresponding driver state perception result. In some embodiments, for each modality's driver observations, a driving scenario factor and the corresponding state detection capability factor can be used for modification, thereby determining the corresponding driver state perception result.

[0091] In some embodiments, driver state observations from various modalities may also be used. The corresponding state detection capability factor Construct a relationship table between driver state observation results and driver state perception results. For example, include driver state observation results from various modalities. The corresponding state detection capability factor As the driver's state observation results of this mode The corresponding driver state perception results are used to construct a relationship table between driver state observation results and driver state perception results.

[0092] Back Figure 2 After determining the driver state perception results corresponding to at least two modalities of driver observation results as described above, in step 230, the driver state perception results corresponding to at least two modalities of driver state observation results are subjected to perception fusion to determine the driver state.

[0093] In some embodiments, when performing perception fusion, the driver state perception results corresponding to the driver state observation results of at least two modalities can be summed and averaged to determine the probability corresponding to each driver state, and then the driver state with the highest probability can be determined as the perceived driver state.

[0094] In some embodiments, the driver state observations for each modality may have corresponding fusion weights. In this case, the driver state is determined by perceptual fusion using the driver state perceptions corresponding to at least two modalities of driving state observations and the fusion weights corresponding to the driver state observations for each modality.

[0095] In some embodiments, the fusion weights corresponding to the driver state observation results for each modality may include at least one of the following fusion weights: a fusion weight reflecting the hardware sensing capability of the driver state sensor corresponding to the driver state observation results for that modality under ideal conditions (hereinafter referred to as the "basic perception capability factor"); a fusion weight reflecting the driver's modal perception adaptation to the driver state observation results for that modality (hereinafter referred to as the "driver personalization factor"); a fusion weight reflecting the data reliability of the driver state sensing data sensed by the driver state sensor corresponding to the driver state observation results for that modality (hereinafter referred to as the "observation result quality factor"); and a fusion weight reflecting the influence of the driving environment on the perception capability of the driver state sensor corresponding to the driver state observation results for that modality (hereinafter referred to as the "environment factor").

[0096] The basic perception capability factor corresponding to the driver state observation results for each modality can be determined based on the hardware configuration of the driver state sensor corresponding to that modality. For example, the hardware configuration of the driver state sensor and the modality type of the driver state observation results can be provided to a pre-trained basic perception capability factor determination model to determine the basic perception capability factor corresponding to the driver state observation results for that modality. Alternatively, the basic perception capability factor can be determined based on historical statistical data of the driver state sensor's observation results for that modality. The value of the basic perception capability factor is a real number between 0 and 1. The larger the value of the basic perception capability factor, the stronger the hardware perception capability of the driver state sensor corresponding to the driver state observation results for that modality, and the more reliable the perceived driver perception results.

[0097] Driver personalization factors reflect a driver's adaptation to modal perception. Adaptation varies across different perception modalities. Driver personalization factors can be dynamically learned from historical driver data, statistically analyzed based on long-term behavioral patterns, or actively adjusted by the user. For example, if historical driver data reveals a low resting heart rate, the driver personalization factor from the heart rate sensor can be adjusted accordingly. Increase it to 1.2. Based on historical data analysis showing that the driver frequently adjusts the seat, the driver personalization factor of the seat posture sensor will be adjusted. Set to 0.8. Since the elderly driver selected "High Sensitivity Mode," the driver personalization factor for all sensors was adjusted. Adjusted to 1.5.

[0098] Driver personalization factors can reflect a driver's individual information (daily baseline vital signs data, typical sitting posture, etc.). By assigning driver personalization factors, errors in driver state perception caused by individual driver characteristics and / or differences in driving style can be reduced.

[0099] The observation result quality factor corresponding to the driver observation results for each modality reflects the reliability of the driver state sensor's driver observation results for that modality. The observation result quality factor Confidence is a real number between 0 and 1. The larger the Confidence value, the more reliable the observation result. The observation result quality factor Confidence can be calculated based on Confidence = 1 - information entropy (O_i). For example, if the heart rate signal is stable, the observation result quality factor Confidence = 0.9. The environmental factor EnvFactor corresponding to the driver observation results for each modality reflects the influence of the vehicle's external environment on the driver state sensor's perception ability when sensing the driver observation results for that modality. The environmental factor EnvFactor is a real number between 0 and 1. The larger the environmental factor EnvFactor value, the weaker the interference caused by the vehicle's external environment. When the environmental factor EnvFactor is 1, the vehicle's external environment does not interfere with the driver's state perception. For example, the environmental factor EnvFactor for the camera in low light is 0.7. In bumpy scenarios, the environmental factor EnvFactor for steering wheel capacitance detection is 0.6.

[0100] In some embodiments, perceptual fusion can be performed according to formula (2):

[0101] (2),

[0102] in, This represents the fusion weight corresponding to the driver state observation result of the i-th mode. This represents the driver state perception result of the i-th mode sensed by the driver state sensor, and... It is the result of perception fusion, including the probability of occurrence of various driver states after fusion.

[0103] After the fusion is completed as described above, the driver state with the highest fusion probability is taken as the perceived driver state.

[0104] After performing perception fusion according to formula (2), the perceived driver state can be determined according to formula (3). :

[0105] (3).

[0106] In some embodiments, when the fusion weights include at least two fusion weights, weight fusion can also be performed on the fusion weights. For example, in some examples, normalized weight fusion can be performed on the fusion weights.

[0107] For example, multiple fusion weights can be fused according to formula (4):

[0108] (4),

[0109] in, This represents the basic perception capability factor corresponding to the driver state observation result of the i-th modality. This represents the observation quality factor corresponding to the driver state observation result of the i-th mode. This represents the environmental factor corresponding to the driver's state observation result of the i-th mode. This represents the driver personalization factor corresponding to the driver state observation result of the i-th mode. Indicates the first The basic perception ability factor corresponding to the driver state observation results of each modality. Indicates the first The quality factor of the observation results corresponding to the driver state observation results of each modality. Indicates the first The environmental factors corresponding to the driver state observation results of each modality. Indicates the first The driver personalization factor corresponding to the driver state observation results of each modality, and n represents the total number of modalities.

[0110] Accordingly, when performing perception fusion, the driver's state can be determined by using the driver state perception results corresponding to at least two modalities of driving state observation results and the fusion weights after weighted fusion corresponding to the driver state observation results of each modality.

[0111] Figure 6 An example schematic diagram of a driver state perception process according to an embodiment of the present disclosure is shown.

[0112] like Figure 6As shown, after obtaining driver state observation results from at least two modalities (e.g., driver state observation results based on driver posture, driver state observation results based on seat position, and driver state observation results based on physiological characteristics), driving scene factors and state detection capability factors corresponding to the driver state observation results of each modality are obtained. Then, based on the driving scene factors, the state detection capability factors corresponding to the driver state observation results of each modality, and the driver state observation results of various modalities, the driver state perception results corresponding to the driver state observation results of each modality are determined. For example, a driver state observation result-driver state perception result relationship table can be constructed based on the driving scene factors and the state detection capability factors corresponding to the driver state observation results of each modality, and the driver state perception results corresponding to the driver state observation results of each modality can be found from the driver state observation result-driver state perception result relationship table.

[0113] After determining the driver state perception results corresponding to the driver state observation results of each modality, the basic perception capability factor (hardware perception fusion weight), environmental factor (environmental fusion weight), observation result quality factor (data quality fusion weight), and driver personalization factor (driver personalization fusion weight) corresponding to the driver state observation results of each modality are obtained. The basic perception capability factor, environmental factor, observation result quality factor, and driver personalization factor are then used to perform multimodal fusion on the driver state perception results corresponding to the driver state observation results of each modality, thereby determining the driver state.

[0114] By utilizing the aforementioned driver state perception scheme, the software recognition capabilities of the driver state sensor are adjusted based on the driver state observation results for each modality. Subsequently, the perception fusion of driver state perception results from multiple modalities is performed. This allows the differences in the software recognition capabilities of the driver state sensor for driver state observation results of different modalities to be considered when the driver state sensor perceives the driver state observation results. This improves the perception accuracy of driver state observation results of different modalities, thereby enhancing the accuracy of driver state determination.

[0115] By utilizing the aforementioned driver state perception scheme and further limiting the software recognition capability to the state detection capability factor of the driver state sensor, the differences in the state detection capability of the driver state sensor for different modalities can be considered when the driver state sensor perceives the driver state observation results, thereby improving the perception accuracy of driver state observation results for different modalities.

[0116] By utilizing the aforementioned driver state perception scheme, a relationship table between driver state observation results and driver state perception results is constructed based on the software recognition capabilities of driver state sensors and vehicle driving scenarios. The driver state perception results corresponding to the driver state observation results are determined by looking up the table. Then, the driver state perception results of multiple modalities are fused to further improve the accuracy of driver state determination.

[0117] By using the above-mentioned driver state determination scheme, and by limiting the driver state observation results of at least two modalities to have perceptual synergy in driver state perception, the fused multi-modal driver state perception results can compensate for each other's deficiencies or defects in driver state perception, thereby further improving the accuracy of driver state determination.

[0118] Figure 7 An example block diagram of a device 700 for sensing a driver's state (hereinafter referred to as a "driver state sensing device") according to an embodiment of the present disclosure is shown. Figure 7 As shown, the driver state perception device 700 includes a state observation result acquisition unit 710, a state perception result determination unit 720, and a state perception result fusion unit 730.

[0119] The state observation result acquisition unit 710 is configured to acquire driver state observation results in at least two modalities perceived by the driver state sensor. The operation of the state perception result acquisition unit 710 can be referenced above. Figure 2 The operation described in 210.

[0120] The state perception result determination unit 720 is configured to determine the driver state perception result corresponding to each of at least two modalities of driver state observation results, based on the driver state observation result of that modality and the software recognition capability of the corresponding driver state sensor. The determined driver state perception result is used to indicate the probability of occurrence of various driver states under the driver state observation results of that modality. The operation of the state perception result determination unit 720 can be referred to the above reference. Figure 2 The operation described in 220.

[0121] The state perception result fusion unit 730 is configured to perform perception fusion on driver state perception results from various modalities to determine the driver's state. The operation of the state perception result fusion unit 730 can be referenced above. Figure 2 The operation described in 230.

[0122] In some embodiments, the state perception result determination unit 720 may be configured to obtain the driver state perception result corresponding to the driver state observation result of that mode from the driver state observation result-driver state perception result relationship table for each of the driver state observation results of at least two modes.

[0123] In some embodiments, the driver state observation result-driver state perception result relationship table can be constructed based on the vehicle driving scenario and the software recognition capability of the driver state sensor.

[0124] Figure 8 An example block diagram of a relation table construction unit 800 according to an embodiment of the present disclosure is shown. Figure 8 As shown, the relation table construction unit 800 includes a state detection capability factor determination module 810, a driving scenario factor determination module 820, a state perception result determination module 830, and a relation table construction module 840.

[0125] The state detection capability factor determination module 810 is configured to determine the state detection capability factor corresponding to the driver state observation results for various modalities. For example, the state detection capability factor determination module 810 can perform statistical analysis based on historical data and determine the state detection capability factor corresponding to the driver state observation results for various modalities based on the statistical results. The operation of the state detection capability factor determination module 810 can be referred to the above. Figure 4 The operation described in 410.

[0126] The driving scenario factor determination module 820 is configured to determine driving scenario factors. For example, the driving scenario factor determination module 820 can determine driving scenario factors based on experience data or historical statistical data. The operation of the driving scenario factor determination module 820 can be referenced above. Figure 4 The operation described in 420.

[0127] The state perception result determination module 830 is configured to determine the driver state perception result corresponding to the driver state observation results of various modalities based on driving scenario factors and state detection capability factors corresponding to the driver state observation results of various modalities. The operation of the state perception result determination module 830 can be referenced above. Figure 4 The operation described in 430.

[0128] The relation table construction module 840 is configured to construct a driver state observation-driver state perception result relation table based on the driver state observation results and corresponding driver state perception results for each modality. The operation of the relation table construction module 840 can be referenced above. Figure 4 The operation described in 440.

[0129] In some embodiments, the driver state observation result-driver state perception result relationship table can be constructed based on the software recognition capability of the driver state sensor. In this case, the relationship table construction unit 800 may not include the driving scenario factor determination module 820.

[0130] In some embodiments, the state perception result determination unit 720 can correct the driving state observation results for each modality based on the software recognition capability of the driver state sensor, thereby determining the driver state perception result corresponding to the driver state observation results for that modality. For example, for each modality of driver observation results, the state perception result determination unit 720 can modify the state detection capability factor corresponding to the driver observation results for that modality, thereby determining the driver state perception result corresponding to the driver state observation results for that modality. In some embodiments, for each modality of driver observation results, the state perception result determination unit 720 can modify the driving scenario factor and the state detection capability factor corresponding to the driver observation results for that modality, thereby determining the driver state perception result corresponding to the driver state observation results for that modality.

[0131] In some embodiments, when performing perception fusion, the state perception result fusion unit 730 can sum and average the driver state perception results corresponding to at least two modal driver state observation results to determine the probability corresponding to each driver state, and then determine the driver state with the highest probability as the perceived driver state.

[0132] In some embodiments, the driver state observation results for each modality may have corresponding fusion weights. In this case, the state perception result fusion unit 730 can use the driver state perception results corresponding to the driving state observation results of at least two modalities and the fusion weights corresponding to the driver state observation results of each modality to perform perception fusion to determine the driver state.

[0133] In some embodiments, when the fusion weights include at least two fusion weights, the state perception result fusion unit 730 can also perform weight fusion on the fusion weights. For example, in some examples, the state perception result fusion unit 730 can perform normalized weight fusion on the fusion weights. Then, the state perception result fusion unit 730 can use the driver state perception results corresponding to the driving state observation results of at least two modalities and the fusion weights after weight fusion corresponding to the driver state observation results of each modality to perform perception fusion to determine the driver state.

[0134] As per the above reference Figures 1 to 8This specification describes a method, a driver state sensing device, and a driver state sensing system for sensing a driver's state according to embodiments thereof. The aforementioned driver state sensing device can be implemented in hardware, software, or a combination of both.

[0135] Figure 9 An example schematic diagram of a driver perception device 900 implemented using a computer system according to an embodiment of the present disclosure is shown. Figure 9 As shown, the driver perception device 900 may include at least one processor 910, a memory (e.g., non-volatile memory) 920, a RAM 930, and a communication interface 940, and the at least one processor 910, memory 920, RAM 930, and communication interface 940 are connected together via a bus 960. The at least one processor 910 executes at least one computer-readable instruction (i.e., the elements implemented in software above) stored or encoded in the memory.

[0136] In one embodiment, computer-executable instructions are stored in a memory, which, when executed, cause at least one processor 910 to: acquire driver state observations in at least two modalities perceived by a driver state sensor; for each of the at least two modalities of driver state observations, determine a driver state perception result corresponding to that modality of driver state observation based on the driver state observation result of that modality and the software recognition capability of the corresponding driver state sensor, the driver state perception result indicating the probability of occurrence of various driver states under that modality of driver state observation; and perform perceptual fusion on the driver state perception results corresponding to the at least two modalities of driver state observations to determine the driver state.

[0137] It should be understood that the computer-executable instructions stored in memory, when executed, cause at least one processor 910 to perform the above-described combinations in the various embodiments of this specification. Figures 1-8 The description includes various operations and functions.

[0138] According to one embodiment, a program product, such as a machine-readable medium (e.g., a non-transitory machine-readable medium), is provided. The machine-readable medium may have instructions (i.e., the elements implemented in software as described above), which, when executed by a machine, cause the machine to perform the above-described combinations of the various embodiments of this specification. Figures 1-8 The various operations and functions described. Specifically, a system or apparatus equipped with a readable storage medium storing software program code that implements the functions of any of the embodiments described above, and enabling the computer or processor of the system or apparatus to read and execute the instructions stored in the readable storage medium.

[0139] In this case, the program code read from the readable medium itself can perform the functions of any of the above embodiments, and therefore the machine-readable code and the readable storage medium storing the machine-readable code constitute part of this disclosure.

[0140] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer or the cloud via a communication network.

[0141] According to one embodiment, a computer program product is provided, the computer program product including a computer program, which, when executed by a processor, causes the processor to perform the above-described combinations of the various embodiments of this specification. Figures 1-8 The description includes various operations and functions.

[0142] In addition to the content described herein, various modifications may be made to the disclosed embodiments and implementations without departing from the scope of this disclosure. Therefore, the descriptions and examples herein should be interpreted illustratively rather than limitingly. The scope of this disclosure should be measured solely by reference to the claims.

Claims

1. A method for sensing a driver's state, comprising: Acquire driver state observations in at least two modalities perceived by driver state sensors; For each of the at least two modalities of driver state observation results, based on the driver state observation results of that modality and the software recognition capability of the corresponding driver state sensor, a driver state perception result corresponding to that modality of driver state observation results is determined. The driver state perception result is used to indicate the probability of occurrence of various driver states under the driver state observation results of that modality. The software recognition capability of the driver state sensor corresponding to each modality of driver state observation results is characterized as a state detection capability factor used to indicate the perception probability of various driver state observation results given by the driver state sensor under the known real driver state. as well as The driver's state is determined by perceptual fusion of the driver state perception results corresponding to the driver state observation results of the at least two modalities.

2. The method as described in claim 1, wherein, For each of the at least two modalities of driver state observation results, based on the driver state observation results of that modality and the software recognition capability of the corresponding driver state sensor, the driver state perception result corresponding to that modality of driver state observation results is determined, including: For each of the at least two modal driver state observation results, based on the vehicle driving scenario and the software recognition capability of the driver state observation results and the corresponding driver state sensor, the driver state perception result corresponding to the driver state observation results of that modality is determined.

3. The method as described in claim 2, wherein, For each of the at least two modalities of driver state observation results, based on the vehicle driving scenario and the software recognition capability of the corresponding driver state sensor, the driver state perception result corresponding to that modality of driver state observation result is determined, including: For each of the at least two modalities of driver state observation results, the driver state perception result corresponding to that modality of driver state observation result is obtained from the driver state observation result-driver state perception result relationship table, which is constructed based on the vehicle driving scenario and the software recognition capability of the driver state sensor.

4. The method of claim 3, wherein, The software recognition capability of the driver state sensor corresponding to the driver state observation results of each modality includes the algorithm processing capability of the driver state sensor when sensing the driver state observation results of that modality.

5. The method of claim 4, wherein, Vehicle driving scenarios are characterized as driving scenario factors that indicate the probability of various driver states occurring within a given driving scenario. The driver state observation result-driver state perception result relationship table is constructed based on the following construction process: For each mode of driver state observation results, the probability of occurrence of various driver states under the driver state observation results of that mode is determined based on the state detection factor and driving scenario factor corresponding to the driver state observation results of that mode, and is used as the driver state perception result corresponding to the driver state observation results of that mode. as well as The driver state observation result-driver state perception result relationship table is constructed based on the driver state observation results and the corresponding driver state perception results for each modality.

6. The method as described in any one of claims 1 to 5, wherein, Determining the driver's state by performing perceptual fusion on the driver state perception results corresponding to the driver state observation results of at least two modalities includes: The driver's state is determined by perceptual fusion using the driver state perception results corresponding to the driving state observation results of at least two modalities and the fusion weights corresponding to the driver state observation results of each modality.

7. The method of claim 6, wherein, The fusion weights corresponding to the driver state observations for each modality include at least one of the following fusion weights: Fusion weights are used to reflect the hardware sensing capability of the driver state sensor corresponding to the driver state observation result of this mode under ideal conditions. Fusion weights used to reflect the fit of modal perception to driver state observations of this modality; Fusion weights are used to reflect the data reliability of driver state sensing data sensed by driver state sensors corresponding to driver state observation results of this mode; as well as Fusion weights are used to reflect the influence of the driving environment on the perception capability of the driver state sensor corresponding to the driver state observation results of this mode.

8. The method of claim 7, wherein, The fusion weights include at least two fusion weights, and the method further includes: The fusion weights are then fused, and Determining the driver's state using perception fusion based on the driver state perception results corresponding to the driving state observation results of at least two modalities and the fusion weights corresponding to the driver state observation results of each modality includes: The driver's state is determined by perceptual fusion using the driver state perception results corresponding to the driving state observation results of at least two modalities and the fusion weights corresponding to the driver state observation results of each modality after weighted fusion.

9. The method of claim 8, wherein, The weight fusion of the fusion weights includes: The fusion weights are then normalized and fused.

10. The method of claim 1, wherein, The driver state observation results of at least two modalities have perceptual synergy in driver state perception.

11. The method of claim 10, wherein, The driver state sensors are selected from a given set of driver state sensors based on at least one of sensor cost and engineering feasibility, as well as the perceptual synergy between the driver state observations sensed by the driver state sensors and the driver state perception.

12. A device for sensing the state of a driver, comprising: The state observation result acquisition unit is configured to acquire driver state observation results in at least two modalities perceived by the driver state sensor; The state perception result determination unit is configured to determine the driver state perception result corresponding to the driver state observation result of each of the at least two modes of driver state observation results, based on the driver state observation result of that mode and the software recognition capability of the corresponding driver state sensor. The driver state perception result is used to indicate the probability of occurrence of various driver states under the driver state observation result of that mode. The software recognition capability of the driver state sensor corresponding to the driver state observation result of each mode is characterized as a state detection capability factor used to indicate the perception probability of various driver state observation results given by the driver state sensor under the known real driver state. as well as The state perception result fusion unit is configured to perform perception fusion on the driver state perception results corresponding to the driver state observation results of the at least two modalities to determine the driver state.

13. A system for sensing driver status, comprising: Driver status sensor; And the device for sensing the driver's state as described in claim 12.

14. A vehicle comprising: The system for sensing driver status as described in claim 13.

15. A device for sensing the state of a driver, comprising: At least one processor; Memory coupled to the at least one processor; as well as A computer program stored in the memory, which is executed by the at least one processor to implement the method for sensing the driver's state as described in any one of claims 1 to 11.

16. A computer-readable storage medium storing executable instructions that, when executed, cause a processor to perform the method for sensing a driver's state as described in any one of claims 1 to 11.

17. A computer program product comprising a computer program that, when executed by a processor, performs the method for sensing a driver's state as described in any one of claims 1 to 11.

Images