Human body function state detector

Abstract

According to one aspect of the present disclosure, there is provided a computer-implemented method for detecting a human functional state, the method comprising: acquiring one or more signals from one or more sensors, where the signals are representative of a plurality of physiological parameters of the user; determining a plurality of conditions associated with the user based on the value of the physiological parameter; determining a functional state of the user based on the condition; and generating an output signal to indicate a functional state of the user.

Description

[0001] Cross - reference to related applications

[0002] This application claims the priority of Russian Application No. 2022130395, entitled "HUMAN FUNCTIONAL STATE DETECTOR", filed on November 23, 2022. The entire content of the application listed above is hereby incorporated by reference for all purposes. Technical field

[0003] The present disclosure relates to methods and systems for detecting the functional state of a human body. Background art

[0004] Identifying a person's mental and physical state is an important aspect of the development of a human - machine interface (HMI). The ability to reliably identify a person's state without affecting human behavior can be used to solve various industrial problems. The scope of application can range from monitoring drivers and workers to improving the interaction in computer games and social media. In such applications, knowledge of the human state can be used to mitigate risks and increase efficiency.

[0005] There are many ways to identify a human mental state. For example, it is well - known to monitor a driver in a car to detect whether he / she is asleep. For this purpose, a camera arranged on the dashboard (or elsewhere) records an image of the driver's face. The image is processed to determine physiological parameters, such as the eye closure or blink frequency. Based on these parameters, it can be determined whether the driver is in a potentially dangerous drowsy state, which can lead to him / her falling asleep and causing an accident.

[0006] It is well - known to use machine learning and / or artificial neural networks in advertising and marketing to identify human emotions from facial video images, for example, to evaluate the effectiveness of a given advertisement. Further, it is known to use wearable devices (smartwatches, fitness trackers) with a photoplethysmograph (green LED and photoresistor of the same wavelength) to record the heart rate and analyze the recorded sequence to detect the stress level of the accessory wearer.

[0007] These systems and other state-of-the-art systems are designed to match specific tasks. Specifically, these systems identify the presence or degree of a single condition (such as drowsiness or stress) based on one or more biological signals. In the broadest sense, biological signals describe or represent the activities, states, or functions of biological objects. For example, facial images can be used to identify the degree of human eye closure. In this example, the biological signal can be a series of digital pictures of a human face, and this series of digital pictures can be processed to derive the degree of eye closure. Another example is the output of a photodetector of a photoplethysmograph, which is proportional to the intensity of a green light beam reflected from the skin, and the intensity of the green light beam is in turn proportional to the oxygen concentration in the blood flowing through the blood vessels in the skin. Another example is that the micro-displacement of the human back reflects the breathing pattern.

[0008] To identify the human body state, the biological signal is input into a state detector. The state detector is a set of mathematical operations and signal transformations that convert the input signal or a part of it into a detector output. The state detector output is a digital or logical value (1 / 0, yes / no, true / false) associated with a specific state or state level. For example, for a drowsiness detector, 0 indicates "not drowsy" (awake), and 1 indicates "drowsy" (asleep). If the digital value is between 0 and 1, then this value can be associated with a drowsiness level. If the detector output is 0.1 at one time point (or for one person) and 0.2 at another time point (or for another person), then this indicates that a more drowsy state is observed at the second time point (or for another person).

[0009] Figure 1 A typical process performed by a system including a single-component state detector as described above is shown. The system can be extended to include two or more single-component state detectors, each dedicated to detecting different components of the human body state, such as stress and drowsiness. However, the overall state of a person at each time point includes or reflects many (all) aspects of the person's behavior and functions. While it makes sense to methodologically decompose the overall state into different components to simplify the research, it may result in results that do not represent the whole picture. Additionally, biological signal sensors will always have some inevitable noise levels that affect the accuracy of state detection. Depending on the task, the typical accuracy rate is between 70% and 90%. This means that prior art systems may make false detections of the human body state and may not be reliable enough.

[0010] In the automotive field, the detection of biological signals is typically based on sensors that are remote from the driver. Signals from such remote sensors have a higher noise level than signals from contact sensors. Additionally, the automotive environment itself is noisy. There may be mechanical or environmental noise. Image sensors may be exposed to changing light conditions. Therefore, detecting the functional state of the human body with sufficient accuracy poses a challenge, particularly in the automotive field.

[0011] The present disclosure aims to improve the accuracy and reliability of detecting the functional state of the human body. Summary of the Invention

[0012] The present disclosure provides a method and system for simultaneously and consistently identifying multiple human state components. Taking into account the interdependencies between the multiple individual components, the accuracy and reliability of detecting the functional state of the human body are thereby improved. For example, in the context of intervening in vehicle operation when a critical state of the driver is detected, this enables a more confident response to the detected functional state of the human body.

[0013] The present disclosure recognizes that physiology places constraints on the possible combinations of certain components of the human state. For example, it is unlikely for a person to simultaneously experience a high stress level and a high drowsiness level. On the other hand, happiness can occur simultaneously with alertness. Therefore, detecting one component may increase or decrease the likelihood of the presence of another component. By defining such interrelationships, the detection of each component can be improved. This in turn enhances the accuracy of determining the overall functional state of the human body.

[0014] To achieve these and other desired effects, according to one aspect of the present disclosure, there is provided a computer-implemented method for detecting the functional state of a user, the method comprising: obtaining one or more signals from one or more sensors, wherein the signals represent multiple physiological parameters of the user; determining multiple conditions associated with the user based on the values of the physiological parameters; determining the functional state of the user based on a combination of the conditions; and generating an output signal to indicate the functional state of the user.

[0015] The method is particularly suitable for implementation in a vehicle. However, the present disclosure is not limited thereto, and the method can also be implemented in other systems in which the functional state of a user is to be detected.

[0016] The signal can be a video signal acquired by one or more cameras arranged in the vehicle and pointing at the user. For example, the video signal can be acquired from light reflected from the user's skin onto the image sensor of the camera. The change in light intensity corresponds to the change in blood volume, from which the heartbeat parameters can be derived. Thus, in one embodiment, the one or more parameters include one or more heartbeat parameters, particularly heart rate, heart rate variability, and / or heart inter-beat interval.

[0017] Alternatively or additionally, one or more cameras can be arranged to capture images of one or both eyes of the user. Known feature detection methods can be used to identify the user's eyes and parameters related to the eyes, such as the position of the eyes or eyelids and their movement. Thus, in one embodiment, the one or more physiological parameters include one or more eye parameters, particularly eye movement, eyelid movement, eye position, eyelid position, and / or eye gaze.

[0018] The heartbeat features and eye parameters can be advantageously combined to determine different components or aspects of the user's state. The user state can include different physiological or psychological components, or a combination thereof. These different components of the user state are also referred to herein as "conditions", while the user state is also referred to as the user's "overall" state. The detected individual components can be combined to derive the user state. For example, an increased heart rate and rapid eye movement (saccades) may both point to the same state, i.e., a high stress level. Thus, combining the detection of an increased heart rate and saccades can more reliably determine the user's overall state.

[0019] The state of the user indicates the user's adaptability to control, particularly to driving the vehicle. In one embodiment, the user state indicates one of the following states: fit to drive, conditionally fit to drive, and unfit to drive. If it is determined that the user state is "fit to drive", no response action is taken. If it is determined that the user state is "conditionally fit to drive", appropriate actions can be taken. For example, a message recommending slowing down the vehicle speed or taking a break can be displayed. If it is determined that the user state is "unfit to drive", more aggressive actions can be taken. For example, a loud reminder signal can be generated, or the vehicle speed can be slowed down. However, the present disclosure is not limited to these states and actions. For example, "conditionally fit to drive" may have different levels, each associated with a corresponding action. In addition, the present disclosure aims to cover applications in vehicles other than automobiles, such as airplanes.

[0020] The state of a user may be associated with different components or conditions. For example, the user state can be associated with different levels of stress, cognitive demand, or drowsiness. Individual conditions may be related to or dependent on other individual conditions. They can complement each other or be mutually exclusive. For example, when a user faces a higher level of cognitive demand, she / he may simultaneously experience a higher level of stress. Thus, knowledge of one or more conditions can be used to more reliably detect other individual conditions. Accordingly, in one embodiment of the present disclosure, the method includes determining at least one condition in the condition based on at least one other condition in the condition. Specifically, the method may include determining at least one condition in the condition based on a combination or pattern of other conditions.

[0021] On the other hand, when a user is drowsy, a high level of stress generally does not occur. In terms of physiological parameters, when a user has half-closed eyes for a period of time, a too-high heart rate generally does not occur. Thus, the interdependence of individual conditions may be achieved in a mutually exclusive manner. Detecting such mutually exclusive conditions enables the identification of potential errors, for example, due to a noisy image signal caused by insufficient light. Thus, considering multiple conditions enables the elimination of errors and a more reliable determination of the user's overall state.

[0022] The cognitive load of a user can include a cognitive load related to controlling a vehicle. For example, if a user's eye gaze indicates that she / he is looking at the road ahead, it can be determined that the current cognitive load is related to driving. On the other hand, a user's eye gaze can indicate that she / he is looking at a mobile phone, and it can be determined that the current cognitive load is not related to driving. For example, differentiating such conditions enables the determination of different categories of the "conditionally fit to drive" state.

[0023] In one embodiment, determining the state of a user includes selecting one of a plurality of predetermined states, where each of the plurality of predetermined states is associated with a corresponding combination of conditions. For example, if there is simultaneously a "no stress" condition (e.g., indicated by a normal heart rate) and an "attention" condition (e.g., indicated by an eye gaze pointing ahead on the road), the "fit to drive" state can be determined. On the other hand, if the conditions are contradictory, then, for example, based on stored information, it can be determined that one of the conditions is more likely to occur. In this case, the more likely condition is selected to determine the user state, and the other conditions are discarded. Alternatively, the determination of the user state can be suspended until the detection of the contradictory conditions ends. Additionally, an error signal can also be generated.

[0024] In one embodiment, several conditions are considered, such as three or more conditions, and the method includes determining that some of the conditions (e.g., at least two of the conditions) are mutually exclusive, and discarding one or all of the mutually exclusive conditions when determining the state of the user. This embodiment can establish a desired confidence level by requiring a minimum number of consistent conditions as a prerequisite for determining the overall state of the user. However, the present disclosure is not limited to a specific number of conditions. Additionally, the confidence level can be adjusted according to the user state to be determined. For example, the "fit to drive" state may require more consistent conditions than the "conditionally fit to drive" state. In another example, the "not fit to drive" state may require the absence of any combination of inconsistent conditions.

[0025] In one embodiment, generating an output signal indicative of the state of the user includes one or more of the following: alerting the user of the vehicle, controlling the vehicle, particularly adjusting the user interface in the vehicle or adjusting one or more vehicle control systems, and communicating the state of the user to another vehicle or an external server or control station. Thereby, the safety of operating the vehicle can be improved.

[0026] According to another aspect of the present disclosure, a system for detecting the functional state of a user is provided. The system includes: one or more sensors arranged in the vehicle to detect one or more signals, where the signals represent multiple physiological parameters of the user; and at least one processing device configured to: determine a plurality of conditions associated with the user based on the values of the physiological parameters; determine the functional state of the user based on a combination of the conditions; and generate an output signal to indicate the functional state of the user. Similar to the above method, the system enables improving the reliability and accuracy of determining the overall state of the user, such as the adaptability of the user to operate (drive) the vehicle.

[0027] The system can be configured to implement any of the above operations, functions, and advantages. Specifically, the at least one processing device can be configured to determine the state of the user by selecting one of a plurality of predetermined states, where each of the plurality of predetermined states is associated with a corresponding combination of conditions. Additionally, the system can be configured to determine the state of the user based on several (e.g., three or more) conditions, where the processing device is configured to determine that some of the conditions (e.g., at least two) are mutually exclusive, and discard one or all of the mutually exclusive conditions when determining the state of the user.

[0028] The system can be an in-vehicle system and includes one or more in-vehicle sensors to generate signals that are processed to determine the user state. The sensors can include one or more image sensors, particularly a combination of different cameras, such as RGB and infrared cameras, and / or cameras arranged at different positions of the vehicle. In this way, the problem of changing lighting conditions can be solved, and the accuracy of image processing and feature detection can be improved.

[0029] In one embodiment, the at least one processing device includes at least one processing device arranged at a remote server or control station. Thus, the method of the present disclosure can be implemented partially or entirely away from the vehicle, thereby reducing the processing load in the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The features, objects, and advantages of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which like reference numerals refer to like or similar elements.

[0031] Figure 1 A conventional method for detecting the functional state of a human body is shown;

[0032] Figure 2 A system for determining the functional state of a user of a vehicle according to an embodiment of the present disclosure is schematically shown;

[0033] Figure 3 A flowchart of a method for determining the functional state of a user of a vehicle according to an embodiment of the present disclosure is shown;

[0034] Figure 4 A flowchart of a method for determining the functional state of a user of a vehicle according to another embodiment of the present disclosure is shown;

[0035] Figure 5 A flowchart of a method for determining the functional state of a user of a vehicle according to yet another embodiment of the present disclosure is shown; and

[0036] Figure 6 A flowchart of a method for training a neural network according to another embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0037] The present disclosure provides a method and system for detecting the functional state of a human body. Multiple human state components, also referred to as conditions, are simultaneously and consistently identified. In the illustrated exemplary embodiment, multiple interrelated conditions of a user of a vehicle are detected and monitored. Example conditions are stress and fatigue. The conditions are processed to determine the functional state of the user. The functional state can indicate whether the user is fit to operate the vehicle. By considering combinations of interrelated conditions, individual conditions can be detected more accurately, and the state of the user can be determined more reliably.

[0038] Reference Figure 2 , according to an embodiment, system 100 includes a first sensor 102 and a second sensor 104. In the illustrated embodiment, sensors 102 and 104 are cameras. Cameras 102 and 104 can be mounted in a vehicle operated by user 106. The vehicle can be an automobile, but the present disclosure is not limited thereto. Cameras 102 and 104 are arranged to point at the user's face to detect light reflected from the user's face. Cameras 102 and 104 generate output signals representing the detected light. The output signals are transmitted to a processing device 108 communicatively coupled to cameras 102 and 104. The processing device 108 can be arranged in the vehicle, e.g., in the head unit of an automobile. Alternatively, the processing device 108 can be arranged at a location remote from the vehicle, e.g., at a remote control station. In another alternative, the processing device 108 can include several processing elements, some or all of which can be arranged in the vehicle or remotely. In any case, the processing device 108 is configured to process the signals received from cameras 102 and 104 to determine the condition and overall state of user 106, as described in more detail below. After processing is complete, an output signal is generated and transmitted to interface 110. Interface 110 can be arranged to transmit the output signal to another entity, e.g., to a display in the vehicle, thereby enabling the display of the determined user state or associated information. Alternatively, the output signal can be transmitted to the control unit of the vehicle, thereby enabling intervention in the operation of the vehicle.

[0039] The first camera 102 can be arranged to detect light reflected from a selected skin patch 112 on the user's face. The skin patch 112 can be located on either the user's forehead or the user's cheek. The light reflected from such a skin patch is particularly suitable for rPPG processing by the processing device 108. RPPG processing includes detecting changes in corresponding pixels in consecutive image frames, such as changes in color and / or intensity. Such changes can be caused by instantaneous changes in blood flow under the skin patch 112. This can provide information about the heartbeat parameters of user 106, including but not limited to heart rate, heart beat intensity, heart rhythm, and inter-beat interval.

[0040] The second camera 104 may be arranged to record a series of images of an eye region 114 that includes one of the user's eyes. The output signal of camera 104 is transmitted to the processing device 108. The processing device 108 processes the received signal to determine selected eye parameters such as eye movement or eyelid movement, eye gaze, and eyelid position. These eye parameters also indicate the user's condition such as fatigue, stress, and distraction.

[0041] Cameras 102 and 104 may have an RGB sensor that is capable of capturing a series of image frames using the visible spectrum at a resolution sufficient to identify facial contours and features such as the positions of the eyes, eyebrows, nose, and mouth. Each pixel may have an associated red-green-blue (RGB) value. Capturing visible light and thereby allowing RGB values to be associated with pixels allows for the selection of color components that are most suitable for rPPG and further analysis. For example, when natural light fluctuates or is insufficient, an optional light source (not shown) may be added to provide artificial light and illuminate the face of user 106, thereby enabling more reliable feature detection.

[0042] In another embodiment, one or both of cameras 102 and 104 are infrared cameras. In this embodiment, the system 100 may also include an infrared light source (not shown) to illuminate the face of user 106 with infrared light. Using infrared imaging and infrared illumination can achieve more reliable feature detection at night or when there is a lack of natural light or insufficient natural light.

[0043] Cameras 102 and 104 may self-adjust. Specifically, cameras 102 and 104 may be configured to detect facial features and adjust camera parameters (such as orientation, focus, aperture, etc.) for the purpose of optimizing the detection of light from the face of user 106.

[0044] Figure 3 A method 200 according to one embodiment of the present disclosure is shown. The method may be performed by Figure 2The processing device 108 executes. In step 201, signals are obtained from multiple sensors in the vehicle (e.g., from cameras 102 and 104). Thus, the signals can be image signals representing light reflected from the skin patch 112 and the eye region 114 of the user 106. These signals contain information representing different physiological parameters of the user 106 (such as heart rate and eye parameters). In steps 202 and 203, this information is processed to determine at least a first condition and a second condition of the user 106. For example, step 202 may include extracting and processing the heart rate parameter to determine the heart rate. The heart rate may correspond to the stress level of the user. For example, an increase in heart rate indicates an increase in stress level. This corresponds to the first condition. Step 203 may include extracting and processing the eye parameters to determine the second condition of the user 106. For example, the eye parameters may correspond to the eye movement rate. A high eye movement rate may also indicate an increase in stress level. This corresponds to the second condition. Thus, in this example, the first condition and the second condition correspond to each other ("increase in stress level"). In step 204, the first condition and the second condition are processed to determine the overall state of the user. In this example, the first condition and the second condition are the same, and it can be determined that the user state corresponds to the first condition and the second condition, that is, the state corresponding to "increase in stress level". In step 205, the state determined in step 204 can be used to generate an output signal. For example, the state corresponding to "increase in stress level" can generate an output signal for intervening in the vehicle operation. For example, the output signal can be used to activate a speed limiter to prevent the user from driving at a speed exceeding a certain limit.

[0045] Thus, the user state is determined based on at least two conditions. If the conditions are consistent with each other, the user state can be determined with greater confidence. This in turn enables actions that might not otherwise be recommended, such as intervening in the operation of the vehicle. Additionally, in one embodiment, different confidence levels can be defined for different actions. For example, intervening in the operation of the vehicle may require the presence of at least three consistent conditions. Specifically, this may require three different physiological parameters to indicate the same condition, such as "increase in stress level". These can be an overly high heart rate, overly fast eye movement, and overly fast eyelid movement.

[0046] In another example, if it is determined that the user state is a normal stress level, no output signal is generated. In another example, if the first condition and the second condition are inconsistent with each other, it can be determined that the user state is uncertain, and the output signal may be an error signal.

[0047] Figure 4 Method 300 according to another embodiment is shown. Method 300 may also be performed by Figure 2The processing device 108 executes. In this embodiment, the user state is determined based on selecting and / or combining the individual conditions of the user. In step 301, signals are obtained from multiple sensors. Step 301 can be the same as step 201 in Figure 2 . In step 302, at least three conditions are determined. Each condition can be determined based on a physiological parameter different from other conditions. For example, a first condition can be determined based on a heart rate parameter. A second condition can be determined based on an eye movement parameter. A second condition can be determined based on an eyelid movement parameter. In step 303, the first condition, the second condition, and the third condition are compared. If all three conditions are found to be consistent with each other, the method proceeds to step 304 and the user state is determined based on all three conditions. However, if one of the three conditions is found to be inconsistent with the other two conditions in step 303, then in step 304, the other two conditions are selected to determine the user state. Thus, this embodiment provides a certain degree of redundancy and increases the consistency of determining the user state over a longer period of time. In step 305, an output signal is generated based on the determined user state. Step 305 can be the same as step 205 in Figure 3 .

[0048] Method 300 is an illustrative example, and the present disclosure is not limited thereto. For example, based on different physiological parameters derived from the same or different sensor signals, there can be any number of conditions. Additionally, for some or all of the conditions, the same parameters can be used, but these parameters are derived from different signals or sensors. Further, a first number of conditions can be determined first, and then a second number of conditions are determined only if at least one of the first number of conditions is found to be inconsistent with the other conditions of the first number of conditions.

[0049] Figure 5 Method 400 according to another embodiment is shown. In this embodiment, the interdependence between conditions is considered to improve the detection of individual conditions and refine the determination of the overall user state. In step 401, signals are obtained from multiple sensors. Step 401 can be the same as step 301 in Figure 4 . In step 402, at least three conditions are determined. Each condition can be determined based on different physiological parameters, as explained in conjunction with Figure 4 . The conditions can be interrelated, as schematically shown by the overlapping ellipses of the first condition and the second condition. For example, if the second condition is detected simultaneously, the detection of the first condition may be more accurate. Similarly, if both the first condition and the second condition are detected simultaneously, the detection of the third condition may be less accurate. Thus, the interrelationship between different conditions can be used to verify the correct detection of individual conditions.

[0050] In Figure 5For simplicity, only three situations are shown. However, the present disclosure is not limited thereto. There may be multiple situations, some of which "overlap" with each other, while others are mutually exclusive. The existence of combinations of situations may increase or decrease the likelihood of certain individual situations occurring. In addition, based on these interrelationships, all situations as a whole form a pattern indicating the overall state of the user. Different such patterns may be associated with different user states.

[0051] In step 403, the knowledge of these interrelationships is used to verify the result of step 402. In the example shown, the detection of the first and second situations is confirmed, while the detection of the third situation is discarded. Thus, in the example shown, only two situations are selected, but these two situations can be used to determine the overall state of the user with greater confidence.

[0052] In step 404, the situations selected in step 403 are processed to determine the user state. In step 405, an output signal is generated based on the user state. Steps 404 and 405 can be the same as Figure 3 steps 304 and 305 in

[0053] In one embodiment, the interrelationships between situations can be used to determine the probability of correctly detecting individual situations. Each pattern composed of a set of detected situations can indicate the probability of the existence of an individual situation. For example, referring to Figure 5 , if the first and second situations are detected simultaneously, the probability of the correctness of the detection of the first situation can be set to 80%. If the third situation is also detected simultaneously, and the third situation is considered inconsistent with the first and second situations, the probability of correctness associated with the first situation may be reduced to 70%. These values represent illustrative, non-limiting examples.

[0054] In another example, again referring to Figure 5 , step 403 can be part of an iterative process. For any given combination of multiple detected situations, the situation determined to be the least likely can be discarded. The resulting new combination of situations can undergo the same determination and selection. This can be repeated until a predetermined threshold is reached. The threshold can be achieved based on the number of remaining situations, the probability of individual remaining situations, the probability of the combination of remaining situations, etc.

[0055] The selection of situations in step 403 can be implemented by a neural network. Such a neural network can be trained based on different situation patterns and their interrelationships with individual situations. In principle, the same applies to the determination of the user state in step 404. For this purpose, a neural network can be trained based on different combinations of situations and their associations with the overall user state.

[0056] More specifically, in one embodiment, machine learning (“ML”) is used to implement a detector that defines the relationship between physiological parameters or biosignals and individual components of the user's overall state. Such a detector is also referred to herein as an “ML detector” and can be implemented by a neural network trained to learn these relationships. Generally, the training data indicating such relationships can be obtained through experiments, during the normal operation of the system implementing the method, or simply based on experience.

[0057] Figure 6 An exemplary training process of the ML detector is shown. The ML detector is trained using known or hypothesized relationships between the user's physiological parameters and psychophysiological conditions. For example, a person solving a complex math problem is considered to experience a high level of cognitive load. Thus, in step 501, physiological parameters are measured while the user solves a complex problem (e.g., a math problem). In step 502, the measured values are input as training data into the ML detector. Inputting such data into the ML detector is expected to produce an output indicating a high level of cognitive load. The expected output of the ML detector corresponds to the “target value”. In step 503, the actual values output by the ML detector are measured, and in step 504, these measured values are compared with the target value. If in step 505 it is found that the difference between the target value and the measured value is greater than a predetermined threshold, the process proceeds to step 506. In step 506, a so-called error function is generated to identify a measure of the deviation between the actual behavior and the expected behavior of the ML detector for a particular instance of the physiological parameter. This error function is used to adjust the “behavior” of the ML detector to reduce the difference between the target value and the actual value. For example, this error function is used to adjust the neural network weights. The process of adjusting the weights in the neural network is generally referred to as backpropagation.

[0058] Steps 502 to 506 can be repeated until a stop condition is met. There can be different stop conditions. In one example, the stop condition is met when the difference between the target value and the actual value is below a predetermined threshold. In another example, the stop condition is met when the accuracy of the ML detector stops increasing or if the amount of increase is below a threshold. In a further example, steps 502 to 506 are repeated for a predetermined number of adjustment iterations, e.g., until a predetermined iteration "budget" available for training the ML detector is consumed. In any case, when the stop condition is met in step 505, the process proceeds to step 507. Step 507 is an optional step to test the accuracy and reliability of the ML detector using a test dataset. When using the ML detector in a "real-life" application, the test dataset should adequately represent realistically important scenarios. If the accuracy achieved during training (whether step 507 is performed or not) is considered satisfactory, the training process can be terminated (step 508). Otherwise, it can be repeated with different training data. Thereafter, the ML detector is ready to be used to identify the human functional state in real-life situations.

[0059] A more specific illustrative example of training the ML detector is described below. The training process can proceed according to the steps of Figure 6 . In this example, the ML detector implements a two-component cognitive load detector for a vehicle. The ML detector is used to detect and indicate whether a driver is experiencing cognitive load and whether the cognitive load is related to driving. Appropriate training data is generated based on the following protocol that includes several phases. The protocol can be implemented during real-life driving or in a driving simulator. During each phase, the physiological parameters (bio-signals) of the user are measured while meeting the indicated requirements. Then the measured values are used as input (training) data for the ML detector. For each phase and for each requirement, the expected output (target value) for the corresponding situation is known.

[0060] Phase 1:

[0061] Requirement: There are no complex driving-related tasks. (The term "complex" is intended to indicate tasks that are significantly beyond ordinary driving situations). There are no additional non-driving-related tasks.

[0062] Target value: The ML detector will output a value of 0 for "driving task" and a value of 0 for "non-driving task".

[0063] Phase 2:

[0064] Requirement: There are no complex driving-related tasks. There are additional non-driving-related tasks.

[0065] Target value: The ML detector will output a value of 0 for "driving task" and a value of 1 for "non-driving task".

[0066] Phase 3:

[0067] Requirement: There are complex driving-related tasks. There are no additional non-driving-related tasks.

[0068] Target value: The ML detector will output value 1 for "driving tasks" and value 0 for "non-driving tasks".

[0069] Phase 4:

[0070] Requirement: There are complex driving-related tasks. There are additional non-driving-related tasks.

[0071] Target value: The ML detector will output value 1 for "driving tasks" and value 1 for "non-driving tasks".

[0072] After training, when the ML detector is fed data representing physiological parameters obtained in real-life driving situations, it can generate an output signal that indicates the presence or absence of a high cognitive load related or unrelated to driving:

[0073] Output for driving-related tasks:

[0074] 0: Indicates that the subject is not solving any complex driving-related tasks. (However, the driver may be experiencing a low-level cognitive load associated with ordinary driving tasks).

[0075] 1: Indicates that the user is experiencing a high-level cognitive load due to complex driving-related tasks.

[0076] Output for non-driving-related tasks:

[0077] 0: The driver is not experiencing a significant cognitive load due to non-driving-related tasks.

[0078] 1: The driver is experiencing a significant cognitive load due to non-driving-related tasks.

[0079] The following is a similar example of an ML detector that implements a three-component cognitive load detector also for vehicles. In this example, the overall cognitive load level of the driver can be additionally identified. The following protocol can be used to train the three-component cognitive load detector:

[0080] Phase 1:

[0081] Requirement: There are no complex driving-related tasks. There are no additional non-driving-related tasks.

[0082] Target value: The ML detector will output value 0 for "driving tasks", value 0 for "non-driving tasks" and value 0 for "overall load".

[0083] Phase 2:

[0084] Requirement: There are no complex driving-related tasks. There are additional non-driving-related tasks.

[0085] Target values: The ML detector will output value 0 for "driving task", value 1 for "non-driving task", and value 1 for "overall load".

[0086] Phase 3:

[0087] Requirement: There are complex driving-related tasks. There are no additional non-driving-related tasks.

[0088] Target values: The ML detector will output value 1 for "driving task", value 0 for "non-driving task", and value 1 for "overall load".

[0089] Phase 4:

[0090] Requirement: There are complex driving-related tasks. There are additional non-driving-related tasks.

[0091] Target values: The ML detector will output value 1 for "driving task", value 1 for "non-driving task", and value 2 for "overall load".

[0092] After training, when the ML detector is fed data representing physiological parameters obtained in real-life driving situations, it can generate an output signal that indicates the presence or absence of a high cognitive load related or unrelated to driving. These outputs are the same as those of the above two-component cognitive load detector. Additionally, the ML detector generates an output indicating the overall cognitive load level of the driver:

[0093] 0: Low overall load.

[0094] 1: Medium overall load.

[0095] 2: High overall load.

[0096] In another example, the ML detector can implement a three-component cognitive load detector similar to the above detector, but trained to determine the overall cognitive load of the driver in a wider range of states. The following protocol can be used to train this three-component cognitive load detector:

[0097] Phase 1:

[0098] Requirement: There are no complex driving-related tasks. There are no additional non-driving-related tasks.

[0099] Target values: The ML detector will output a value of 0 for "driving task", a value of 0 for "non-driving task" and a value of 1 for "overall load". A value of 1 for "overall load" indicates the basic level of cognitive load in normal driving situations without being distracted by non-driving related tasks.

[0100] Phase 2:

[0101] Requirements: There are no complex driving-related tasks. There are additional non-driving related tasks.

[0102] Target values: The ML detector will output a value of 0 for "driving task", a value of 1 for "non-driving task" and a value of 2 for "overall load".

[0103] Phase 3:

[0104] Requirements: There are complex driving-related tasks. There are no additional non-driving related tasks.

[0105] Target values: The ML detector will output a value of 1 for "driving task", a value of 0 for "non-driving task" and a value of 2 for "overall load".

[0106] Phase 4:

[0107] Requirements: There are complex driving-related tasks. There are additional non-driving related tasks.

[0108] Target values: The ML detector will output a value of 1 for "driving task", a value of 1 for "non-driving task" and a value of 3 for "overall load".

[0109] Phase 5:

[0110] Requirements: The overall activity level is very low due to (e.g.) drowsiness.

[0111] Target values: The ML detector will not output values (or "undefined") for "driving task" and "non-driving task", and will output a value of 0 for "overall load".

[0112] After training, when fed data representing physiological parameters obtained in real-life driving situations, the ML detector can generate an output signal corresponding to those of the above-mentioned three-component cognitive load detector. However, in this example, the output indicating the driver's overall cognitive load has a higher resolution:

[0113] 0: Very low overall load.

[0114] 1: Low overall load.

[0115] 2: Medium overall load

[0116] 3: High overall load.

[0117] The above represents a simple illustrative example of how different components of the driver's state can be combined to determine whether the driver is in a state suitable for driving. An output of 0 or 3 for the total load can be considered a critical output and is used to intervene in the operation of the vehicle, as described above. An output of 1 or 2 can be considered to correspond to "healthy", that is, a safe driving state.

[0118] As described above, the input signal of the ML detector represents one or more physiological parameters indicating different components of the user's state. The input signal can be obtained from sensors such as Figure 2 cameras 102 and 104 in. The input signal can represent a single parameter from which multiple components of the user's state can be derived. Alternatively, the input signal can represent a combination of different physiological parameters, such as eye movement, eye closure, gaze direction, and combinations thereof; heartbeat parameters, for example, pulse wave, IBI (interbeat interval), HR (heart rate), HRV (heart rate variability), and one or more of their derivatives; respiratory information, such as frequency, respiratory pattern; muscle activity, temperature, pressure, blood oxygen saturation, etc.

[0119] Such physiological parameters can be derived from various physical signals in active and passive sensing systems, for example, video data from IR or visible light range cameras with or without active backlighting; reflected radar signals at different frequencies and different construction principles (pulse, constant radiation, beam pattern control, etc.); voltages derived from contact sensors related to muscle activity (different muscle groups); heartbeat activity (ECG); brain activity (EEG); pressure sensors, compression sensors, displacement sensors, etc. that can be attached or linked to the human body; etc.

[0120] The components of the user's overall state can represent different emotions and combinations thereof, which affect one or more measurable physiological parameters. Such emotions can include anger, disgust, fear, joy, sadness, surprise, indifference, etc. Such emotions can be described or measured in different ways, for example, based on sensory activities (including subtypes such as vision, hearing, touch, smell, vestibular, taste); motor activities (including subtypes); cognitive processes (including subtypes such as memory, attention, thinking); emotional processes (including subtypes); etc.

[0121] The present disclosure is not limited to a specific type of component. However, any component should be detectable, for example, by measuring physiological parameters. Specifically, each component should be reproducible in a training or experimental environment so that the measurement of the associated physiological parameters can be interpreted.

[0122] The above method has been described in connection with a state detector, but it can also be used as a data mining tool to extract and identify the psychophysiological state of a person based on data describing the person's physiological parameters.

[0123] The description of the embodiments and aspects has been presented for illustrative and explanatory purposes only. In light of the foregoing, suitable modifications and variations can be made to the embodiments and aspects without departing from the scope of protection as determined by the claims, and different embodiments and aspects can be combined where possible and appropriate.

Claims

1. A computer-implemented method for detecting a functional state of a user, the method comprising: Obtaining one or more signals from one or more sensors, wherein the signals represent multiple physiological parameters of the user; Determining multiple conditions associated with the user based on the values of the physiological parameters; Determining the functional state of the user based on a combination of the multiple conditions; And Generating an output signal to indicate the functional state of the user.

2. The computer-implemented method according to claim 1, further comprising: Determining at least one of the conditions based on at least one other condition among the multiple conditions.

3. The computer-implemented method according to claim 2, further comprising: Determining at least one of the conditions based on a combination or pattern of other conditions among the conditions.

4. The computer-implemented method according to claim 1, wherein each of the conditions represents a level of cognitive load, stress, or drowsiness of the user, and / or wherein the functional state of the user indicates the user's adaptability to an operation, particularly driving a vehicle.

5. The computer-implemented method according to claim 4, wherein the cognitive load includes a cognitive load related and / or unrelated to the user's operation, particularly a cognitive load related and / or unrelated to driving the vehicle.

6. The computer-implemented method according to any one of the preceding claims, wherein determining the functional state of the user includes selecting one of a plurality of predetermined states, wherein each of the plurality of predetermined states is associated with a corresponding combination or pattern of the conditions.

7. The computer-implemented method according to any one of the preceding claims, further comprising: Determining that some of the conditions are mutually exclusive; And Discarding one or more or all of the mutually exclusive conditions when determining the functional state of the user.

8. The computer-implemented method according to any one of the preceding claims, wherein the method is implemented in a vehicle, and wherein generating an output signal indicating the functional state of the user includes one or more of the following: Reminding the user of the vehicle; Controlling the vehicle, particularly adjusting the user interface in the vehicle or adjusting one or more vehicle control systems; Communicating the functional state of the user to another vehicle or an external server or control station.

9. The computer-implemented method according to any one of the preceding claims, wherein the one or more physiological parameters include: One or more heartbeat parameters, particularly heart rate, heart rate variability, and / or inter-beat interval; And / or One or more eye parameters, particularly eye movement, eyelid movement, eye position, eyelid position, and / or eye fixation.

10. A system for detecting a functional state of a user, the system comprising: One or more sensors that detect one or more signals, wherein the signals represent multiple physiological parameters of the user; And At least one processing device coupled to the sensor and configured to: Determine multiple conditions associated with the user based on the values of the physiological parameters; Determine the functional status of the user based on a combination of multiple conditions; and Generate an output signal to indicate the functional status of the user.

11. The system according to claim 10, wherein the processing device is configured to: Determine at least one of the conditions based on at least one other condition among the conditions, in particular based on a combination or pattern of at least one other condition among the conditions.

12. The system according to claim 10 or 11, wherein the processing device is configured to determine the functional status of the user by selecting one of a plurality of predetermined states, wherein each of the plurality of predetermined states is associated with a corresponding combination of the conditions.

13. The system according to any one of claims 10 to 12, wherein the processing device is configured to: Determine that some of the conditions are mutually exclusive; and Discard one or more or all of the mutually exclusive conditions when determining the functional status of the user.

14. The system according to any one of claims 10 to 13, wherein the system is a vehicle-mounted system, and wherein generating an output signal indicating the functional status of the user includes one or more of the following: Remind the user of the vehicle; Control the vehicle, in particular adjust the user interface in the vehicle or adjust one or more vehicle control systems; Communicate the functional status of the user to another vehicle or an external server or a control station.

15. The system according to any one of claims 10 to 14, wherein the system is a vehicle-mounted system, and / or wherein the one or more sensors include one or more image sensors, in particular RGB and / or infrared cameras.

Images