Driver state recognition system and method

Abstract

The embodiment of the invention provides a driver state recognition system and method. The method comprises the following steps: acquiring driving information of a vehicle; the driving information of the vehicle is sent to a driver detection module; detecting various types of state information of the driver; sending the various types of state information to a driver detection module; an attitude video and / or a face video of a driver are / is collected, and the collected attitude video and / or the face video are / is sent to the driver detection module; calculating according to the received posture video and / or face video, and identifying a preset personalized model corresponding to the driver; judging whether the driver is in an abnormal state or not through a preset personalized model; when it is judged that the driver is in the abnormal state, reminding information is sent to the driver. Incremental training and evolution evaluation of the preset personalized module can be carried out through the collected information, and through the scheme of the embodiment of the invention, the safety in the vehicle driving process can be improved.

Description

Technical Field

[0001] The present application relates to the field of information technology, and in particular to a driver status identification system and method. Background Art

[0002] With the rise in car ownership, most families now own their own vehicle. However, while vehicles bring convenience to people's lives, they also often carry certain risks. For example, due to driver errors or environmental factors, drivers may become distracted or experience abnormal behaviors, posing significant risks to passengers and passersby. Summary of the Invention

[0003] The purpose of the embodiments of the present application is to provide a driver status recognition system and method for identifying and reminding drivers of distraction. The specific technical solution is as follows:

[0004] In a first aspect of an embodiment of the present application, a driver status recognition system is provided, the system comprising: a vehicle sensor, an in-vehicle camera, multiple types of human body sensors, and a driver detection module;

[0005] The vehicle sensor is used to obtain vehicle driving information; and send the vehicle driving information to the driver detection module;

[0006] The multiple types of human body sensors are used to detect multiple types of status information of the driver; and send the multiple types of status information to the driver detection module;

[0007] The in-car camera is arranged at the A-pillar and / or the rearview mirror of the vehicle; the in-car camera is used to collect the posture video and / or facial video of the driver, and send the collected posture video and / or facial video to the driver detection module;

[0008] The driver detection module is used to perform calculations based on the received posture video and / or facial video and identify a preset personalized model corresponding to the driver; determine whether the driver is in an abnormal state based on the preset personalized model; and send a reminder message to the driver when it is determined that the driver is in an abnormal state.

[0009] In a possible implementation, the vehicle sensor is used to obtain vehicle steering wheel angle, lane offset, and vehicle speed information.

[0010] In one possible implementation, the multiple types of human body sensors include: one or more of a wristband-type skin electrocardiogram sensor, a belt-type respiratory sensor, a wearable eye tracker, and a head-mounted EEG sensor;

[0011] The wristband-type skin electrocardiogram sensor is used to obtain the driver's skin conductivity, skin electrocardiogram frequency, heart rate variability, heart rate acceleration, and ventricular beat interval;

[0012] The belt-type respiratory sensor is used to obtain the driver's respiratory frequency and respiratory variability parameters;

[0013] The wearable eye tracker is used to obtain the driver's blink duration, eyelid closure speed, pupil diameter change rate, blink frequency change, pupil dilation amplitude, gaze duration and scan path length;

[0014] The head-mounted EEG sensor is used to obtain the driver's EEG signals, EMG signals, and EOG signals.

[0015] In a possible implementation, the in-vehicle camera is specifically configured to capture the posture video and / or the facial video, and send the captured posture video and / or the facial video to the driver detection module;

[0016] The driver detection module specifically splits the multiple types of status information and extracts feature variables based on the splitting results; time-tags the extracted feature vectors to obtain first time series data; extracts human body key point coordinates and / or skeleton data based on the posture video; calculates the corresponding center point coordinates of the left hand, right hand, left arm, right arm, body, head, and shoulder based on the extracted human body key point coordinates and / or skeleton data; calculates the projection of the connection vector of each arm and the corresponding hand of the driver to the body midline vector based on the extracted human body key point coordinates and / or skeleton data. distance; calculating the angle between target vectors based on the extracted human body key point coordinates and / or skeleton data; time-marking the center point coordinates, the distance, and the angle to obtain second time series data; and / or, extracting facial key point coordinates based on the facial video; constructing a facial feature model of the driver based on the facial key point coordinates; calculating at least one of the driver's head turning angle, yawning frequency, and blinking frequency based on the facial feature model; time-marking at least one of the head turning angle, yawning frequency, and blinking frequency to obtain third time series data;

[0017] Whether the driver is in an abnormal state is determined based on the first time series data, the second time series data and / or the third time series data, and a preset personalized model.

[0018] In one possible implementation, the training process of the preset personalized model includes:

[0019] Obtain multiple sets of initial data;

[0020] By using principal component analysis dimensionality reduction method, the initial data is reduced into multiple groups of main feature data;

[0021] Dividing the multiple sets of main feature data into training sets and test sets;

[0022] The preset LSTM / Transformer model is trained using the training set and tested using the test set to obtain the preset personalized model that meets the preset test requirements.

[0023] In a possible implementation, the driver detection module is specifically configured to send a reminder instruction to the wristband-type skin conduction electrocardiogram sensor when determining that the driver is in an abnormal state, so that the wristband-type skin conduction electrocardiogram sensor reminds the driver through vibration and / or sound.

[0024] In a possible implementation, the driver detection module is further used to identify the preset personalized model corresponding to the driver based on the driving information, the status information, at least one of the posture video and / or facial video, and a personalized model library.

[0025] In one possible implementation, the driver detection module specifically sends the first time series data, the second time series data and / or the third time series data to the back end to verify the confidence of the preset personalized model, and perform incremental training on the preset personalized model when the confidence does not meet the preset confidence requirement.

[0026] In one possible implementation, the incremental training process includes:

[0027] The preset personalized model is used as the teacher model, and the model to be updated is used as the student model for training to obtain a trained model;

[0028] The preset personalized model, the model to be updated and the trained model are evaluated in terms of weighted WF1, catastrophic forgetting rate and AUC-PR gain, and the model determined by the evaluation is added to the personalized model library.

[0029] A second aspect of an embodiment of the present application provides a driver status recognition method, which is applied to a driver status recognition system, the system comprising: a vehicle sensor, an in-vehicle camera, multiple types of human body sensors, and a driver detection module;

[0030] The vehicle sensor is used to obtain vehicle driving information and send the vehicle driving information to the driver detection module; the multiple types of human body sensors are used to detect multiple types of driver status information and send the multiple types of status information to the driver detection module; the in-vehicle camera is set at the A-pillar and / or rearview mirror of the vehicle; the in-vehicle camera is used to capture the driver's posture video and / or facial video and send the captured information to the driver detection module;

[0031] The method comprises:

[0032] Performing calculations based on the received posture video and / or facial video, and identifying a preset personalized model corresponding to the driver;

[0033] It is determined by the preset personalized model whether the driver is in an abnormal state; when it is determined that the driver is in an abnormal state, a reminder message is sent to the driver.

[0034] In a possible implementation, the vehicle sensor is used to obtain vehicle steering wheel angle, lane offset, and vehicle speed information.

[0035] In one possible implementation, the multiple types of human body sensors include: one or more of a wristband-type skin electrocardiogram sensor, a belt-type respiratory sensor, a wearable eye tracker, and a head-mounted EEG sensor;

[0036] The wristband-type skin electrocardiogram sensor is used to obtain the driver's skin conductivity, skin electrocardiogram frequency, heart rate variability, heart rate acceleration, and ventricular beat interval;

[0037] The belt-type respiratory sensor is used to obtain the driver's respiratory frequency and respiratory variability parameters;

[0038] The wearable eye tracker is used to obtain the driver's blink duration, eyelid closure speed, pupil diameter change rate, blink frequency change, pupil dilation amplitude, gaze duration and scan path length;

[0039] The head-mounted EEG sensor is used to obtain the driver's EEG signals, EMG signals, and EOG signals.

[0040] In one possible implementation, determining whether the driver is in an abnormal state based on the driving information, the state information, the posture video and / or the facial video, and a preset personalized model includes:

[0041] Splitting the multiple types of status information and extracting feature variables based on the splitting results; time-marking the extracted feature vectors to obtain first time series data;

[0042] Extracting human body key point coordinates and / or skeleton data based on the posture video; calculating the corresponding center point coordinates of the left hand, right hand, left arm, right arm, body, head, and shoulder based on the extracted human body key point coordinates and / or skeleton data; calculating the distance between the connection vector of each arm and the corresponding hand of the driver and the projection to the body midline vector based on the extracted human body key point coordinates and / or skeleton data; calculating the angle between target vectors based on the extracted human body key point coordinates and / or skeleton data; time-marking the center point coordinates, the distance, and the angle to obtain second time series data; and / or, extracting facial key point coordinates based on the facial video; constructing the driver's facial feature model based on the facial key point coordinates; calculating at least one of the driver's head turning angle, yawning frequency, and blinking frequency based on the facial feature model; time-marking at least one of the head turning angle, yawning frequency, and blinking frequency to obtain third time series data;

[0043] Whether the driver is in an abnormal state is determined based on the first time series data, the second time series data and / or the third time series data, and a preset personalized model.

[0044] In one possible implementation, the training process of the preset personalized model includes:

[0045] Obtain multiple sets of initial data;

[0046] By using principal component analysis dimensionality reduction method, the initial data is reduced into multiple groups of main feature data;

[0047] Dividing the multiple sets of main feature data into training sets and test sets;

[0048] The preset LSTM / Transformer model is trained using the training set and tested using the test set to obtain the preset personalized model that meets the preset test requirements.

[0049] In a possible implementation, when determining that the driver is in an abnormal state, sending a reminder message to the driver includes:

[0050] When it is determined that the driver is in an abnormal state, a reminder instruction is sent to the wristband-type skin electrocardiogram sensor, so that the wristband-type skin electrocardiogram sensor reminds the driver through vibration and / or sound.

[0051] In one possible implementation, the method further includes:

[0052] The preset personalized model corresponding to the driver is identified based on the driving information, the state information, at least one of the posture video and / or the facial video, and a personalized model library.

[0053] In one possible implementation, the method further includes:

[0054] The first time series data, the second time series data and / or the third time series data are sent to the back end to verify the confidence of the preset personalized model, and to perform incremental training on the preset personalized model when the confidence does not meet the preset confidence requirement.

[0055] In one possible implementation, the incremental training process includes:

[0056] The preset personalized model is used as the teacher model, and the model to be updated is used as the student model for training to obtain a trained model;

[0057] The preset personalized model, the model to be updated and the trained model are evaluated in terms of weighted WF1, catastrophic forgetting rate and AUC-PR gain, and the model determined by the evaluation is added to the personalized model library.

[0058] Another aspect of the present application provides an electronic device, including:

[0059] Memory for storing computer programs;

[0060] The processor is configured to implement any of the above-mentioned driver status identification methods when executing a program stored in the memory.

[0061] In another aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, any of the above-mentioned driver status recognition methods is implemented.

[0062] In another aspect of an embodiment of the present application, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute any of the above-mentioned driver status identification methods.

[0063] Beneficial effects of the embodiments of the present application:

[0064] An embodiment of the present application provides a driver status recognition system and method, the system comprising: a vehicle sensor, an in-vehicle camera, multiple types of human body sensors, and a driver detection module; the vehicle sensor is used to obtain vehicle driving information; and the vehicle driving information is sent to the driver detection module; the multiple types of human body sensors are used to detect multiple types of driver status information; and the multiple types of status information are sent to the driver detection module; the in-vehicle camera is arranged at the A-pillar and / or rearview mirror of the vehicle; the in-vehicle camera is used to collect the driver's posture video and / or facial video, and send the collected posture video and / or facial video to the driver detection module; the driver detection module is used to calculate based on the received posture video and / or facial video, and identify a preset personalized model corresponding to the driver; determine whether the driver is in an abnormal state through the preset personalized model; and send a reminder message to the driver when it is determined that the driver is in an abnormal state. Through the solution of the present application, it is possible to obtain a video of the driver's posture and / or face, and then determine whether the driver is in an abnormal state through the preset personalized model corresponding to the driver. Then, when it is determined that the driver is in an abnormal state, a reminder message is sent to the driver. This not only makes it possible to determine the driver's abnormal state, but also to provide a reminder, thereby improving the safety of the vehicle driving process.

[0065] Of course, it is not necessary to achieve all the advantages described above at the same time when implementing any product or method of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0067] Figure 1 A schematic diagram of the structure of a driver status recognition system provided in an embodiment of the present application;

[0068] Figure 2 A schematic diagram of data collection by a vehicle sensor provided in an embodiment of the present application;

[0069] Figure 3 A schematic diagram of data collection for a wristband-type skin electrocardiogram sensor provided in an embodiment of the present application;

[0070] Figure 4 A schematic diagram of data collection for a belt-type respiratory sensor provided in an embodiment of the present application;

[0071] Figure 5 A schematic diagram of data collection for a wearable eye tracker provided in an embodiment of the present application;

[0072] Figure 6 A schematic diagram of a driver detection module provided in an embodiment of the present application;

[0073] Figure 7 A schematic diagram of key points of the body provided in an embodiment of the present application;

[0074] Figure 8 A schematic diagram of the key points of the hand provided in an embodiment of the present application;

[0075] Figure 9 A schematic diagram of the classification of the coordinate part provided in the embodiment of the present application;

[0076] Figure 10 A schematic diagram of partial classification of vectors provided in an embodiment of the present application;

[0077] Figure 11 A schematic diagram of the classification of angles provided in the embodiment of the present application

[0078] Figure 12 A schematic diagram of the classification of angles provided in an embodiment of the present application;

[0079] Figure 13 A flowchart of a driver status identification method provided in an embodiment of the present application;

[0080] Figure 14 A flow chart of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0081] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field based on this application are within the scope of protection of this application.

[0082] In a first aspect of the embodiment of the present application, a driver status recognition system is first provided. Figure 1 , Figure 1 A schematic diagram of the structure of a driver status recognition system provided in an embodiment of the present application; the system includes: a vehicle sensor 101, an in-vehicle camera 102, various types of human body sensors 103, and a driver detection module 104;

[0083] The vehicle sensor 101 is used to obtain vehicle driving information and send the vehicle driving information to the driver detection module;

[0084] The multiple types of human body sensors 103 are used to detect multiple types of status information of the driver; and send the multiple types of status information to the driver detection module;

[0085] The in-vehicle camera 102 is arranged at the A-pillar and / or the rearview mirror of the vehicle; the in-vehicle camera 102 is used to collect the posture video and / or facial video of the driver and send the collected posture video and / or facial video to the driver detection module;

[0086] The driver detection module 104 is configured to perform calculations based on the received posture video and / or facial video and identify a preset personalized model corresponding to the driver; determine whether the driver is in an abnormal state using the preset personalized model; and send a reminder message to the driver when it is determined that the driver is in an abnormal state.

[0087] Among them, when the vehicle sensor in the embodiment of the present application obtains the driving information of the vehicle, the obtained driving information of the vehicle may include information such as the vehicle's speed and steering wheel angle. The vehicle sensor, the in-vehicle camera, and various types of human body sensors in the embodiment of the present application may all be communicatively connected to the driver detection module, so that after the vehicle sensor obtains the driving information of the vehicle, the vehicle's driving information may be sent to the driver detection module via the communication connection. The various types of human body sensors may include a breathing sensor, an electrocardiogram sensor, and the like. Various types of status information of the driver may be detected by the various types of human body sensors, and the various types of status information may be sent to the driver detection module via the communication connection.

[0088] The in-car camera in the embodiment of the present application can be installed at the A-pillar and / or rearview mirror of the vehicle, and during the installation process, the direction of the camera can be toward the driver, so as to facilitate the collection of the driver's posture video and / or facial video, and the posture video and / or facial video can be sent to the driver detection module via a communication connection. In actual use, the above-mentioned communication connection can be a wired or wireless communication connection. Among them, the driver's facial video can only include the driver's face, while the posture video can include the driver's arms and / or upper limbs.

[0089] In an embodiment of the present application, the driver detection module can calculate and identify a preset personalized model corresponding to the driver based on the received information. Specifically, multiple models can be pre-set, and then a model for the current driver can be identified. This preset personalized model can then be used to determine whether the driver is in an abnormal state. The preset personalized model can be specific to the current driver or a group of people. For example, personalized models can be set for different age groups, different genders, or different individuals. In one example, multiple types of models can be pre-set, and then a model corresponding to the current driver can be identified based on one or more of the driving information, the status information, the posture video, and / or the facial video. This can then be used to determine whether the driver is distracted. Specifically, the preset personalized model can determine whether the current driver is distracted based on one or more of the status information, the posture video, and the facial video. When the driver is determined to be in an abnormal state, a reminder message can be sent to the driver. Specifically, the reminder can be provided through sound, vibration, or other means. In one example, a sound alert can be sent when the driver is determined to be distracted. In another example, when the driver is distracted, a reminder can be given by vibrating the seat. The abnormal state in the embodiment of the present application can refer to a state such as distraction, fatigue, abnormal mood, sudden illness, etc.

[0090] It can be seen that through the solution of the present application, the posture video and / or facial video of the driver can be obtained, so as to determine whether the driver is in an abnormal state through the preset personalized model corresponding to the driver, and then when the driver is determined to be in an abnormal state, a reminder message is sent to the driver, which not only can realize the judgment of the driver's abnormal state, but also can provide a reminder, thereby improving the safety of the vehicle driving process.

[0091] In a possible implementation, the vehicle sensor is used to obtain vehicle steering wheel angle, lane offset, and vehicle speed information.

[0092] In one possible implementation, the multiple types of human body sensors include: one or more of a wristband-type skin electrocardiogram sensor, a belt-type respiratory sensor, a wearable eye tracker, and a head-mounted EEG sensor;

[0093] The wristband-type skin electrocardiogram sensor is used to obtain the driver's skin conductivity, skin electrocardiogram frequency, heart rate variability, heart rate acceleration, and ventricular beat interval;

[0094] The belt-type respiratory sensor is used to obtain the driver's respiratory frequency and respiratory variability parameters;

[0095] The wearable eye tracker is used to obtain the driver's blink duration, eyelid closure speed, pupil diameter change rate, blink frequency change, pupil dilation amplitude, gaze duration and scan path length;

[0096] The head-mounted EEG sensor is used to obtain the driver's EEG signals, EMG signals, and EOG signals.

[0097] Specifically, in the embodiment of the present application, the vehicle sensor can be used to obtain vehicle parameters such as the vehicle steering wheel angle, lane offset, and vehicle speed. In an example, see Figure 2 , Figure 2 This is a schematic diagram of data collection from a vehicle sensor provided in an embodiment of the present application. The vehicle sensor can collect information such as steering wheel angle, lane offset, vehicle speed, collision time, and headway. Lane offset refers to the distance a vehicle deviates from the center of its normal lane during driving, typically calculated by measuring the offset between the center of the vehicle and the centerline of the lane. Collision time refers to the time of collision if a collision occurs. Headway represents the time difference between the front ends of the two vehicles passing the same location, typically calculated by dividing the headway distance between the front and rear vehicles by the speed of the rear vehicle. Specifically, headway can also represent the maximum reaction time the driver of the rear vehicle has when the front vehicle brakes.

[0098] The wristband skin electrocardiogram sensor can be used to obtain parameters such as skin conductivity, skin electrocardiogram frequency, heart rate variability, heart rate acceleration, ventricular beat interval and heart rate trend fluctuation analysis. For an example, see Figure 3 , Figure 3This is a data acquisition diagram of the wristband-type skin electrocardiogram sensor provided in an embodiment of the present application. The wristband-type skin electrocardiogram sensor is used to collect information such as the driver's heart rate variability, ventricular beat interval, SDNN (Standard Deviation of NNIntervals), pNN50, heart rate detrended fluctuation analysis, skin conductivity and skin electrical response frequency. Heart rate variability (HRV) refers to the variation in heart rate between heartbeats. It contains information about the regulation of the cardiovascular system by neurohumoral factors, thereby assessing the severity and prevention of cardiovascular diseases and potentially serving as a valuable indicator for predicting sudden cardiac death and arrhythmic events. Interventricular beat interval (ISBI) refers to the time interval between heartbeats and can be measured via electrocardiogram (ECG). SDNN measures the standard deviation of the sinus interval (SDNN), a parameter that reflects the regulatory capacity of the autonomic nervous system and overall cardiac health. pNN50 represents the time interval between heartbeats, representing the percentage of consecutive cardiac cycles with a difference greater than 50 milliseconds. Skin conductivity reflects the electrical conductivity of the skin surface and is closely related to sympathetic nervous system activity. When the human body is tense, excited, or stressed, sympathetic nervous system activation stimulates sweat gland secretion, increasing electrolyte concentrations in the skin's surface and enhancing electrical conductivity. This property makes it useful for mood monitoring and stress assessment. Skin galvanic response (GSSR) frequency can be measured using a galvanic skin stimulation feedback device. Specifically, GSSR reflects changes in skin electrical conduction when stimulated, reflecting the activity of the autonomic nervous system.

[0099] A belt-mounted respiratory sensor can be used to obtain respiratory rate and respiratory variability parameters. For an example, see Figure 4 , Figure 4 A schematic diagram of data acquisition for a belt-mounted respiratory sensor provided in an embodiment of the present application. Specifically, the belt-mounted respiratory sensor can collect respiratory rate and respiratory variability parameters. Respiratory rate represents the number of breaths per minute, while respiratory variability indicates the degree to which respiratory rate, tidal volume, and other parameters vary with the respiratory cycle during respiration.

[0100] Wearable eye trackers can be used to obtain parameters such as blink duration, eyelid closure speed, pupil diameter change rate, blink frequency change, pupil dilation amplitude, fixation duration, and scan path length. Head-mounted EEG sensors can be used to obtain EEG, EMG, and EOG signals and perform denoising to obtain denoised EEG signals. The in-car camera can be installed on the vehicle's A-pillar and rearview mirror to obtain real-time video segments of the driver's body posture and face. For an example, see Figure 5 , Figure 5Schematic diagram of data collection for a wearable eye tracker provided in an embodiment of the present application. The wearable eye tracker can collect parameters such as blink duration, eyelid closure speed, pupil diameter change rate, blink frequency change, pupil dilation amplitude, fixation duration, and saccade path length. Among them, blink duration represents the time taken for a single blink; eyelid closure speed represents the time taken for the eyelid to close; pupil diameter change rate refers to the degree of change in pupil diameter per unit time, which can be used to quantify the dynamic response speed of the pupil to specific stimuli (such as light, pain, or changes in neural status); blink frequency change can represent the amount of change in blink frequency; pupil dilation amplitude can represent the amount of change in pupil dilation per unit time; fixation duration can represent the time period when the eyeball is relatively still between two saccades; and saccade path length can represent the actual geometric length of the eyeball's motion trajectory when it moves rapidly (saccade motion) between two fixation points.

[0101] In a possible implementation, the in-vehicle camera is specifically configured to capture the posture video and / or the facial video, and send the captured posture video and / or the facial video to the driver detection module;

[0102] The driver detection module specifically splits the multiple types of status information and extracts feature variables based on the splitting results; time-tags the extracted feature vectors to obtain first time series data; extracts human body key point coordinates and / or skeleton data based on the posture video; calculates the corresponding center point coordinates of the left hand, right hand, left arm, right arm, body, head, and shoulder based on the extracted human body key point coordinates and / or skeleton data; calculates the projection of the connection vector of each arm and the corresponding hand of the driver to the body midline vector based on the extracted human body key point coordinates and / or skeleton data. distance; calculating the angle between target vectors based on the extracted human body key point coordinates and / or skeleton data; time-marking the center point coordinates, the distance, and the angle to obtain second time series data; and / or, extracting facial key point coordinates based on the facial video; constructing a facial feature model of the driver based on the facial key point coordinates; calculating at least one of the driver's head turning angle, yawning frequency, and blinking frequency based on the facial feature model; time-marking at least one of the head turning angle, yawning frequency, and blinking frequency to obtain third time series data;

[0103] Whether the driver is in an abnormal state is determined based on the first time series data, the second time series data and / or the third time series data, and a preset personalized model.

[0104] For an example, see Figure 6 , Figure 6A schematic diagram of a driver detection module provided in an embodiment of the present application. The driver detection module can obtain data input from a wristband-type electrocardiogram (ECG) skin sensor, a belt-type respiratory sensor, a wearable eye tracker, and vehicle sensors, and can also obtain filtered and denoised information from a head-mounted EEG sensor. When receiving data from an in-vehicle camera, the driver detection module can extract skeleton coordinate information from the captured video, where the skeleton coordinate information can include a coordinate component, a vector component, and an angle component.

[0105] In the embodiment of the present application, the sensor data can be first split and calculated into feature variables, such as splitting the EEG signal filtering into α, β, and θ bands after denoising. The coordinates and skeleton data of the key points of the body are obtained through the mediapipe framework (a framework for constructing audio, video or any time series data) for the video segment of the A-pillar camera in the car, and the coordinates and skeleton data of the key points of the body are split into three modes: coordinate part, projection part, and angle part. The coordinate part (X, Y, Z) is split into seven parts: left and right hand, left and right arm, body, head, and shoulder, and the center point coordinates of each part are calculated. For example, see Figure 7 , Figure 7 A schematic diagram of the key points of the body provided in an embodiment of the present application, in which 1 is the nose, 2 to 4 are the right eye, 5 to 7 are the left eye, 8 is the right ear, 9 is the left ear, 10 to 11 are the lips, 12 is the left shoulder, 13 is the right shoulder, 14 is the right elbow, 15 is the right wrist, 16 is the left elbow, 17 is the left wrist, and 18 to 19 are the hips. In an example, there are 21 key points on the hand, see Figure 8 , Figure 8 A schematic diagram of the key points of the hand provided in the embodiment of the present application. In the figure, 20 is the palm; 21, 25, 29, 33, and 37 represent the metacarpophalangeal joints of different fingers; 22, 26, 30, 34, and 38 represent the proximal interphalangeal joints of different fingers; 23, 27, 31, 35, and 39 represent the distal interphalangeal joints of different fingers; and 24, 28, 32, 36, and 40 represent the distal interphalangeal joints of different fingers. Specifically, the coordinates of each point are (x0~x 20 , y0~y 20 , z0~z 20 ) The center point is calculated as The same applies to the remaining six parts. Figure 9 , Figure 9 A schematic diagram of the classification of the coordinate part provided in the embodiment of the present application. The coordinate part can include seven parts: left and right hands, left and right arms, body, head, and shoulders. The projection part is to project the connecting vector of each key point of the driver's arm and hand onto the midline of the body to obtain the vector distance. Figure 10 , Figure 10A schematic diagram of the classification of vector parts provided in an embodiment of the present application, wherein the vector part may include the vector distance L obtained by projecting the connecting vector of the arm key point 106 to the body midline 106 , and the projection of the connection vector of the arm key point 102 to the body midline to obtain the vector distance L 102 The angle part is used to calculate the angle between vectors. Figure 11 , Figure 11 A schematic diagram of the partial classification of angles provided for an embodiment of the present application may include an arm angle <116 corresponding to arm key point 116 and an arm angle <112 corresponding to arm key point 112. Similarly, for the rearview mirror camera, the facial key point coordinate data is obtained, and facial features are constructed through cubic spline interpolation. At the same time, the head turning angle, yawning frequency, blinking frequency, etc. are calculated, and the above feature data are automatically time-annotated and converted into time series data. In an embodiment of the present application, the driver can also be identified through vehicle sensor data and facial features, and a personalized model library can be established.

[0106] In one possible implementation, the training process of the preset personalized model includes: obtaining multiple groups of initial data; reducing the initial data into multiple groups of main feature data through principal component analysis dimensionality reduction method; dividing the multiple groups of main feature data into training sets and test sets; training the preset LSTM (Long Short-Term Memory) / Transformer model through the training set, and testing it through the test set to obtain the preset personalized model that meets the preset test requirements. In an embodiment of the present application, the initial data can be for recruiting drivers of different ages, genders, and driving experience to conduct simulated driving experiments, and driving tests are conducted at different times to stimulate abnormal states of the driver, such as fatigue, distraction, abnormal emotions, etc. Specifically, the initial data of the simulated driving experiment is reduced to multiple main feature data with a cumulative contribution rate of >95% through principal component analysis, and the factor loading coefficient of each principal component is obtained and the PCA (principal component analysis) matrix is frozen. The initial data was randomly divided into an 80% training set and a 20% test set, and a manually selected buffer feature set with typical characteristics was used for training. The model was trained using an LSTM / Transformer model and evaluated using the test set. Evaluation metrics included accuracy, recall, and F1 value (a value used to measure model accuracy) to measure the model's predictive performance. Finally, the optimal machine learning model was obtained through ten-fold cross-validation and set as the general model for cold start initialization.

[0107] In one possible implementation, the driver detection module is specifically configured to, upon determining that the driver is in an abnormal state, send a reminder instruction to the wristband-type skin conduction electrocardiogram sensor, so that the wristband-type skin conduction electrocardiogram sensor alerts the driver through vibration and / or sound. Specifically, when the driver detection module confirms that the driver is in an abnormal state based on real-time data, the wristband-type skin conduction electrocardiogram sensor may vibrate and sound reminders to ensure driving safety.

[0108] In one possible implementation, the driver detection module is further configured to identify the driver's corresponding preset personalized model based on at least one of the driving information, the state information, the posture video, and / or the facial video, and a personalized model library. Specifically, driver identification can be performed to determine the driver's identity, and the personalized model library can be searched and adapted. If the driver is confirmed to be a new driver, a cold-start initialized universal model is activated and established as the new driver model.

[0109] In one possible implementation, the driver detection module specifically sends the first time series data, the second time series data, and / or the third time series data to the backend to verify the confidence of the preset personalized model, and to perform incremental training on the preset personalized model when the confidence does not meet the preset confidence requirement. In one possible implementation, the incremental training process includes: using the preset personalized model as a teacher model and the model to be updated as a student model for training to obtain a trained model; evaluating the preset personalized model, the model to be updated, and the trained model in terms of weighted WF1 (used to characterize weighted accuracy), catastrophic forgetting rate, and AUC-PR (a probability value) gain, and adding the model determined by the evaluation to the personalized model library.

[0110] For details, see Figure 12 , Figure 12 A schematic diagram of the angle classification provided by the embodiment of this application. Confidence testing can be performed through real-time user feedback or subsequent feedback, and update trigger conditions such as driving time and low confidence in the original model can be established to further divide the new data into an 80% training set and a 20% test set. The current personalized model is used as the teacher model M t , the model to be updated is the student model M s .

[0111] Establish the joint loss function:

[0112] L total =α×CrossEntorpy(M s (X new ), Y new )+β×KL(Mt (X buf ) / M s (X buf ),

[0113] Among them, α and β are the new task loss weight and distillation loss weight. The new task loss weight increases with the increase of data volume. CrossEntorpy is the cross entropy loss function Among them, C is the total number of categories, N is the total number of samples, n is the sample number, and i is the category number. total Incremental training is performed to obtain the trained model, and the teacher model and cold start model are retained. The three models are input into the evolutionary evaluation module.

[0114] The evolutionary evaluation module uses the buffer data and the new data test set as the test set to evaluate the model and establish an evaluation matrix, including:

[0115] 1. Weighted F1: Among them, F1 K The F1 score of the k-th anomaly, ω K is the risk weighting coefficient, which increases as the severity of the abnormal state category risk increases;

[0116] 2. Catastrophic Forgetting Rate (CFR): is the F1 value of the teacher model for the k-th type of anomaly, is the F1 value of the new model for the k-th type of anomaly;

[0117] 3. AUC-PR Gain (APCG): AUC_PR is the area under the PR curve.

[0118] The evolution formula is EI = 0.5 × WF1 + 0.3 × (1-CFR) + 0.2 × max (APCG, 0), where WF1 is used to determine core performance, CFR is mainly used to prevent incremental model forgetting, and APCG is mainly used to confirm model interpretability. When APCG < -0.2, a rollback is forced. When EI > 0.85, an incremental update is triggered. This model is used as a driver model and entered into the personalized model library. Nearly 200 sets of driver data are buffered and replaced.

[0119] In an example, the facial camera captures the driver's facial features and matches them with the personalized driver model library. If there is no matching record, the general cold start model is loaded and a new driver profile is created locally. At the start of the startup, each sensor collects multi-source data, and the vehicle sensor collects driving habit information. The driver is verified twice. If the verification fails, the facial data is re-matched. After successful verification, the real-time feature vector is input into the driver state detection model. When the model determines that it is an abnormal state, it triggers the bracelet to vibrate and voice reminders, and records the event data at the same time. If the driver gives feedback, it is labeled; if the driver does not give feedback, the maximum probability judgment is taken for automatic labeling. The driving is often up to standard; abnormal state feedback is triggered multiple times; when the confidence of abnormal state judgment is low for multiple times in a row, the incremental learning update is triggered, and the personalized incremental learning strategy module is called. The old model is used as a teacher and the AdamW optimizer is used to pass L total Incremental training is performed to obtain the trained model. After training is complete, the evolutionary evaluation module evaluates the model using the buffer data and the new data test set as the test set. When APCG < -0.2, a rollback is forced. When EI > 0.85, an incremental update is triggered, and this model is added to the personalized model library as the driver model. Nearly 200 sets of driver data are replaced in the buffer.

[0120] A second aspect of an embodiment of the present application provides a driver status recognition method, which is applied to a driver status recognition system, the system comprising: a vehicle sensor, an in-vehicle camera, multiple types of human body sensors, and a driver detection module;

[0121] The vehicle sensor is used to obtain vehicle driving information and send the vehicle driving information to the driver detection module; the multiple types of human body sensors are used to detect multiple types of driver status information and send the multiple types of status information to the driver detection module; the in-vehicle camera is set at the A-pillar and / or rearview mirror of the vehicle; the in-vehicle camera is used to capture the driver's posture video and / or facial video and send the captured information to the driver detection module;

[0122] The method, see Figure 13 , Figure 13 A flowchart of a driver status identification method provided in an embodiment of the present application includes:

[0123] Step S131, performing calculations based on the received posture video and / or facial video, and identifying a preset personalized model corresponding to the driver;

[0124] Step S132: determining whether the driver is in an abnormal state through the preset personalized model; when it is determined that the driver is in an abnormal state, sending a reminder message to the driver.

[0125] In a possible implementation, the vehicle sensor is used to obtain vehicle steering wheel angle, lane offset, and vehicle speed information.

[0126] In one possible implementation, the multiple types of human body sensors include: one or more of a wristband-type skin electrocardiogram sensor, a belt-type respiratory sensor, a wearable eye tracker, and a head-mounted EEG sensor;

[0127] The wristband-type skin electrocardiogram sensor is used to obtain the driver's skin conductivity, skin electrocardiogram frequency, heart rate variability, heart rate acceleration, and ventricular beat interval;

[0128] The belt-type respiratory sensor is used to obtain the driver's respiratory frequency and respiratory variability parameters;

[0129] The wearable eye tracker is used to obtain the driver's blink duration, eyelid closure speed, pupil diameter change rate, blink frequency change, pupil dilation amplitude, gaze duration and scan path length;

[0130] The head-mounted EEG sensor is used to obtain the driver's EEG signals, EMG signals, and EOG signals.

[0131] In one possible implementation, determining whether the driver is in an abnormal state based on the driving information, the state information, the posture video and / or the facial video, and a preset personalized model includes:

[0132] Splitting the multiple types of status information and extracting feature variables based on the splitting results; time-marking the extracted feature vectors to obtain first time series data;

[0133] Extracting human body key point coordinates and / or skeleton data based on the posture video; calculating the corresponding center point coordinates of the left hand, right hand, left arm, right arm, body, head, and shoulder based on the extracted human body key point coordinates and / or skeleton data; calculating the distance between the connection vector of each arm and the corresponding hand of the driver and the projection to the body midline vector based on the extracted human body key point coordinates and / or skeleton data; calculating the angle between target vectors based on the extracted human body key point coordinates and / or skeleton data; time-marking the center point coordinates, the distance, and the angle to obtain second time series data; and / or, extracting facial key point coordinates based on the facial video; constructing the driver's facial feature model based on the facial key point coordinates; calculating at least one of the driver's head turning angle, yawning frequency, and blinking frequency based on the facial feature model; time-marking at least one of the head turning angle, yawning frequency, and blinking frequency to obtain third time series data;

[0134] Whether the driver is in an abnormal state is determined based on the first time series data, the second time series data and / or the third time series data, and a preset personalized model.

[0135] In one possible implementation, the training process of the preset personalized model includes:

[0136] Obtain multiple sets of initial data;

[0137] By using principal component analysis dimensionality reduction method, the initial data is reduced into multiple groups of main feature data;

[0138] Dividing the multiple sets of main feature data into training sets and test sets;

[0139] The preset LSTM / Transformer model is trained using the training set and tested using the test set to obtain the preset personalized model that meets the preset test requirements.

[0140] In a possible implementation, when determining that the driver is in an abnormal state, sending a reminder message to the driver includes:

[0141] When it is determined that the driver is in an abnormal state, a reminder instruction is sent to the wristband-type skin electrocardiogram sensor, so that the wristband-type skin electrocardiogram sensor reminds the driver through vibration and / or sound.

[0142] In one possible implementation, the method further includes:

[0143] The preset personalized model corresponding to the driver is identified based on the driving information, the state information, at least one of the posture video and / or the facial video, and a personalized model library.

[0144] In one possible implementation, the method further includes:

[0145] The first time series data, the second time series data and / or the third time series data are sent to the back end to verify the confidence of the preset personalized model, and to perform incremental training on the preset personalized model when the confidence does not meet the preset confidence requirement.

[0146] In one possible implementation, the incremental training process includes:

[0147] The preset personalized model is used as the teacher model, and the model to be updated is used as the student model for training to obtain a trained model;

[0148] The preset personalized model, the model to be updated and the trained model are evaluated in terms of weighted WF1, catastrophic forgetting rate and AUC-PR gain, and the model determined by the evaluation is added to the personalized model library.

[0149] It can be seen that through the solution of the present application, the posture video and / or facial video of the driver can be obtained, so as to determine whether the driver is in an abnormal state through the preset personalized model corresponding to the driver, and then when the driver is determined to be in an abnormal state, a reminder message is sent to the driver, which not only can realize the judgment of the driver's abnormal state, but also can provide a reminder, thereby improving the safety of the vehicle driving process.

[0150] The present application also provides an electronic device, such as Figure 14 Shown, including:

[0151] Memory 1401, used for storing computer programs;

[0152] The processor 1402 is configured to execute the program stored in the memory 1401 and implement the following steps:

[0153] Performing calculations based on the received posture video and / or facial video, and identifying a preset personalized model corresponding to the driver;

[0154] It is determined by the preset personalized model whether the driver is in an abnormal state; when it is determined that the driver is in an abnormal state, a reminder message is sent to the driver.

[0155] The communication bus mentioned in the electronic device mentioned above may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0156] The communication interface is used for communication between the above electronic device and other devices.

[0157] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0158] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0159] In another embodiment provided in the present application, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned driver status identification methods are implemented.

[0160] In another embodiment provided by the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any of the driver status identification methods in the above embodiments.

[0161] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state drive (SSD).

[0162] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0163] Each embodiment in this specification is described in a related manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences from other embodiments. In particular, the method, electronic device, and storage medium embodiments are generally similar to the system embodiments, so their descriptions are relatively simple. For related portions, refer to the descriptions of the system embodiments.

[0164] The above description is only a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application are included in the scope of protection of the present application.

Claims

1. A driver status recognition system, characterized in that: The system includes: vehicle sensors, in-vehicle cameras, various types of human body sensors, and a driver detection module; The vehicle sensor is used to obtain vehicle driving information; and send the vehicle driving information to the driver detection module; The multiple types of human body sensors are used to detect multiple types of status information of the driver; and send the multiple types of status information to the driver detection module; The in-car camera is arranged at the A-pillar and / or the rearview mirror of the vehicle; the in-car camera is used to collect the posture video and / or facial video of the driver, and send the collected posture video and / or facial video to the driver detection module; The driver detection module is used to perform calculations based on the received posture video and / or facial video and identify a preset personalized model corresponding to the driver; determine whether the driver is in an abnormal state based on the preset personalized model; and send a reminder message to the driver when it is determined that the driver is in an abnormal state.

2. The system according to claim 1, wherein: The vehicle sensor is used to obtain vehicle steering wheel angle, lane offset and vehicle speed information.

3. The system according to claim 1, wherein: The multiple types of human body sensors include: one or more of a wristband-type skin electrocardiogram sensor, a belt-type respiratory sensor, a wearable eye tracker, and a head-mounted EEG sensor; The wristband-type skin electrocardiogram sensor is used to obtain the driver's skin conductivity, skin electrocardiogram frequency, heart rate variability, heart rate acceleration, and ventricular beat interval; The belt-type respiratory sensor is used to obtain the driver's respiratory frequency and respiratory variability parameters; The wearable eye tracker is used to obtain the driver's blink duration, eyelid closure speed, pupil diameter change rate, blink frequency change, pupil dilation amplitude, gaze duration and scan path length; The head-mounted EEG sensor is used to obtain the driver's EEG signals, EMG signals, and EOG signals.

4. The system according to claim 1, wherein: The in-vehicle camera is specifically used to collect the posture video and / or the facial video, and send the collected posture video and / or the facial video to the driver detection module; The driver detection module specifically splits the multiple types of status information and extracts feature variables based on the splitting results; time-tags the extracted feature vectors to obtain first time series data; extracts human body key point coordinates and / or skeleton data based on the posture video; calculates the corresponding center point coordinates of the left hand, right hand, left arm, right arm, body, head, and shoulder based on the extracted human body key point coordinates and / or skeleton data; calculates the projection of the connection vector of each arm and the corresponding hand of the driver to the body midline vector based on the extracted human body key point coordinates and / or skeleton data. distance; calculating the angle between target vectors based on the extracted human body key point coordinates and / or skeleton data; time-marking the center point coordinates, the distance, and the angle to obtain second time series data; and / or, extracting facial key point coordinates based on the facial video; constructing a facial feature model of the driver based on the facial key point coordinates; calculating at least one of the driver's head turning angle, yawning frequency, and blinking frequency based on the facial feature model; time-marking at least one of the head turning angle, yawning frequency, and blinking frequency to obtain third time series data; Whether the driver is in an abnormal state is determined based on the first time series data, the second time series data and / or the third time series data, and a preset personalized model.

5. The system according to claim 1, wherein: The training process of the preset personalized model includes: Obtain multiple sets of initial data; By using principal component analysis dimensionality reduction method, the initial data is reduced into multiple groups of main feature data; Dividing the multiple sets of main feature data into training sets and test sets; The preset LSTM / Transformer model is trained using the training set and tested using the test set to obtain the preset personalized model that meets the preset test requirements.

6. The system according to claim 3, wherein: The driver detection module is specifically configured to send a reminder instruction to the wristband-type skin conduction electrocardiogram sensor when determining that the driver is in an abnormal state, so that the wristband-type skin conduction electrocardiogram sensor reminds the driver through vibration and / or sound.

7. The system according to claim 1, wherein: The driver detection module is specifically used to identify the preset personalized model corresponding to the driver based on the driving information, the state information, at least one of the posture video and / or facial video, and a personalized model library.

8. The system according to claim 4, wherein: The driver detection module specifically sends the first time series data, the second time series data and / or the third time series data to the back end to verify the confidence of the preset personalized model, and performs incremental training on the preset personalized model when the confidence does not meet the preset confidence requirement.

9. The system according to claim 8, characterized in that The incremental training process includes: The preset personalized model is used as the teacher model, and the model to be updated is used as the student model for training to obtain a trained model; The preset personalized model, the model to be updated and the trained model are evaluated in terms of weighted WF1, catastrophic forgetting rate and AUC-PR gain, and the model determined by the evaluation is added to the personalized model library.

10. A driver status recognition method, characterized in that: Applied to a driver status recognition system, the system includes: vehicle sensors, in-vehicle cameras, various types of human body sensors, and a driver detection module; The vehicle sensor is used to obtain vehicle driving information and send the vehicle driving information to the driver detection module; the multiple types of human body sensors are used to detect multiple types of driver status information and send the multiple types of status information to the driver detection module; the in-vehicle camera is set at the A-pillar and / or rearview mirror of the vehicle; the in-vehicle camera is used to capture the driver's posture video and / or facial video and send the captured information to the driver detection module; The method comprises: identifying, based on the received information, a preset personalized model corresponding to the driver, and determining, using the preset personalized model, whether the driver is in an abnormal state; and sending a reminder message to the driver when it is determined that the driver is in an abnormal state; The method of judging whether the driver is in an abnormal state based on the driving information, the state information, the posture video and / or the facial video, and the preset personalized model includes: splitting the multiple types of state information and extracting feature variables based on the splitting results; time-marking the extracted feature vectors to obtain first time series data; extracting human body key point coordinates and / or skeleton data based on the posture video; calculating the corresponding center point coordinates of the left hand, right hand, left arm, right arm, body, head, and shoulder based on the extracted human body key point coordinates and / or skeleton data; calculating the distance between the connection vector of each arm and the corresponding hand of the driver and the body midline vector based on the extracted human body key point coordinates and / or skeleton data; The extracted human body key point coordinates and / or skeleton data are used to calculate the angle between the target vectors; the center point coordinates, the distance and the angle are time-labeled to obtain second time series data; and / or, based on the facial video, the facial key point coordinates are extracted; based on the facial key point coordinates, a facial feature model of the driver is constructed; based on the facial feature model, at least one of the driver's head turning angle, yawning frequency, and blinking frequency is calculated; at least one of the head turning angle, yawning frequency, and blinking frequency is time-labeled to obtain third time series data; based on the first time series data, the second time series data and / or the third time series data, and a preset personalized model, it is determined whether the driver is in an abnormal state; The first time series data, the second time series data and / or the third time series data are sent to the back end to verify the confidence of the preset personalized model, and to perform incremental training on the preset personalized model when the confidence does not meet the preset confidence requirement, wherein the personalized model after incremental training is also used to update the preset personalized model after evolutionary evaluation.

Images