Driver emergency handling method and device

Abstract

A method and apparatus for managing an emergency situation are provided. The method may include obtaining image data representing one or more photographed images of a driver of a vehicle; determining that the driver meets an age threshold; determining emotion classification of the driver; determining a value associated with a heart rate of the driver, where the value associated with the heart rate corresponds to the image data; determining the presence of an emergency based on the driver satisfying an age threshold, based on the emotion classification, and based on the heart rate; outputting a first request of a user response from the driver; and send an emergency rescue request to an emergency dispatch service provider based on the user response not being received from the driver within a predetermined period of time after the first request is output.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority and benefit to Korean Patent Application No. 10-2024-0082335, filed on June 25, 2024, with the Korean Intellectual Property Office, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to a method and apparatus for vehicle emergency management, and more specifically, to a method and apparatus for managing emergency situations of drivers. Background Technology

[0004] When a driver encounters an emergency while operating a vehicle, such as shock, unconsciousness, or cardiac arrest, it can lead to a serious traffic accident. When a driver loses consciousness, he or she may lose control of the vehicle, increasing the risk of collision with other vehicles or pedestrians and potentially causing loss of life and property. Therefore, the need for emergency management during vehicle operation is crucial. Summary of the Invention

[0005] This disclosure attempts to provide a method and device for handling driver emergencies, which can quickly and accurately identify and handle driver emergencies that may occur during vehicle operation by using an onboard camera to detect the driver's state in a non-contact manner.

[0006] According to one or more example embodiments of this disclosure, a method performed by a vehicle device may include: acquiring image data representing one or more photographic images of the vehicle driver via a vehicle camera; determining that the driver meets an age threshold based on the image data having one or more predetermined image features; determining an emotion category of the driver based on facial expression analysis performed on the image data; determining a value associated with the driver's heart rate, wherein the value associated with the heart rate corresponds to the image data; determining the existence of an emergency based on the driver meeting the age threshold, the emotion category, and the heart rate; outputting a first request for a user response from the driver via a vehicle user interface and based on the existence of the emergency; and sending an emergency rescue request to an emergency dispatch service provider if no user response is received from the driver within a predetermined time period after the first request is output.

[0007] The method may further include: based on the existence of a second emergency, outputting a second request from a user response from the driver through the vehicle's user interface; based on receiving a user response to the second request, outputting a third request through the user interface, the third request being used to instruct the driver to agree to transfer control of the vehicle; and based on receiving the instruction of consent, transferring control of the vehicle to an entity interface different from the driver's.

[0008] This entity may include a remote server configured to remotely control vehicles.

[0009] The entity may include a computing device located in the vehicle and configured to control the vehicle to perform autonomous driving.

[0010] The method may further include: based on the absence of a user response from the driver within the predetermined time period, outputting a second request through the user interface, the second request instructing the driver to consent to the transfer of control of the vehicle; and based on the absence of the consent instruction within the predetermined time period, acquiring additional image data via the camera representing one or more additional photographic images of the driver, and confirming the existence of an emergency based on the additional image data.

[0011] One or more predetermined image features may be associated with at least one of the driver's hair or the driver's wrinkles. Determining that the driver meets an age threshold may include estimating the driver's age based on image data using a first model trained to recognize the presence of gray hair and wrinkles.

[0012] Determining an emotion classification may include: determining global features for multi-scale application from image data using multiple multi-scale blocks with filters of different sizes; determining local features for attention application from image data using a Convolutional Block Attention (CBAM) module, which includes a channel attention module and a spatial attention module, and applying the channel attention module and spatial attention module sequentially; inputting the global and local features into a Graph Convolutional Network (GCN) combiner to perform feature combination; and determining the driver's emotion classification using a classifier based on the result of the feature combination.

[0013] The method may further include: selecting a dominant feature by applying a feature selector to global and local features; and extracting a patch image of the face corresponding to the location of the dominant feature from the global features. Performing feature combination may include: inputting features obtained by magnifying the patch image and applying attention to the magnified patch image together with the global and local features into a GCN combiner.

[0014] Determining a driver's heart rate may include: obtaining a first-band image and a second-band image of different wavelengths from image data; measuring a first remote heartbeat signal from the first-band image; measuring a second remote heartbeat signal from the second-band image; determining a first mass fraction based on the first-band image; determining a second mass fraction based on the second-band image; determining a first effective heart rate interval based on the first remote heartbeat signal and the first mass fraction; determining a second effective heart rate interval based on the second remote heartbeat signal and the second mass fraction; and determining a complementary heart rate based on the first and second effective heart rate intervals.

[0015] Determining the first quality score and the second quality score may include at least one of the following: determining the first motion quality score and the second motion quality score; determining the first illumination quality score and the second illumination quality score; and determining the first signal quality score and the second signal quality score.

[0016] According to one or more example embodiments of this disclosure, an apparatus may include: one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the apparatus to: acquire image data representing one or more photographic images of a vehicle driver via a vehicle's camera; determine that the driver meets an age threshold based on the image data having one or more predetermined image features; determine the driver's emotion category based on facial expression analysis performed on the image data; determine a value associated with the driver's heart rate, wherein the value associated with the heart rate corresponds to the image data; determine the existence of an emergency based on the driver meeting the age threshold, the emotion category, and the heart rate; output a first request for a user response from the driver via a user interface of the vehicle based on the existence of the emergency; and send an emergency rescue request to an emergency dispatch service provider if no user response is received from the driver within a predetermined time period after the output of the first request.

[0017] When executed by the one or more processors, the instructions may further cause the device to: output a second request through a user interface based on receiving a user response from the driver within a predetermined time period, the second request instructing the driver to consent to the transfer of control of the vehicle, and, based on receiving the consent instruction, transfer control of the vehicle to an entity different from the driver.

[0018] This entity may include a remote server configured to remotely control vehicles.

[0019] The entity may include a computing device located in the vehicle and configured to control the vehicle to perform autonomous driving.

[0020] When executed by the one or more processors, the instructions may also cause the device to: output a second request via a user interface based on the absence of a user response from the driver within a predetermined time period, the second request instructing the driver to consent to the transfer of control of the vehicle; and, based on the absence of consent instruction within the predetermined time period, acquire additional image data via a camera representing one or more additional photographic images of the driver, and confirm the existence of an emergency based on the additional image data.

[0021] One or more predetermined image features may be associated with at least one of the driver's hair or the driver's wrinkles. When executed by the one or more processors, the instructions may cause the device to determine that the driver meets an age threshold by estimating the driver's age based on image data using a first model trained to recognize the presence of gray hair and wrinkles.

[0022] When executed by the one or more processors, the instructions enable the device to determine the emotion classification by: determining global features with applied multi-scales from the image data using multiple multi-scale blocks with filters of different sizes; determining local features with applied attention from the image data using a convolutional block attention module (CBAM), which includes a channel attention module and a spatial attention module, and applying the channel attention module and the spatial attention module sequentially; inputting the global and local features into a graph convolutional network (GCN) combiner to perform feature combination; and determining the driver's emotion classification by using a classifier based on the result of the feature combination.

[0023] When executed by the one or more processors, the instructions may also cause the apparatus to: select a dominant feature by applying a feature selector to global and local features; and extract a patch image of the face corresponding to the location of the dominant feature from the global features. When executed by the one or more processors, the instructions may cause the apparatus to perform feature combination by inputting features obtained by magnifying the patch image and applying attention to the magnified patch image, along with global and local features, into a GCN combiner.

[0024] When executed by the one or more processors, the instructions enable the device to determine the driver's heart rate by: obtaining a first band image and a second band image of different bands from image data; measuring a first remote heartbeat signal of the first band image; measuring a second remote heartbeat signal of the second band image; determining a first mass fraction based on the first band image; determining a second mass fraction based on the second band image; determining a first effective heart rate interval based on the first remote heartbeat signal and the first mass fraction; determining a second effective heart rate interval based on the second remote heartbeat signal and the second mass fraction; and determining a complementary heart rate based on the first effective heart rate interval and the second effective heart rate interval.

[0025] When executed by the one or more processors, the instructions can cause the device to determine a first quality score and a second quality score in the following ways: determining a first motion quality score and a second motion quality score; determining a first illumination quality score and a second motion illumination score; and determining a first signal quality score and a second signal quality score. Attached Figure Description

[0026] Figure 1 This is a schematic diagram illustrating a device for managing emergency situations for a driver, according to an exemplary embodiment.

[0027] Figure 2 This is a schematic diagram illustrating a method for managing a driver's emergency situation according to an exemplary embodiment.

[0028] Figure 3 This is a schematic diagram illustrating a method for managing a driver's emergency situation according to an exemplary embodiment.

[0029] Figure 4 This is a schematic diagram illustrating a method for managing a driver's emergency situation according to an exemplary embodiment.

[0030] Figure 5 This is a schematic diagram illustrating a method for managing a driver's emergency situation according to an exemplary embodiment.

[0031] Figure 6 This is a schematic diagram illustrating a method for managing a driver's emergency situation according to an exemplary embodiment.

[0032] Figure 7 This is a schematic diagram illustrating an example implementation of a device and method for managing a driver's emergency situation according to an exemplary embodiment.

[0033] Figure 8 This is a schematic diagram illustrating an example implementation of a device and method for managing a driver's emergency situation according to an exemplary embodiment.

[0034] Figure 9 This is a schematic diagram illustrating a computing device according to an exemplary embodiment. Detailed Implementation

[0035] The invention will now be described more fully with reference to the accompanying drawings, which illustrate exemplary embodiments of the invention. As those skilled in the art will recognize, the described exemplary embodiments can be modified in various ways without departing from the spirit or scope of this disclosure. Therefore, the drawings and description should be considered illustrative in nature, not restrictive. Throughout the specification, the same reference numerals denote the same elements.

[0036] Throughout the specification and claims, unless expressly stated to the contrary, the word "comprising" and variations thereof, such as "comprising" or "including," are to be understood as implying the inclusion of the stated element but not excluding any other element. Ordinal terms, such as "first" and "second," are used to describe various components, but these components are not limited by these terms. These terms are used only to distinguish one component from another.

[0037] The terms "component," "unit," and "module" used in this specification may refer to a unit capable of performing at least one function or operation described herein, which may be implemented in hardware or wiring, software, or a combination of hardware, wiring, and software. Furthermore, at least some configurations or functions of the device and method for managing driver emergencies according to the exemplary embodiments described below may be implemented as programs or software, and the programs or software may be stored on a computer-readable medium.

[0038] To manage driver emergencies, it may be necessary to quickly and accurately identify or determine a driver's unconscious or drowsy state. In some implementations, monitoring devices may be worn on the driver's body to monitor the wearer's state. However, if the monitoring device is not worn correctly by the driver, or if the driver intentionally chooses not to wear the monitoring device, the practicality of such a device may be limited, and it may be unable to detect emergencies.

[0039] According to the Society of Automotive Engineers (SAE), the automation levels of autonomous vehicles can be categorized as follows: Level 0, corresponding to "no automation," involves the autonomous driving system temporarily engaging in emergency situations (e.g., automatic emergency braking) and / or simply providing warnings (e.g., blind spot warning, lane departure warning, etc.), and expects the driver to operate the vehicle. Level 1, corresponding to "driver assistance," involves the system performing some driving functions (e.g., steering, acceleration, braking, lane centering, adaptive cruise control, etc.) while the driver operates the vehicle in normal driving conditions, and expects the driver to determine the system's operating status and / or timing, perform other driving functions, and respond to (e.g., resolve) emergency situations. Level 2, corresponding to "partial automation," involves the system performing steering, acceleration, and / or braking under driver supervision, and expects the driver to determine the system's operating status and / or timing, perform other driving functions, and respond to (e.g., resolve) emergency situations. At Level 3 of autonomous driving, the SAE classification standard can correspond to "conditional automation," in which the system drives the vehicle under limited conditions (e.g., performing driving functions such as steering, acceleration, and / or braking), but transfers driving control to the driver when the required conditions are not met. The driver is expected to determine the system's operating state and / or timing, and take over control in emergency situations, but not otherwise operate the vehicle (e.g., steering, acceleration, and / or braking). At Level 4 of autonomous driving, the SAE classification standard can correspond to "high automation," in which the system performs all driving functions, and the driver is expected to take over control of the vehicle only in emergency situations. At Level 5 of autonomous driving, the SAE classification standard can correspond to "full automation," in which the system performs all driving functions without any driver assistance (including in emergency situations), and the driver is expected to perform no driving functions other than determining the system's operating state. Although this disclosure applies the SAE classification standard to autonomous driving classification, other classification methods and / or algorithms can be used in one or more configurations described herein. One or more features associated with autonomous driving control can be activated based on one or more autonomous driving control settings configured (e.g., based on at least one of: autonomous driving classification, selection of vehicle autonomous driving level, etc.).

[0040] Based on one or more of the features described herein (e.g., determining the existence of an emergency), the operation of the vehicle can be controlled. Vehicle control may include various operational controls associated with the vehicle (e.g., automatic driving control, sensor control, braking control, braking time control, acceleration control, rate of change of acceleration control, alarm timing control, forward collision warning timing control, etc.).

[0041] For example, one or more auxiliary devices (e.g., engine brakes, exhaust brakes, hydraulic reducers, electric reducers, regenerative brakes, etc.) can also be controlled based on one or more features described herein (e.g., determining the existence of an emergency). For example, one or more communication devices (e.g., modems, network adapters, radio transceivers, antennas, etc., capable of communicating via one or more wired or wireless communication protocols, such as Ethernet, Wi-Fi, Near Field Communication (NFC), Bluetooth, Long Term Evolution (LTE), 5G New Radio (NR), Vehicle-to-Everything (V2X), etc.) can also be controlled based on one or more features described herein (e.g., determining the existence of an emergency).

[0042] For example, one or more minimum risk maneuvers (MRM) operations can be controlled based on one or more features described herein (e.g., determining the existence of an emergency). A minimum risk maneuver operation (e.g., minimum risk maneuver, minimum risk maneuver) can be a maneuver by which the vehicle minimizes (e.g., reduces) the risk of collision with surrounding vehicles to achieve a lower (e.g., minimum) risk state. A minimum risk operation can be an operation activated during autonomous driving when the driver is unable to respond to an intervention request. During a minimum risk maneuver, one or more processors in the vehicle can control the vehicle's driving operations for a set time period.

[0043] For example, one or more biased driving operations can be controlled based on one or more features described herein (e.g., determining the presence of an emergency). The drive control unit can perform biased driving control. To perform biased driving, the drive control unit can control the vehicle to travel within the lane by maintaining a lateral distance between the vehicle's center position and the lane center. For example, the drive control unit can control the vehicle to remain within the lane, but not in the center of the lane.

[0044] The driving control unit can identify the target lateral distance for deflection used in driving control. For example, the target lateral distance for deflection may include an intentionally adjusted lateral distance between the vehicle and a reference point (such as the lane center or another vehicle) that the vehicle may graphically maintain during maneuvers such as lane changes. This adjustment can be made to improve the vehicle's stability, safety, and / or performance under different driving conditions. For example, during lane changes, the driving control system may bias the lateral distance to maintain a safer clearance from adjacent vehicles, taking into account factors such as vehicle speed, road conditions, and / or the presence of obstacles.

[0045] For example, one or more sensors (e.g., IMU sensor, camera, lidar, radar, blind spot monitoring sensor, lane departure warning sensor, parking sensor, light sensor, rain sensor, traction control sensor, anti-lock braking system sensor, tire pressure monitoring sensor, seat belt sensor, airbag sensor, fuel sensor, emission sensor, throttle position sensor, inverter, converter, motor controller, power distribution unit, high-voltage wiring and connectors, auxiliary power module, charging interface, etc.) may also be used for control based on one or more features described herein (e.g., determining the existence of an emergency).

[0046] Operational control for autonomous driving of vehicles can include various driving controls of the vehicle by the vehicle control unit (e.g., acceleration, deceleration, steering control, gear shifting control, braking system control, traction control, stability control, cruise control, lane keeping assist control, collision avoidance system control, emergency braking assist control, traffic sign recognition control, adaptive headlight control, etc.).

[0047] Figure 1 This is a schematic diagram illustrating a device for managing emergency situations for a driver, according to an exemplary embodiment.

[0048] refer to Figure 1 According to an exemplary embodiment, the device 10 for managing driver emergencies can execute program code loaded into one or more memory devices via one or more processors. For example, the device 10 for managing driver emergencies can be implemented as a computing device 50, as referred to later. Figure 9 As described. In this case, one or more processors may correspond to processor 510 of computing device 50, and one or more memory devices may correspond to memory 520 of computing device 50. Program code may be executed by one or more processors to perform functions that identify driver situations occurring in the vehicle and manage emergency situations. The term "module" is used herein to logically distinguish these functions performed by program code.

[0049] An apparatus 10 for managing emergency situations for a driver, according to an exemplary embodiment, may include an image data acquisition module 110, an elderly person determination module 120, an emergency situation determination module 130, and an emergency handling module 140. Each module or component of the apparatus 10 may be implemented in software, hardware, or a combination of both. One of the plurality of modules or components of the apparatus 10 may be implemented in one or more processors.

[0050] The image data acquisition module 110 can capture images of the driver using a camera installed inside the vehicle and acquire first image data. Herein, the first image data refers to data relating to an image including a facial region for performing facial expression recognition of the driver and a skin region for performing heart rate calculation, and can be in the form of a still image or video comprising multiple frames.

[0051] The elderly person determination module 120 can remove noise from the first image data acquired by the image data acquisition module 110 to obtain second image data. Specifically, the elderly person determination module 120 can apply some recovery algorithms to the first image data to compensate for jitter, or apply some optical correction techniques to the first image data to remove optical noise, so as to obtain second image data with jitter and light noise removed.

[0052] The elderly person determination module 120 can determine whether a driver is elderly (e.g., whether the driver meets an age requirement) based on identifiable points of an elderly person in the second image data (e.g., one or more predetermined image features). In some exemplary embodiments, identifiable points of an elderly person may include hair regions and wrinkle regions. The elderly person determination module 120 can predict whether a driver is elderly from the second image data using a first model trained to predict (e.g., identify) the presence of gray hair and wrinkles from these points. In some exemplary embodiments, the first model may include a convolutional neural network (CNN) model, but the scope of the invention is not limited thereto.

[0053] The emergency determination module 130 can use facial expression recognition of the second image data to perform driver emotion classification.

[0054] The emergency determination module 130 can extract global features applied to it from the second image data through a multi-scale module comprising multiple multi-scale blocks of filters with different sizes. Here, the global features can be features extracted from the entire facial region (or first-level features). The multi-scale module can capture spatial context from the image by using filters of multiple sizes, not limited to filters of a single size. The multi-scale module can include multiple multi-scale blocks of filters with different sizes, and each multi-scale block is capable of extracting features of a different size for the input data. For example, the multi-scale module can include first to fourth multi-scale blocks, and the first and second multi-scale blocks are implemented with 3×3 convolutions using 256 filters, while the third and fourth multi-scale blocks are implemented with 3×3 convolutions using 512 filters. Global features can be extracted from the first to fourth multi-scale blocks.

[0055] The emergency determination module 130 can extract local features from the second image data to which a Convolutional Block Attention (CBAM) module has applied attention. These local features can be features (or second-level features) extracted from partial regions of the face. The CBAM can include two types of attention mechanisms: a channel attention module and a spatial attention module, and these can be applied sequentially. In other words, the CBAM can first apply channel attention, which learns the importance of each channel and adjusts the activation of each channel for each channel, and then apply spatial attention, which learns the importance of each region of the image and adjusts the activation of each location based on the application of channel attention. By increasing attention to existing convolutional layers in this way, the neural network can better focus on important parts of the input image and improve the performance of the convolutional neural network. For example, facial regions can include, for instance, the region LE including the left eye, the region RE including the right eye, the region NO including the nose, the region LM including the left part of the mouth, and the region RM including the right part of the mouth as local regions. Local features can be extracted using the first to fourth CBAMs, which take the region LE (left eye), RE (right eye), NO (nose), LM (left part of mouth), and RM (right part of mouth) as input. The first to fourth CBAMs are sequential, and each of them can be implemented as a 3×3 convolution with 256 filters.

[0056] The emergency situation determination module 130 can input global and local features into a graph convolutional network (GCN) combiner to perform feature combination, and can perform driver emotion classification through a classifier based on the combined features. Specifically, the emergency situation determination module 130 can construct a graph with nodes, each node including feature vectors representing the relationships between nodes and edges, combine the features of each node with the features of its neighboring nodes, and generate a new feature representation of the central node based on the features of its neighboring nodes. Therefore, the features of the nodes in the graph and the relationships between features can be learned. The global features combined by the graph convolutional network combiner and the local features combined by the graph convolutional network combiner can be combined again by the graph convolutional network combiner to generate the final features.

[0057] In some exemplary embodiments, the emergency determination module 130 may apply a feature selector to global and local features to select dominant features. For example, the emergency determination module 130 may primarily select features corresponding to the top specific percentage of global features with high classification reliability values, and use features corresponding to the bottom specific percentage of global features determined to have low classification reliability values ​​as mean squared error (MSE) loss. For example, a predetermined number of 12 features may be primarily selected from global features, and among the selected features, features corresponding to the bottom 25% of features determined to have low classification reliability can be used as MSE loss. On the other hand, the emergency determination module 130 may primarily select a predetermined number of features from local features, and secondarily select features corresponding to the top specific percentage of selected features determined to have high classification reliability. For the secondarily selected features, a graph convolutional network combiner may be used for feature combination. For example, a predetermined number of 12 features may be primarily selected from local features, and among the selected local features, the top 25% of features determined to have high classification reliability may be secondarily selected. Furthermore, among the selected local features, the bottom 25% of features identified as having low classification reliability can be used as the MSE loss.

[0058] The emergency determination module 130 can extract patch images of the face corresponding to the locations of dominant features in the global features. These patch images can be used to extract features extracted from fine regions of the face from an image corresponding to the entire facial region. Therefore, the number of patch regions can be set to be greater than the number of local regions, as the patch regions are intended to take into account fine regions of the face. The emergency determination module 130 can perform feature combination by inputting features obtained by magnifying the patch image and applying attention to the magnified patch image, along with global and local features, into a graph convolutional network combiner. The global features combined by the graph convolutional network combiner, the local features combined by the graph convolutional network combiner, and the features obtained by magnifying the patch image and applying attention to the magnified patch image can be combined again by the graph convolutional network combiner to generate the final features.

[0059] The emergency situation determination module 130 can classify the driver's emotions into one of anger, disgust, fear, happiness, neutrality, sadness, and surprise by inputting the final features combined by the graph convolutional network combiner into a classifier.

[0060] Simultaneously, the emergency determination module 130 can calculate the driver's heart rate based on the second image data. The emergency determination module 130 can acquire first-band and second-band images of different wavelengths from the second image data, and can respectively measure a first remote heartbeat signal of the first-band image and a second remote heartbeat signal of the second-band image. For example, the first-band image may include a visible light image, and the second-band image may include an infrared image. Of course, the scope of this disclosure is not limited thereto; the emergency determination module 130 can acquire images corresponding to any first frequency band, not necessarily limited to the visible light band, and images having any second frequency band different from the first frequency band but not necessarily limited to the infrared band. The emergency determination module 130 can acquire the average brightness value of the skin region of the facial area acquired from the first-band and second-band images, perform signal processing preprocessing including trend line removal and bandpass filtering, and extract the remote heartbeat signal using an algorithm for extracting the heartbeat signal. In some exemplary embodiments, the algorithm for extracting the heartbeat signal may include at least one of the following: a chromaticity-based method (Chrom), optical noise injection technique (ONIT), principal component analysis (PCA), a plane orthogonal to skin color (POS), a green method, and distance PPG. The emergency determination module 130 can measure the remote heart rate by performing frequency analysis on the extracted remote heartbeat signal and calculating the maximum frequency component.

[0061] The emergency situation determination module 130 can calculate a first quality score and a second quality score for the first band image and the second band image, respectively, and select a first effective heart rate interval based on the first remote heartbeat signal and the first quality score. Furthermore, the emergency situation determination module 130 can select a second effective heart rate interval based on a second remote heartbeat signal and the second quality score. The quality score can be used to select an effective heart rate interval determined as a suitable interval for analysis from the remote heartbeat signals measured in the acquired images, thereby improving the reliability of heart rate acquisition.

[0062] In some exemplary embodiments, the first quality score may include a first motion quality score as an indicator for measuring noise caused by driver head movements, facial muscle movements due to dialogue or facial expressions, etc. For example, the first motion quality score of a first band image can be calculated by measuring changes in facial feature points in adjacent frames, and motion intensity can be measured using the first motion quality score. That is, changes in facial feature points in adjacent frames can be measured to quantitatively measure facial movements caused by facial expressions and dialogue, which are noise factors in remote heart rate measurements. In some exemplary embodiments, the second quality score may include a second motion quality score calculated for a second band image in substantially the same manner as the first motion quality score.

[0063] In some exemplary embodiments, the first quality score may include a first illumination quality score as an indicator for measuring noise caused by illumination variations. For example, the first illumination quality score can be calculated by measuring the amount of brightness variation in a first band image, i.e., measuring the time-series illumination difference. For example, the input brightness information can be extracted from the Y values ​​in the YCbCr color space of an RGB camera, or from a single-channel image of an NIR camera, and the extracted brightness information can be used as a quality score to quantitatively provide the validity of the extracted heartbeat signal. In some exemplary embodiments, the second quality score may include a second illumination quality score calculated for a second band image in substantially the same manner as the first illumination quality score.

[0064] In some exemplary embodiments, the first quality score may include a first signal quality score, which measures the quality of the heartbeat signal using frequency analysis and the SNR exponent of the remote heartbeat signal. For example, the first signal quality score can be calculated by measuring the signal quality based on the spectral characteristics of the remote heartbeat signal from a first band image. A heartbeat cycle is characterized by a gradual change in BPM over a time series; for example, the BPM change over a 10-second time series might be 6 BPM or less. This indicates that the intensity of the frequency component corresponding to the heartbeat signal is higher than other frequency components, which may indicate that the frequency power of the heart rate band is high in the spectrum. Therefore, the quality of the remote heartbeat signal can be evaluated by treating the remote heart rate bandwidth as a signal and the remaining bandwidth as noise. In some exemplary embodiments, the second quality score may include a second signal quality score calculated for the second band image in substantially the same manner as the first signal quality score.

[0065] The emergency situation determination module 130 can predict error values ​​based on a model trained with a first quality score and a second quality score, and select a first effective heart rate interval and a second effective heart rate interval based on the predicted error values. Specifically, intervals with predicted error values ​​equal to or less than a predetermined first threshold can be included in the first effective heart rate interval and used to calculate heart rate, while intervals with predicted error values ​​exceeding the predetermined first threshold may be excluded from the first effective heart rate interval and not used to calculate heart rate. The first threshold can be determined as an error value corresponding to the top x% (where x is a positive real number) of the minimum error value of the prediction result of the model trained with the first quality score. For example, the first threshold can be an error value corresponding to the top 10% of the minimum error value of the prediction result of the model trained with the first quality score, and a heartbeat signal with the top 10% reliability can be selected as the first effective heart rate interval. The second effective heart rate interval can be selected in the same manner as the first effective heart rate interval.

[0066] Based on the selected first and second effective heart rate intervals, the emergency determination module 130 can calculate a complementary heart rate as the driver's heart rate. As used herein, a complementary heart rate can refer to a heart rate whose accuracy is improved by considering the mutual characteristics of remote heartbeat signals from different wavelength bands. Therefore, compared to calculating the heart rate for a single frequency band, by using two frequency bands in a complementary combination, intervals whose performance is degraded by noise in each frequency band can be replaced with intervals based on another frequency band with good performance, and the time width for extracting the heart rate can be extended by using these two frequency bands complementaryly. When a portion of the first and second effective heart rate intervals overlaps and the heart rate predicted from the overlapping intervals is different, the emergency determination module 130 can calculate the complementary heart rate by using predetermined weights.

[0067] The emergency situation determination module 130 can determine whether the driver is in an emergency based on the results of emotion classification and heart rate. Specifically, the emergency situation determination module 130 can determine whether the driver is in an emergency based on the results of emotion classification and a predetermined standard for heart rate.

[0068] When an emergency situation is determined to be in the driver's possession, the emergency handling module 140 may output a first request for the driver's response. If no response to the first request is received from the driver within a predetermined time period, the emergency handling module 140 may send an emergency rescue request to an external system. In some exemplary embodiments, the external system may include a system providing a call center or a system providing an ambulance. The external system may be, for example, an emergency dispatch service provider. The first request may be output through the vehicle's user interface. The user interface may be a device through which a human user can interact with a device (e.g., the vehicle). The user interface may be located inside the vehicle, for example, in the dashboard, console, center console, instrument panel, side mirrors, rearview mirrors, steering wheel, car seats, glove box, armrests, headrests, interior walls, ceiling, etc. The user interface may include input interfaces that can receive input from a human user and / or output interfaces through which data or information can be output to a human user. Input interfaces may include, for example, buttons, knobs, toggle switches, switches, dials, sliders, keyboards, touchscreens, microphones, cameras, wheels, pedals, levers, etc. Output interfaces may include, for example, lights, lamps, indicators, screens, displays, consoles, gauges, meters, speakers, actuators (e.g., for tactile feedback or haptic feedback), etc. A first request may be output through one or more output interfaces of the vehicle. The first request may be visual, auditory, and / or output via tactile feedback (e.g., haptic feedback).

[0069] When the driver responds to the first request within a predetermined time period, the emergency handling module 140 can output a second request to the driver, requesting a response regarding whether the driver agrees to the transfer of control of the vehicle. The driver's response can be received through one of the input interfaces discussed herein. If the driver's response indicates that the driver agrees to the transfer of control of the vehicle, the emergency handling module 140 can perform an operation to transfer control of the vehicle to another entity. On the other hand, if the driver's response indicates that the driver does not agree to the transfer of control of the vehicle, the process of using a camera to capture images of the driver and determine whether the driver is in an emergency can be repeated (e.g., continuously).

[0070] The emergency handling module 140 can transfer control to a remote server that can remotely control the vehicle. For example, control can be transferred to a remote server outside the vehicle to perform vehicle control in an emergency.

[0071] In some exemplary embodiments, the emergency handling module 140 may transfer control to an onboard computing device capable of controlling the vehicle to perform autonomous driving. For example, the transfer of control may enable the vehicle to autonomously perform vehicle control.

[0072] According to this exemplary embodiment, by recognizing the driver's facial expressions, an emergency situation of the driver can be quickly detected in a non-contact manner through emotion classification and heart rate calculation, and appropriate management methods for the emergency situation, such as transferring control, can be provided based on the driver's consciousness or cognitive state.

[0073] Figure 2 This is a schematic diagram illustrating a method for managing a driver's emergency situation according to an exemplary embodiment.

[0074] refer to Figure 2 A method for managing a driver's emergency situation according to an exemplary embodiment may include: capturing the driver and acquiring first image data by using a camera installed inside the vehicle (S201); removing noise from the first image data to obtain second image data, and determining whether the driver is elderly based on identifiable points of elderly people in the second image data (S202); performing emotion classification on the driver by facial expression recognition of the second image information (S203); calculating the driver's heart rate based on the second image data (S204); determining whether the driver is in an emergency situation based on the result of the emotion classification and the heart rate when it is determined that the driver is in an emergency situation (S205); outputting a request for the driver to respond (S206); and sending an emergency rescue request (e.g., a distress signal) to an external system when the driver does not respond to the request within a predetermined time period (S207).

[0075] For further details of the method, please refer to the description of the exemplary embodiments described herein; therefore, repeated descriptions are omitted herein.

[0076] Figure 3 This is a schematic diagram illustrating a method for managing a driver's emergency situation according to an exemplary embodiment.

[0077] refer to Figure 3 According to an exemplary embodiment, a method for managing a driver's emergency may include: capturing the driver using a camera installed inside the vehicle (S301); removing image noise from the captured image (S302); detecting facial principal points (e.g., hair color and skin wrinkles) in the noise-removed image (S303); and performing an elderly detection algorithm to determine whether the driver is elderly (S304).

[0078] When it is determined that the driver is an elderly person (S304, "Yes"), the method can perform the step of sending the preprocessed image of the elderly person to the next task (S305). See later. Figure 4 Describe the next task.

[0079] Figure 4 This is a schematic diagram illustrating a method for managing a driver's emergency situation according to an exemplary embodiment.

[0080] refer to Figure 4 A method for managing a driver's emergency according to an exemplary embodiment may include: obtaining information based on... Figure 3 Operation 305 sends an image of an elderly person (S401); detects facial regions in the image of an elderly person (S402); detects facial features (e.g., eyes, nose, and mouth) in the facial regions (S403); aligns the face based on the facial features by rotation, movement, zoom, etc. (S404); and extracts feature values ​​from the aligned face and performs a facial expression recognition algorithm (S405).

[0081] In some exemplary embodiments, operation S405 may include receiving input of a face based on facial feature alignment (S4051), extracting global features (S4052), extracting local features (S4053), identifying micro-patterns (S4054), and performing emotion classification by using a facial expression classifier (S4055).

[0082] This method can use a facial expression classifier to determine whether the driver has expressed a negative emotion as a result of emotion classification. When it is determined that the driver has expressed a negative emotion (S406, "Yes"), the method can send incomplete situation information to the next task (S407). See below for further details. Figure 5Describe the next task. Conversely, if it is determined that the driver is not in a negative situation (S406, "No"), the method can proceed to operation S405.

[0083] Figure 5 This is a schematic diagram illustrating a method for managing a driver's emergency situation according to an exemplary embodiment.

[0084] refer to Figure 5 A method for managing a driver's emergency according to an exemplary embodiment may include: obtaining information based on... Figure 3 The operation S305 sends an image of an elderly person (S501); detects a skin region from the elderly person's image from which heart rate parameters can be extracted (S502); extracts heart rate-related signals from the skin region and performs noise removal (S503); executes a heart rate estimation algorithm (S504); calculates the heart rate using the heart rate estimation algorithm (S505); and sends the heart rate information as an indicator of unstable conditions (S506).

[0085] Figure 6 This is a schematic diagram illustrating a method for managing a driver's emergency situation according to an exemplary embodiment.

[0086] refer to Figure 6 According to an exemplary embodiment, a method for managing a driver's emergency may include: capturing the driver's image using a camera installed inside the vehicle (S601); and performing elderly person recognition on the captured image (S602).

[0087] If elderly person identification fails in the captured image (S602, "No"), the method can proceed to operation S601 to continue monitoring via the camera. Conversely, if elderly person identification is successful in the captured image (S602, "Yes"), the method can perform facial expression recognition of the elderly person and rPPG detection (S603), and determine whether the driver is in an unstable situation during driving based on the detected facial expression and rPPG (S604).

[0088] When it is determined that the driver is not in an unstable situation (S604, "No"), the method may proceed to operation S601 to continue monitoring via the camera. Alternatively, when it is determined that the driver is in an unstable situation (S604, "Yes"), the method may perform a determination of the transfer of control and request a response from the driver (S605), and determine when a driver consent response occurred (S606).

[0089] When it is determined that no driver consent response has occurred (S606, "No"), the method can identify that the driver is in a dangerous situation and, upon identification of the dangerous situation, initiate a connection with a call center or ambulance (S609). Conversely, when it is determined that a driver consent response has occurred (S606, "Yes"), the method can proceed to determine whether the driver consents to the transfer of control (S607). When it is determined that the driver does not consent to the transfer of control (S607, "No"), the method can proceed to operation S601 to continue monitoring via the camera. Conversely, when it is determined that the driver consents to the transfer of control (S607, "Yes"), the method can execute the transfer of control (S608).

[0090] Figure 7 This is a schematic diagram illustrating an example implementation of a device and method for managing a driver's emergency situation according to an exemplary embodiment.

[0091] refer to Figure 7 Global features applied at multiple scales can be extracted using a multi-scale module, and local features with attention applied can be extracted using CBAM. By applying a feature selector to both global and local features, the dominant features can be selected. Furthermore, the global features combined by the graph convolutional network combiner and the local features combined by the graph convolutional network combiner can be combined again by the graph convolutional network combiner to generate the final features.

[0092] Simultaneously, a feature selector extracts facial patch images corresponding to the locations of dominant features in the global features. Features obtained by magnifying the patch images and applying attention to them, along with the global and local features, are then input into a graph convolutional network combiner to perform feature combination. The global features combined by the graph convolutional network combiner, the local features combined by the graph convolutional network combiner, and the features obtained by magnifying the patch images and applying attention to them can be combined again by the graph convolutional network combiner to generate the final features.

[0093] The final features combined by the graph convolutional network combiner are fed into a classifier to classify the driver’s emotions into one of the following: anger, disgust, fear, happiness, neutrality, sadness, and surprise.

[0094] Figure 8 This is a schematic diagram illustrating an example implementation of a device and method for managing a driver's emergency situation according to an exemplary embodiment.

[0095] refer to Figure 8In operation S801, the emergency determination module 130 can acquire a visible light image and in operation S803, measure a remote heartbeat signal based on visible light. Simultaneously, in operations S805, S807, and S809, the emergency determination module 130 can calculate the motion quality score, the illumination quality score, and the signal quality score, respectively. Then, the emergency determination module 130 can select an effective heart rate zone based on the quality scores calculated in operation S811 and assess the reliability of the heartbeat signal.

[0096] Simultaneously, the emergency determination module 130 can acquire an infrared image in operation S821 and measure an infrared-based remote heartbeat signal in operation S823. Furthermore, in operations S825, S827, and S829, the emergency determination module 130 can calculate the motion quality score, the illumination quality score, and the signal quality score, respectively. Then, the emergency determination module 130 can select an effective heart rate zone based on the quality scores calculated in operation S831 and assess the reliability of the heartbeat signal.

[0097] Then, in operation S841, the emergency determination module 130 can calculate the complementary heart rate based on the reliability of visible light and infrared signals.

[0098] Figure 9 This is a schematic diagram illustrating a computing device according to an exemplary embodiment.

[0099] Now for reference Figure 9 The method and apparatus for managing driver emergencies according to the exemplary embodiment can be implemented using computing device 50.

[0100] The computing device 50 may include at least one of a processor 510, a memory 530, a user interface input device 540, a user interface output device 550, and a storage device 560, which communicate via a bus 520. The computing device 50 may also include a network interface 570 electrically connected to the network 40. The network interface 570 can send or receive signals with another entity via the network 40.

[0101] Processor 510 can be implemented in various types, such as a microcontroller unit (MCU), application processor (AP), central processing unit (CPU), graphics processing unit (GPU), neutral processing unit (NPU), and quantum processing unit (QPU), and can be a predetermined semiconductor device that executes commands stored in memory 530 or storage device 560. Processor 510 can be configured to implement the above-mentioned references. Figures 1 to 8 Describe the functions and methods.

[0102] The memory 530 and storage device 560 may include various forms of volatile or non-volatile storage media. For example, the memory may include read-only memory (ROM) 531 and random access memory (RAM) 532. In some exemplary embodiments, the memory 530 may be located inside or outside the processor 510, and the memory 530 may be connected to the processor 510 in various known ways.

[0103] In some exemplary embodiments, at least some configurations or functions of the method and apparatus for managing a driver's emergency according to the exemplary embodiments can be implemented as programs or software that execute on computing device 50, and the programs or software can be stored on a computer-readable medium. Specifically, the computer-readable medium according to the exemplary embodiments can record programs for performing operations included in the implementation of the method and apparatus for managing a driver's emergency according to the exemplary embodiments on a computer including a processor 510 that executes programs or commands stored in memory 530 or storage device 560.

[0104] In some exemplary embodiments, at least some configurations or functions of the method and device for managing driver emergencies according to the exemplary embodiments may be implemented using the hardware or circuitry of the computing device 50, or may be implemented as separate hardware or circuitry that may be electrically connected to the computing device 50.

[0105] An exemplary embodiment of this disclosure provides a method for handling a driver emergency. The method identifies a driver situation occurring in a vehicle and handles the emergency. The method includes: capturing a driver's image and acquiring first image data using a camera installed inside the vehicle; removing noise from the first image data to obtain second image data, and determining whether the driver is elderly based on identifiable points of an elderly person in the second image; performing emotion classification on the driver by facial expression recognition on the second image data; calculating the driver's heart rate based on the second image data; determining whether the driver is in an emergency based on the emotion classification result and the heart rate; when it is determined that the driver is in an emergency, outputting a first request for the driver to respond; and when the driver does not respond to the first request within a predetermined time period, sending an emergency rescue request to an external system.

[0106] In some exemplary embodiments, the method may further include: when the driver responds to the first request within a predetermined time period, outputting a second request requesting the driver to respond by instructing the driver whether he agrees to transfer control of the vehicle; and when a response instructing the driver to agree to transfer control of the vehicle is received, transferring control of the vehicle to another entity.

[0107] In some exemplary embodiments, transferring control of a vehicle to another entity may include transferring control of the vehicle to a remote server capable of remotely controlling the vehicle.

[0108] In some exemplary embodiments, transferring control of a vehicle to another entity may include transferring control to an onboard computing device capable of controlling the vehicle to perform autonomous driving.

[0109] In some exemplary embodiments, the method may further include: when a response indicating that the driver does not agree to the transfer of control of the vehicle occurs, repeatedly filming the driver using a camera and determining whether the driver is in an emergency.

[0110] In some exemplary embodiments, identifiable points of an elderly person may include hair regions and wrinkle regions, and determining whether a driver is an elderly person may include predicting whether a driver is an elderly person from second image data using a first model trained to predict the presence of gray hair and wrinkles from that point.

[0111] In some exemplary embodiments, performing emotion classification on a driver may include: extracting global features with applied multiscales from second image data via a multiscale module comprising multiple multiscale blocks of filters with different sizes; extracting local features with applied attention from the second image data via a convolutional block attention module (CBAM), which includes a channel attention module and a spatial attention module, and sequentially applying the channel attention module and the spatial attention module; inputting the global and local features into a graph convolutional network (GCN) combiner to perform feature combination; and performing emotion classification on the driver by using a classifier based on the combined features.

[0112] In some exemplary embodiments, the method may further include: selecting a dominant feature by applying a feature selector to global features and local features; and extracting a patch image of the face corresponding to the location of the dominant feature from the global features, wherein performing feature combination may include: performing feature combination by inputting features obtained by magnifying the patch image and applying attention to the magnified patch image together with the global features and local features into a graph convolutional network combiner.

[0113] In some exemplary embodiments, the calculation of the driver's heart rate may include: acquiring first band images and second band images of different bands from second image data; measuring a first remote heartbeat signal and a second remote heartbeat signal of the first band image and the second band image, respectively; calculating a first quality score and a second quality score of the first band image and the second band image, respectively; selecting a first effective heart rate interval based on the first remote heartbeat signal and the first quality score; selecting a second effective heart rate interval based on the second remote heartbeat signal and the second quality score; and calculating a complementary heart rate based on the first effective heart rate interval and the second effective heart rate interval.

[0114] In some exemplary embodiments, the calculation of the first quality score and the second quality score may include: calculating at least one of the first motion quality score and / or the second motion quality score; calculating the first lighting quality score and the second lighting quality score; and calculating the first signal quality score and the second signal quality score.

[0115] Another exemplary embodiment of this disclosure provides an apparatus for handling driver emergencies. The apparatus identifies driver situations occurring in a vehicle and handles emergencies. The apparatus executes program code loaded in one or more memory devices via one or more processors. The program code executes to capture a driver and acquire first image data using a camera installed inside the vehicle, removes noise from the first image data to obtain second image data, determines whether the driver is elderly based on identifiable points of elderly people in the second image data, performs emotion classification on the driver by facial expression recognition of the second image data, calculates the driver's heart rate in relation to the second image data, determines whether the driver is in an emergency based on the results of emotion classification and heart rate, outputs a first request for the driver to respond when the driver is determined to be in an emergency, and sends an emergency rescue request to an external system if there is no response to the driver's first request within a predetermined time period.

[0116] In some exemplary embodiments, program code may be executed to further output a second request when a driver responds to the first request within a predetermined time period, requesting the driver to respond by indicating whether the driver agrees to transfer control of the vehicle, and when a response indicating that the driver agrees to transfer control of the vehicle occurs, to transfer control of the vehicle to another entity.

[0117] In some exemplary embodiments, transferring control of a vehicle to another entity may include transferring control of the vehicle to a remote server capable of remotely controlling the vehicle.

[0118] In some exemplary embodiments, transferring control of a vehicle to another entity may include transferring control to an onboard computing device capable of controlling the vehicle to perform autonomous driving.

[0119] In some exemplary embodiments, program code may be executed to further, upon the occurrence of a response indicating that the driver does not agree to the transfer of control of the vehicle, repeatedly photograph the driver using a camera and determine whether the driver is in an emergency.

[0120] In some exemplary embodiments, identifiable points of an elderly person may include hair regions and wrinkle regions, and determining whether a driver is an elderly person may include predicting whether a driver is an elderly person from second image data using a first model trained to predict the presence of gray hair and wrinkles from that point.

[0121] In some exemplary embodiments, performing driver emotion classification may include: extracting multi-scale global features from second image data using a multi-scale module comprising multiple multi-scale blocks with filters of different sizes; extracting attention-applied local features from the second image data using a convolutional block attention module (CBAM), which includes a channel attention module and a spatial attention module, and sequentially applying the channel attention module and the spatial attention module; inputting the global and local features into a graph convolutional network (GCN) combiner to perform feature combination; and performing driver emotion classification using a classifier based on the combined features.

[0122] In some exemplary embodiments, program code may be executed to further select dominant features by applying a feature selector to global and local features, and to extract a patch image of the face corresponding to the location of the dominant feature from the global features. The execution of feature combination may include performing feature combination by inputting features obtained by magnifying the patch image and applying attention to the magnified patch image together with global and local features into a graph convolutional network combiner.

[0123] In some exemplary embodiments, the calculation of the driver's heart rate may include: acquiring first band images and second band images of different bands from second image data; measuring a first remote heartbeat signal and a second remote heartbeat signal of the first band image and the second band image, respectively; calculating a first quality score and a second quality score of the first band image and the second band image, respectively; selecting a first effective heart rate interval based on the first remote heartbeat signal and the first quality score; selecting a second effective heart rate interval based on the second remote heartbeat signal and the second quality score; and calculating a complementary heart rate based on the first effective heart rate interval and the second effective heart rate interval.

[0124] In some exemplary embodiments, the calculation of the first quality score and the second quality score may include: calculating at least one of the first motion quality score and / or the second motion quality score; calculating the first lighting quality score and the second lighting quality score; and calculating the first signal quality score and the second signal quality score.

[0125] According to an exemplary embodiment, an emergency situation involving a driver can be detected quickly in a non-contact manner, and a suitable handling method for the emergency situation can be provided based on the driver's consciousness or cognitive state.

[0126] Although the exemplary embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, but also includes various modifications and improvements made by those skilled in the art using the basic concepts of the present invention as defined in the appended claims.

Claims

1. A method performed by a device of a vehicle, the method comprising the following steps: Image data representing one or more photographic images of the vehicle's driver is obtained through the vehicle's camera; Based on image data with one or more predetermined image features, determine whether the driver meets the age threshold; Facial expression analysis is performed based on the image data to determine the driver's emotion category; Determine a value associated with the driver's heart rate, wherein the value associated with the heart rate corresponds to the image data; The existence of an emergency is determined based on whether the driver meets an age threshold, based on emotion classification, and based on heart rate. Based on the existence of the emergency situation, and through the vehicle's user interface, a first request for a user response from the driver is output; and If no response is received from the driver within a predetermined time period after the first request is sent, an emergency rescue request is sent to the emergency dispatch service provider.

2. The method according to claim 1, further comprising the following step: Based on the existence of a second emergency, a second request for a user response from the driver is output through the vehicle's user interface; Based on the user response to the second request received, a third request is output through the user interface, the third request being used to instruct the driver to agree to the transfer of control of the vehicle; as well as Based on the instruction received with the consent, control of the vehicle is transferred to an entity different from the driver.

3. The method according to claim 2, wherein, The entity includes a remote server configured to remotely control the vehicle.

4. The method according to claim 2, wherein, The entity includes a computing device located in the vehicle, the computing device being configured to control the vehicle to perform autonomous driving.

5. The method according to claim 1, further comprising the following step: If no user response is received from the driver within the predetermined time period, a second request is output through the user interface, the second request instructing the driver to agree to transfer control of the vehicle. as well as Based on the absence of the instruction to consent within the predetermined time period, additional image data representing one or more additional photographic images of the driver is obtained through the camera, and the existence of an emergency is confirmed based on the additional image data.

6. The method according to claim 1, wherein, The one or more predetermined image features are associated with at least one of the driver's hair or the driver's wrinkles, and The steps for determining whether a driver meets the age threshold include: The driver's age is estimated based on the image data using a first model trained to recognize the presence of gray hair and wrinkles.

7. The method according to claim 1, wherein, The steps to determine the emotion category include: Based on the image data, global features applying the multi-scale method are determined from the image data using multiple multi-scale blocks with filters of different sizes. The Convolutional Block Attention (CBAM) module is used to determine local features from the image data to which attention is to be applied. The CBAM module includes a channel attention module and a spatial attention module, and the channel attention module and the spatial attention module are applied sequentially. The global and local features are input into the Graph Convolutional Network (GCN) combiner to perform feature combination; and Based on the result of the feature combination, the driver's emotion category is determined by using a classifier.

8. The method according to claim 7, further comprising the following step: Dominant features are selected by applying feature selectors to the global features and the local features; and Extract facial patch images corresponding to the locations of the dominant features from the global features. The steps for performing feature combination include: Feature combination is performed by inputting the features obtained by magnifying the patch image and applying attention to the magnified patch image together with the global features and the local features into the graph convolutional network (GCN) combiner.

9. The method according to claim 1, wherein, The steps to determine the driver's heart rate include: First-band images and second-band images of different bands are obtained from the image data; Measure the first remote heartbeat signal of the first band image; Measure the second remote heartbeat signal of the second band image; A first quality score is determined based on the first band image; Determine the second quality fraction based on the second band image; Based on the first remote heartbeat signal and the first quality score, a first effective heart rate zone is determined; Based on the second remote heartbeat signal and the second quality fraction, a second effective heart rate zone is determined; and Complementary heart rates are determined based on the first effective heart rate zone and the second effective heart rate zone.

10. The method according to claim 9, wherein, The steps of determining the first mass fraction and the second mass fraction include at least one of the following: Determine the first motion mass fraction and the second motion mass fraction; Determine the first lighting quality score and the second lighting quality score; and Determine the first signal quality score and the second signal quality score.

11. An apparatus comprising: One or more processors; as well as A memory storing instructions that, when executed by the one or more processors, cause the device to: Image data representing one or more photographic images of the vehicle's driver is obtained through the vehicle's cameras; Based on the image data having one or more predetermined image features, it is determined that the driver meets an age threshold; Based on facial expression analysis of the image data, the driver's emotion category is determined; Determine a value associated with the driver's heart rate, wherein the value associated with the heart rate corresponds to the image data; The existence of an emergency is determined based on whether the driver meets an age threshold, based on emotion classification, and based on heart rate. Based on the existence of the aforementioned emergency, a first request for a user response from the driver is output through the vehicle's user interface; and If no response is received from the driver within a predetermined time period after the first request is sent, an emergency rescue request is sent to the emergency dispatch service provider.

12. The apparatus according to claim 11, wherein, When the instructions are executed by the one or more processors, the device also causes the device to: Based on the user response received from the driver within the predetermined time period, a second request is output through the user interface, the second request instructing the driver to agree to the transfer of control of the vehicle; as well as Based on the instruction received with the consent, control of the vehicle is transferred to an entity different from the driver.

13. The apparatus according to claim 12, wherein, The entity includes a remote server configured to remotely control the vehicle.

14. The apparatus according to claim 12, wherein, The entity includes a computing device located in the vehicle, the computing device being configured to control the vehicle to perform autonomous driving.

15. The apparatus according to claim 11, wherein, When the instructions are executed by the one or more processors, the device also causes the device to: If no user response is received from the driver within the predetermined time period, a second request is output through the user interface, the second request instructing the driver to agree to transfer control of the vehicle. as well as Based on the absence of the instruction to consent within the predetermined time period, additional image data representing one or more additional photographic images of the driver is obtained through the camera, and the existence of an emergency is confirmed based on the additional image data.

16. The apparatus according to claim 11, wherein, The one or more predetermined image features are associated with at least one of the driver's hair or the driver's wrinkles, and When executed by the one or more processors, the instructions cause the device to determine that the driver meets the age threshold in the following manner: The driver's age is estimated based on the image data using a first model trained to recognize the presence of gray hair and wrinkles.

17. The apparatus according to claim 11, wherein, When executed by the one or more processors, the instructions cause the device to determine the emotion category in the following manner: Based on the image data, global features applying the multi-scale method are determined from the image data using multiple multi-scale blocks with filters of different sizes. The Convolutional Block Attention (CBAM) module is used to determine local features from the image data to which attention is to be applied. The CBAM module includes a channel attention module and a spatial attention module, and the channel attention module and the spatial attention module are applied sequentially. The global features and the local features are input into the graph convolutional network (GCN) combiner to perform feature combination; as well as Based on the results of the feature combination, a classifier is used to determine the driver's emotion category.

18. The apparatus according to claim 17, wherein, When the instructions are executed by the one or more processors, the device also causes the device to: The dominant feature is selected by applying a feature selector to both the global and local features; and Extract facial patch images corresponding to the locations of the dominant features from the global features, and When executed by the one or more processors, the instructions cause the device to perform the feature combination in the following manner: The feature combination is performed by inputting the features obtained by magnifying the patch image and applying attention to the magnified patch image together with the global features and the local features into a graph convolutional network (GCN) combiner.

19. The apparatus according to claim 11, wherein, When executed by the one or more processors, the instructions cause the device to determine the driver's heart rate in the following manner: First-band images and second-band images of different bands are obtained from the image data; Measure the first remote heartbeat signal of the first band image; Measure the second remote heartbeat signal of the second band image; A first quality score is determined based on the first band image; Determine the second quality fraction based on the second band image; Based on the first remote heartbeat signal and the first quality score, a first effective heart rate zone is determined; Based on the second remote heartbeat signal and the second quality fraction, a second effective heart rate zone is determined; as well as Complementary heart rates are determined based on the first effective heart rate zone and the second effective heart rate zone.

20. The apparatus according to claim 19, wherein, When executed by the one or more processors, the instructions cause the device to determine the first quality score and the second quality score in the following manner: Determine the first motion mass fraction and the second motion mass fraction; Determine the first lighting quality fraction and the second motion lighting fraction; and Determine the first signal quality score and the second signal quality score.

Images