Method for a driver monitoring system and driver monitoring system

Abstract

The present disclosure describes a method adapted to a driver monitoring system and the driver monitoring system. The method may comprise obtaining a frame sequence, each frame sequence consisting of a plurality of color scalars associated with a plurality of face zones of a driver in a vehicle. For each frame sequence, the method may determine whether a monitoring function is active for a first set of face zones that are shadow face zones; and in response to determining that the monitoring function is active, perform subsequent steps. In some embodiments, the subsequent steps may comprise monitoring the first set of face zones; adaptively determining a second set of face zones that are non-shadow face zones based on the result of the monitoring; and performing calculations for recovering a heartbeat signal only for the second set of face zones.

Description

Technical Field

[0001] The present disclosure relates to a method adapted for a driver monitoring system (DMS) in a vehicle and the DMS using the method, and more particularly, to the method for accelerating a remote photoplethysmography (rPPG) algorithm in the DMS by adaptively processing the shadow of a driver's face.

Background Art

[0002] The DMS is a system well known in the field of ADAS (Advanced Driver Assistance System). The DMS monitors and evaluates the state of a driver during driving, warns the driver if necessary, and finally applies brakes if necessary. The DMS is designed to detect two types of distraction: namely, visual distraction (also known as "eyes on the road") and cognitive distraction (referred to as "mind on the road"). Apart from conventional methods of analyzing gaze and blinking, cognitive load and stress can also be derived from IBI, i.e., InterBeat-Interval (briefly explained as heart rate). The rPPG process is an excellent approach for obtaining the IBI since it does not require contact with human skin.

[0003] The rPPG is an abbreviation for remote photoplethysmography, or simply remote heart rate estimation. This is a simple optical technique used to detect volume changes of blood in peripheral circulation. The rPPG algorithm can non-contact measure the difference in the reflection changes of red, green, and blue light from the skin. The rPPG algorithm measures the contrast between direct reflection and diffuse reflection. The direct reflection is the pure light reflection from the skin, while the diffuse reflection is the reflection remaining after absorption and scattering by skin tissue, which changes as the blood volume changes.

[0004] However, when applying the rPPG algorithm under the scenario of DMS, several problems may be encountered, and the most prominent of these problems is the lack of computing power of the DMS. Therefore, it is necessary to improve the technology.

Summary of the Invention

Means for Solving the Problems

[0005] According to one aspect of the present disclosure, a method adapted to a driver monitoring system is provided. The method may comprise obtaining a frame sequence, each frame sequence consisting of a plurality of color scalars associated with a plurality of face zones of a driver in a vehicle. For each frame sequence, the method may determine whether a monitoring function is active for a first set of face zones that are shadow face zones; and in response to determining that the monitoring function is active, subsequent steps may be performed. In some embodiments, the subsequent steps may comprise monitoring the first set of face zones; adaptively determining a second set of face zones that are non-shadow face zones based on the result of the monitoring; and performing calculations for recovering a heartbeat signal only for the second set of face zones.

[0006] According to another aspect of the present disclosure, a driver monitoring system is provided. The driver monitoring system may include a camera and a processor. The camera may capture a face image of the driver within the vehicle. The processor is coupled to the camera and may obtain a frame sequence from the face image, each frame sequence consisting of a plurality of color scalars associated with a plurality of face zones of the driver. The processor may perform the following for each frame sequence. The processor may determine whether a monitoring function is active for a first set of face zones that are shadow face zones; and in response to determining that the monitoring function is active, may perform subsequent steps. In some embodiments, the processor may monitor the first set of face zones; adaptively determine a second set of face zones that are non-shadow face zones based on the result of the monitoring; and perform calculations for recovering a heartbeat signal only for the second set of face zones.

[0007] According to still another aspect of the present disclosure, a non-transitory computer-readable storage medium comprising computer-executable instructions is provided, which, when executed by a computer, cause the computer to perform the methods disclosed herein.

Brief Description of the Drawings

[0008]

Figure 1

[0009]

Figure 2

[0010]

Figure 3

[0011]

Figure 4

Embodiments for Carrying Out the Invention

[0012] For ease of understanding, where possible, the same reference numbers are used to designate the same elements common to multiple drawings. It is contemplated that elements disclosed in one embodiment may be beneficially utilized in other embodiments without special description. The drawings referred to in this disclosure should not be understood to be drawn to scale unless otherwise specified. Also, for clarity of presentation and explanation, the drawings are often simplified and details or components are omitted. The drawings and the description serve to explain the principles discussed below, and like designations indicate like elements.

[0013] Examples are provided below for illustration purposes. Descriptions of the various embodiments are presented for illustrative purposes and are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.

[0014] In the present disclosure, in order to address the problems faced when transplanting the rPPG algorithm into the vehicle's DMS by accelerating the rPPG algorithm, a new method and system are provided. This is achieved by adaptively processing the shadow of the driver's face. Specifically, the shadow face zones can be determined and continuously monitored, such that instead of all the data associated with all the face zones of the driver's face, only the data associated with the shadow-free face zones is used to perform the rPPG algorithm (e.g., perform calculations or processes to recover the heartbeat signal). In other words, the data associated with the shadow face zones is excluded from the calculations for recovering the heartbeat signal. The method and the system in the present disclosure can further dynamically determine and adjust a set of shadow-free face zones by monitoring the shadow face zones. When the shadow or a part of the shadow disappears, the corresponding face zone considered as a shadow face zone is re-considered as a shadow-free face zone and returned to the calculations for recovering the heartbeat signal. The method and the system proposed in the present disclosure can adaptively implement that only the data associated with the shadow-free face zones is calculated, thereby saving calculation time, resources, and power consumption. The method and the system are described in detail with reference to FIGS. 1 to 4 below.

[0015] In the DMS scenario, the rPPG process obtains, as input, the color scalars of various pre-defined face zones, and the color scalars are values related to the changes in skin light reflection. These pre-defined face zones are generated by linking the detected face landmarks. FIG. 1 shows an example of the definition of the landmarks and the generated face zones. As shown in the example of FIG. 1, in the case of this face, 68 face landmarks can be detected, and thus, the face can be divided into 51 pre-defined face zones. In FIG. 1, for clarity of presentation, only three face zones identified by reference numerals 101, 102, and 103, respectively, are marked with triangles.

[0016] Taking the generated face zone shown in FIG. 1 as an example, in the rPPG process, for the color scalars of all 51 pre-defined face zones, the frame sequence can be considered together or processed. Thus, there will be a total array of 51 color scalars at the same time. Optionally, interpolation is used to supplement the bad scalars for face zones with bad color scalars (e.g., due to being normally undetected or having a leaf shadow on the driver's face). Then, a proposed method involving some calculations is further performed to recover the heartbeat signal, which will be described in detail below.

[0017] FIG. 2 shows an exemplary DMS diagram in a vehicle according to one or more embodiments of the present disclosure. As shown in FIG. 2, the DMS 200 may include a camera 202, which may capture a face image of an occupant such as a driver or a passenger in the vehicle. The camera 202 may be disposed at a location within the vehicle where it can monitor the face of the occupant. The camera 202 may include one or more optical (e.g., visible light) cameras, one or more infrared (IR) cameras, or a combination of optical and IR cameras having one or more fields of view. In some examples, the camera 202 may include one or more lenses and one or more image sensors. In some examples, the camera 202 may be a digital camera configured to acquire a series of images (e.g., frames) at a programmable frequency (e.g., frame rate). In some examples, the frame rate may be selected based on the processing speed of the processor 206 included in the DMS 200.

[0018] The processor 206 may be configured to execute machine-readable instructions stored in the memory 204. The processor 206 may be electronically and / or communicatively coupled to the camera 202, and may process and analyze a face image received from the camera to recover the heartbeat signal. In some examples, the processor 206 may obtain and process a frame sequence from the face image, and each frame sequence may consist of N arrays of color scalars corresponding to N face zones of the driver. N is an integer greater than 1 and represents the number of face zones of the occupant. The generation of the face zones may be the same as or similar to the example described with reference to FIG. 1. The number of N may be preset. The processor 206 may be configured to perform the method as detailed in the following specification with respect to FIGS. 3-4.

[0019] The processor 206 may be single-core or multi-core, and the program executed by the processor 206 may be configured for parallel processing or distributed processing. The processor 206 may be any technically realizable hardware unit configured to perform processing functions and execute software applications, including but not limited to a central processing unit (CPU), a microcontroller unit (MCU), an application-specific integrated circuit (ASIC), a digital signal processor (DSP) chip, a field programmable gate array (FPGA), a graphics board, etc.

[0020] The memory 204 may include any non-transitory tangible computer-readable medium on which programming instructions are stored. The term "tangible computer-readable medium" as used herein is expressly defined to include any kind of computer-readable storage. The example methods described herein may be implemented using encoded instructions (e.g., computer-readable instructions) stored on a non-transitory computer-readable medium such as flash memory, read-only memory (ROM), random access memory (RAM), cache, or any other storage medium on which information is stored for any period of time (e.g., long term, permanent, short term, for temporary buffering and / or for caching of the information). The computer memory of the computer-readable storage medium referred to herein may include volatile and non-volatile, or removable and non-removable media for storing electronically formatted information such as computer-readable program instructions, or modules of computer-readable program instructions, data, etc., and this medium may be stand-alone or part of a computing device. As an example of computer memory, it may include any other medium that can be used to store information in a desired electronic format and that can be accessed by at least a part of the processor(s) or computing device.

[0021] Figure 3 shows an example of the method 300 for accelerating the rPPG process in accordance with one or more embodiments of the present disclosure. The method 300 is described with respect to the system of FIG. 2. For example, the method 300 may be executed by the processor 206 based on instructions / codes stored in the memory 204 of the DMS 200. However, it can be understood that the method may be implemented using other systems and components without departing from the scope of the present disclosure.

[0022] In S302, the processor 206 can obtain a frame sequence from the face image. Each frame sequence may include a plurality of color scalars associated with a plurality of face zones of the driver. For example, assuming there are 51 face zones, there is a total array of 51 color scalars at a time, and one array corresponds to one face zone. For example, assuming the length of the frame sequence is 512, it can be understood that each array includes 512 color scalars, and one array with 512 color scalars corresponds to one face zone.

[0023] In S304, the processor 206 can determine, for each frame sequence, whether the monitor function is active for some shadow face zones. If the determination is "NO", this means that there is no shadow, and the method then proceeds to S306.

[0024] In S306, the processor 206 may perform a calculation or process to recover the heartbeat signal using all the color scalars associated with all the face zones. In some examples, an FFT (Fast Fourier Transform) calculation may be applied to the frame sequence including the color scalars corresponding to all the face zones. Based on the result of the FFT calculation, the SNR (Signal-to-Noise Ratio) values of all the face zones may be calculated. Also, each FFT result may be fed into a neural network to derive the quality value of each face zone. Thus, the quality values of all the face zones may be obtained. Based on the color scalar, the calculated SNR value, and the calculated quality value, a zone value may be calculated. For example, in the case of 51 face zones, 51 zone values for the 51 face zones are obtained. For example, each zone value may be calculated by ColorScalar * SNR * Quality, where ColorScalar represents the value of the color scalar, SNR represents the calculated SNR value, Quality represents the calculated quality value, and the symbol * represents the multiplication operator. Next, the calculated zone values for all the face zones may be summed to obtain a final value indicating the heartbeat signal. Based on the obtained final value, the heartbeat signal may be recovered.

[0025] Returning to S304, if the determination is "YES", the method then proceeds to S308. In S308, the processor 206 may monitor the shadow face zones. In some embodiments, monitoring the shadow face zones may include continuously monitoring these face zones to determine whether any changes have occurred in these face zones, for example, whether one or more face zones are no longer located in the shadow. In other words, it may be determined whether one or more of these monitored face zones should be re-considered as non-shadow face zones. In some embodiments, monitoring the shadow face zones may include calculating the SNR values of these monitored shadow face zones.

[0026] Next, the method proceeds to S310. At S310, the set of shadowless face zones can be adaptively determined or adjusted according to the result of the monitoring. In some embodiments, the processor 206 may compare each of the calculated SNR values with an SNR threshold. If none of the calculated SNR values is greater than the SNR threshold, that is, if these face zones are still in the shadow, the set of shadowless face zones is maintained. If one or more of the calculated SNR values are greater than or equal to the SNR threshold, the set of shadowless face zones is adjusted. In some examples, the processor 206 may determine which face zone(s) the calculated SNR(s) greater than or equal to the threshold correspond to, and then may include the determined one or more face zones in the set of shadowless face zones. It can be understood that the set of shadowless face zones and the set of shadow face zones constitute the entire face zones.

[0027] Next, the method proceeds to S312. At S312, the processor may perform subsequent calculations for recovering the heartbeat signal only for the set of shadowless face zones. In some embodiments, the processor 206 may calculate the SNR value and the quality value only for the set of shadowless face zones based on the above-mentioned FFT result. The processor 206 may further calculate the zone value of the set of shadowless face zones based on the color scalar associated with the shadowless face zones, the calculated SNR value, and the calculated quality value. Further, the processor 206 may sum the calculated zone values to obtain a final value, and may recover the heartbeat signal based on the obtained final value. In some examples, the processor 206 may further determine whether there is one or more zone values smaller than a predefined threshold. If there is one or more zone values smaller than the predefined threshold, the monitoring function is activated for the subsequent frame sequence for those face zones associated with the one or more zone values smaller than the threshold.

[0028] Figure 4 shows an example of another method 400 for accelerating the rPPG process in accordance with one or more embodiments of the present disclosure. The method 400 can be performed for each frame sequence obtained from the face image. As described with respect to FIG. 3, each frame sequence can include a plurality of color scalars associated with a plurality of face zones of the driver.

[0029] At S402, a determination can first be made to determine whether the monitor function is active for some shadow face zones. If the determination is "NO", this means that there is no shadow, and the method then proceeds to S404.

[0030] At S404, the SNR value can be calculated for all face zones, and at S406, the quality value can be obtained for all the face zones. It is contemplated that the steps of S404 and S406 shown consecutively can actually be performed substantially simultaneously, or the steps can be performed in the reverse order depending on the case. At S408, the zone value for all face zones is obtained based on the color scalar, the calculated SNR value, and the calculated quality value. The details of the calculations of S404 to S408 can be similar or the same as the description regarding S306 in FIG. 3. Next, the method proceeds to S410.

[0031] In S410, the method determines whether one or more low zone values exist. Ideally, the light conditions of all face zones in the frame sequence (for example, each of the 51 individual face zones) are expected to be similar, and as a result, the SNR values and signal quality values of each of the 51 face zones are close to each other. As a result, the 51 zone values calculated by (ColorScalar * SNR * Quality) for each are also close to each other among all 51 face zones. However, in actual driving situations, for example, shadows may be cast on the driver's face by the vehicle's sun visor or street trees. Different from fast-moving tree shadows, the shadow caused by the sun visor usually persists at a specific position on the driver's face. As a result, at these specific positions within the shadow (for example, one or more shadow face zones), one or more of the SNR value and quality value are low over a long period of time within the period of the entire frame sequence (that is, the length of the frame sequence), and thus one or more zone values are low. In this scenario, when performing the step of calculating the final value, the values of these low zones will be considered as low-quality "noise". Therefore, the computing power (for calculating SNR and quality) of these shadow face zones is consumed in vain. To solve this problem, the method of the present disclosure determines, for example, in S410 whether one or more low zone values exist. Since the shadow of the sun visor usually persists on a specific face zone (most often the upper part of the face) and the area of the shadow does not change frequently, it is reasonable to exclude the area of the shadow from the calculation of the subsequent frames.

[0032] When returning to S410, the criteria for "low" here can be various. In some examples, the criteria can be a comparison with a pre-defined threshold. If the one or more calculated zone values are smaller than the pre-defined threshold, they can be determined to be "low" zone values. In some examples, the criteria can be a comparison with a proportional threshold based on the ratio of one zone value to other valid zone values. For example, if one or more zone values are less than 1 / 10 of other valid zone values, the one or more zone values can be determined to be "low" zone values.

[0033] If it is determined in S410 that there are no low zone values, the method proceeds to S412. In S412, the method can recover the heartbeat signal. In some examples, in S412 the method can include summing the zone values calculated in S408 to obtain the final value of all the face zones, and recovering the heartbeat signal based on the obtained final value. Next, the method proceeds to S428 and transitions to the subsequent frame sequence.

[0034] If it is determined in S410 that there are one or more low zone values, the method proceeds to S414. In S414, the method can activate the SNR monitoring function for these face zones having low zone values. Next, the method proceeds to S412 and then to S428.

[0035] Return to S402. If the determination result at S402 is "YES", the method then proceeds to S416. At S416, the method monitors the SNR value only for the face zone of the shadow identified in the processing of the previous frame sequence(s). That is, in the SNR monitoring function, it means that only the SNR value of the identified shadow-like region can be calculated. In some examples, the SNR monitoring function (i.e., the calculation of the SNR value for one or more face zones of the shadow) can be executed at a frequency lower than the frame rate. In some examples, for all the identified face zones of the shadow, the SNR value can be calculated. In some examples, instead of calculating the SNR value for every face zone of the shadow, the method can randomly select some face zones from the identified face zones and calculate their SNR values.

[0036] At S418, the method further determines whether the SNR value calculated at S416 is still "low". The criteria for "low" here can be various. In some examples, the criteria can be a comparison with a pre-defined SNR threshold. If one or more calculated SNR values are smaller than the pre-defined SNR threshold, they can be determined to be still "low". In some examples, the criteria can be a comparison with a proportional threshold based on the ratio of one NR value to another valid SNR value. For example, if one or more SNR values are less than 1 / 10 of another valid SNR value, the one or more SNR values can be determined to be still "low".

[0037] At S418, if the method determines that all SNR values are still low, the method proceeds to S420. At S418, if the method determines that one or more SNR values are no longer low, the method proceeds to S422. At S422, the face zones corresponding to the one or more SNR values that are no longer low can be re-considered as face zones that are not in shadow. Next, the method proceeds to S420.

[0038] At S420, only the SNR value of the shadowless face zone can be calculated. Next, the method proceeds to S424. At S424, the quality value of the shadowless face zone can be obtained. Next, the method proceeds to S426. At S426, the zone value of the shadowless face zone can be calculated based on the color scalar associated only with the shadowless face zone, the SNR value calculated at S420, and the quality value calculated at S424. Next, the method proceeds to S410 to determine whether one or more low zone values exist in the face zone.

[0039] At S410, if it is determined that no low zone value exists, the method proceeds to S412. At S412, the method can recover the heartbeat. In some examples, at S412, the method can include summing the zone value calculated at S426 to obtain the final value of the shadowless face zone, and recovering the heartbeat signal based on the obtained final value. Next, the method proceeds to S428 to transition to the subsequent frame sequence.

[0040] At S410, if it is determined that there is one or more low zone values, the method proceeds to S414. At S414, the method can activate the SNR monitoring function for these face zones having low zone values. The method proceeds to S412 and then to S428 to perform the processing of the subsequent frame sequence.

[0041] 1. In some embodiments, a method adapted for a driver monitoring system, comprising: obtaining a frame sequence, each frame sequence including a plurality of color scalars associated with a plurality of face zones of a driver in a vehicle; for each frame sequence, determining whether a monitoring function is active for a first set of face zones that are shadow face zones; in response to determining that the monitoring function is active, performing the following: monitoring the first set of face zones; adaptively determining a second set of face zones that are non-shadow face zones based on the result of the monitoring; and performing calculations for recovering a heartbeat signal only for the second set of face zones.

[0042] 2. The method according to clause 1, wherein monitoring the first set of face zones comprises calculating a first SNR value for the first set of face zones.

[0043] 3. Adaptively determining the second set of face zones based on the result of the monitoring comprises: comparing each first SNR value with an SNR threshold; maintaining the second set of face zones in response to all first SNR values being less than the SNR threshold; and adjusting the second set of face zones in response to one or more first SNR values being greater than or equal to the SNR threshold. The method according to any one of clauses 1 to 2.

[0044] 4. Adjusting the second set of face zones comprises: determining one or more face zones associated with the one or more first SNR values that are greater than or equal to the SNR threshold; and adding the determined one or more face zones to the second set of face zones. The method according to any one of clauses 1 to 3.

[0045] 5. Performing the calculation to recover the heartbeat signal only for the second set of face zones includes: calculating the zone values of the second set of face zones; summing the calculated zone values to obtain a final value; and recovering the heartbeat signal based on the obtained final value. The method according to any one of clauses 1 to 4.

[0046] 6. Calculating the zone values for the second set of face zones includes: calculating a second SNR value for the second set of face zones; calculating a second quality value for the second set of face zones; and calculating the zone values for the second set of face zones based on the color scalar associated with the second set of face zones, the calculated second SNR value, and the calculated second quality value. The method according to any one of clauses 1 to 5.

[0047] 7. Further including: determining whether there is one or more zone values less than a threshold; putting one or more face zones associated with the one or more zone values less than the threshold into the first set of face zones; and activating the monitoring function for the first set of face zones. The method according to any one of clauses 1 to 6.

[0048] 8. Further including performing a calculation to recover the heartbeat signal for all face zones in response to determining that the monitoring function is not active. The method according to any one of clauses 1 to 7.

[0049] 9. Further including: capturing a face image of the driver by a camera of the driver monitoring system; and obtaining the frame sequence from the face image. The method according to any one of clauses 1 to 8.

[0050] 10. A driver monitoring system comprising: a camera configured to capture a face image of a driver in a vehicle; a processor coupled to the camera, wherein: a frame sequence is obtained from the face image, each frame sequence including a plurality of color scalars associated with a plurality of face zones of the driver; for each frame sequence, it is determined whether a monitoring function is active for a first set of face zones that are shadow face zones; and in response to determining that the monitoring function is active, the following are performed: monitoring the first set of face zones; adaptively determining a second set of face zones that are non-shadow face zones based on the result of the monitoring; and performing calculations for recovering a heartbeat signal only for the second set of face zones.

[0051] 11. The driver monitoring system according to clause 10, wherein the processor is configured to calculate a first SNR value for the first set of face zones.

[0052] 12. The driver monitoring system according to any one of clauses 10 to 11, wherein the processor is configured to: compare each first SNR value with an SNR threshold; maintain the second set of face zones in response to all first SNR values being less than the SNR threshold; and adjust the second set of face zones in response to one or more first SNR values being greater than or equal to the SNR threshold.

[0053] 13. The driver monitoring system according to any one of clauses 10 to 12, wherein the processor is configured to: determine one or more face zones associated with the one or more first SNR values that are greater than or equal to the SNR threshold; and include the determined one or more face zones in the second set of face zones.

[0054] 14. The processor is configured to: calculate a zone value for the second set of face zones; sum the calculated zone values to obtain a final value; and recover the heartbeat signal based on the obtained final value, for the driver monitoring system according to any one of clauses 10 to 13.

[0055] 15. The processor is configured to: calculate a second SNR value for the second set of face zones; calculate a second quality value for the second set of face zones; and calculate the zone value for the second set of face zones based on the color scalar, the calculated second SNR value, and the calculated second quality value associated with the second set of face zones, for the driver monitoring system according to any one of clauses 10 to 14.

[0056] 16. The processor is configured to: determine whether there is one or more zone values less than a threshold; place one or more face zones associated with the one or more zone values less than the threshold into the first set of face zones; and activate the monitoring function for the first set of face zones, for the driver monitoring system according to any one of clauses 10 to 15.

[0057] 17. The processor is configured to perform calculations for recovering the heartbeat signal for all the face zones in response to determining that the SNR monitoring function is not active, for the driver monitoring system according to any one of clauses 10 to 15.

[0058] 18. A computer-readable storage medium comprising computer-executable instructions that, when executed by a computer, cause the computer to perform the method according to any one of clauses 1 to 9.

[0059] The descriptions of the various embodiments are presented for illustrative purposes and are not intended to be exhaustive or limited to the disclosed embodiments. The terms used herein are chosen to best explain the principles of the embodiments, practical applications to technologies found in the marketplace, or technological improvements, or to enable those of ordinary skill in the art to understand the embodiments disclosed herein.

[0060] Previously, reference has been made to the embodiments presented in this disclosure. However, the scope of this disclosure is not limited by the specific embodiments described herein. Instead, any combination of the foregoing features and elements, whether or not related to various embodiments, is contemplated for implementing and practicing the contemplated embodiments. Further, the embodiments disclosed herein may achieve advantages over the prior art and over other possible solutions, but whether a particular advantage is achieved by a given embodiment is not limiting of the scope of this disclosure. Thus, the foregoing aspects, features, embodiments, and advantages are merely illustrative and are not to be considered elements or limitations of the appended claims unless explicitly recited therein.

[0061] Aspects of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects that may generally be referred to herein as a "circuit," "module," or "system."

[0062] The present disclosure may be a system, method, and / or computer program product. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to execute aspects of the present disclosure.

[0063] The computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. The computer-readable storage medium can be, for example, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof, but is not limited thereto. A non-exhaustive but more specific list of examples of the computer-readable storage medium is as follows: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disks (DVD), memory sticks, floppy (registered trademark) disks, punch cards or mechanical encoding devices such as groove structures that record instructions, and any suitable combination of the foregoing. The computer-readable storage medium as used herein should not be construed as being a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., an optical pulse passing through an optical fiber cable), or an electrical signal transmitted through a wire.

[0064] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to respective computing / processing devices or to an external computer or external storage device via a network such as, for example, the Internet, a local area network, a wide area network, and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers.

[0065] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products, according to embodiments of the method. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0066] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the block(s) of the flowchart and / or block diagram.

[0067] The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). Also, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

[0068] The foregoing is directed to embodiments of the present disclosure, but other and further embodiments of the present disclosure may be devised without departing from the basic scope thereof, and that scope is defined by the claims that follow.

Claims

1. A method for a driver monitoring system, comprising: obtaining a frame sequence, each frame sequence including a plurality of color scalars associated with a plurality of face zones of a driver in a vehicle, said obtaining; for each frame sequence, determining whether a monitoring function is active for a first set of face zones that are shadow face zones; in response to determining that the monitoring function is active, the following: monitoring the first set of face zones; adaptively determining a second set of face zones that are non-shadow face zones based on the result of said monitoring; performing calculations for recovering a heartbeat signal only for the second set of face zones, said method comprising.

2. The method according to claim 1, wherein said monitoring the first set of face zones comprises calculating a first SNR value for the first set of face zones.

3. Adapting said determination of the second set of face zones based on the result of said monitoring comprises: comparing each first SNR value with an SNR threshold; maintaining the second set of face zones in response to all first SNR values being less than the SNR threshold; adjusting the second set of face zones in response to one or more first SNR values being greater than or equal to the SNR threshold, the method according to claim 2.

4. Adjusting said second set of face zones comprises: determining one or more face zones associated with said one or more first SNR values that are greater than or equal to the SNR threshold; including the determined one or more face zones in the second set of face zones, the method according to claim 3.

5. Performing said calculations for recovering the heartbeat signal only for the second set of face zones comprises: calculating a zone value for the second set of face zones; summing the calculated zone values to obtain a final value; recovering the heartbeat signal based on the obtained final value, the method according to claim 1.

6. Calculating said zone value for the second set of face zones comprises: Calculating a second zone value for the second set of face zones; Calculating a second quality value for the second set of face zones; Calculating the zone value for the second set of face zones based on the color scalar, the calculated second SNR value, and the calculated second quality value associated with the second set of face zones, the method of claim 5 comprising. **Claim 7** Furthermore: Determining whether there is one or more zone values less than a threshold; Putting one or more face zones associated with the one or more zone values less than the threshold into the first set of face zones; Activating the monitoring function for the first set of face zones, the method of claim 5 comprising. **Claim 8** Furthermore, in response to determining that the monitoring function is not activated, performing calculations for recovering a heartbeat signal for all of the face zones, the method of claim 1 comprising. **Claim 9** Furthermore: Capturing a face image of the driver by a camera of the driver monitoring system; Obtaining the frame sequence from the face image, the method of claim 1 comprising. **Claim 10** A driver monitoring system comprising: A camera configured to capture a face image of a driver in a vehicle; A processor coupled to the camera, and: Obtaining the frame sequence from the face image, each frame sequence including a plurality of color scalars associated with a plurality of face zones of the driver; Determining for each frame sequence whether a monitoring function is active for a first set of face zones that are shadow face zones; and In response to determining that the monitoring function is active, the following: Monitoring the first set of face zones; Adaptively determining a second set of face zones that are non-shadow face zones based on the result of the monitoring; Performing calculations for recovering a heartbeat signal only for the second set of face zones, the processor configured to perform. **Claim 11** The driver monitoring system according to claim 10, wherein the processor is configured to calculate a first SNR value for the first set of face zones.

12. The processor is: comparing each first SNR value with an SNR threshold; maintaining the second set of face zones in response to all first SNR values being less than the SNR threshold; and The driver monitoring system according to claim 11, wherein the second set of face zones is adjusted in response to one or more of the first SNR values being greater than or equal to the SNR threshold.

13. The processor is: determining one or more face zones associated with the one or more first SNR values that are greater than or equal to the SNR threshold; and The driver monitoring system according to claim 12, wherein the determined one or more face zones are included in the second set of face zones.

14. The processor is: calculating a zone value for the second set of face zones; summing the calculated zone values to obtain a final value; and The driver monitoring system according to claim 10, wherein the heartbeat signal is recovered based on the obtained final value.

15. The processor is: calculating a second SNR value for the second set of face zones; calculating a second quality value for the second set of face zones; and The driver monitoring system according to claim 14, wherein the zone value for the second set of face zones is calculated based on the color scalar, the calculated second SNR value, and the calculated second quality value associated with the second set of face zones.

16. The processor is: determining whether there are one or more zone values less than a threshold; placing one or more face zones associated with the one or more zone values less than the threshold into the first set of face zones; and The driver monitoring system according to claim 14, wherein the monitoring function is activated for the first set of face zones.

17. The driver monitoring system according to claim 10, wherein the processor is configured to perform calculations for restoring a heartbeat signal for all of the face zones in response to determining that the monitoring function is not active.

18. A computer-readable storage medium comprising computer-executable instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 9.

Images