Driver monitoring method and system

Abstract

A method for driver monitoring is provided, comprising: obtaining gaze tracking data for the driver; calculating, based on the gaze tracking data, an output value representing the cumulative gaze of the driver on various predetermined sub-regions within a time period for the corresponding predetermined sub-region of each predetermined region among multiple predetermined regions, where the multiple predetermined regions represent non-overlapping areas of interest during vehicle driving, and each predetermined region is divided into multiple non-overlapping predetermined sub-regions; and determining the driver's state based on the output values of various predetermined sub-regions within each predetermined region. This improves the usability and performance of the driver monitoring function.

Description

[0001] Specification

[0002] Driver Monitoring Method and System

[0003] Technical Field

[0004] The present disclosure relates to the field of driver monitoring, and more particularly, to eye tracking of the driver.

[0005] Background Art

[0006] Driver monitoring systems are real-time systems based on processing driver facial images, physiological indicators, or vehicle information to determine the driver’s state. These systems can achieve functions such as driver identification, fatigue monitoring, distraction monitoring, and monitoring of dangerous driving behaviors (e.g., drunk driving, holding a phone while driving, and drinking water). Current driver monitoring systems have introduced eye tracking technology, which can detect the driver’s eye movements and head angles to determine the driver’s state and achieve real-time monitoring of the driver.

[0007] As an application of the driver monitoring system, eye tracking technology can be used to monitor whether the driver has checked the surrounding environment and confirmed the lane change after the automatic lane change system outputs an indication that a lane change is possible. The automatic lane change function is provided in Level 2 of the Society of Automotive Engineers (SAE) autonomous driving levels. According to this function, when the vehicle detects that a lane change is possible, it first displays the information indicating that a lane change is possible to the driver. The actual lane change is only executed after the driver has checked the surrounding environment and confirmed that the lane change is suitable. However, it is usually difficult to determine whether the driver has actually checked the surrounding environment. In some cases, the driver may overly trust the system and not genuinely check whether the surrounding environment is suitable for a lane change, which can lead to irreparable harm. Therefore, the driver monitoring system uses eye tracking technology to detect whether the driver has gazed at predefined areas and the duration of the gaze to determine whether the driver has actually confirmed the lane change.

[0008] According to the above method, strict rules need to be predefined, and then based on these rules, it is determined whether the driver has confirmed the lane change. For example, for a left lane change, the predefined area can be the area corresponding to the vehicle’s left rearview mirror; for a right lane change, the predefined area can be the area corresponding to the vehicle’s right rearview mirror; the threshold for the gaze duration can be between 300- 500 milliseconds. Only when it is detected that the driver has gazed at the area corresponding to the left rearview mirror for a duration exceeding the threshold is it determined that the driver has confirmed the left lane change. Similarly, for a right lane change, only when it is detected that the driver has gazed at the area corresponding to the right rearview mirror for a duration exceeding the threshold is it determined that the driver has confirmed the right lane change.

[0009] Summary of the Invention

[0010] It is desired to provide an improved driver monitoring method and system that does not require strictly predefined rules, is applicable to different drivers, and enhances the usability and performance of the driver monitoring function.

[0011] According to one aspect, a method for driver monitoring is provided, comprising: obtaining gaze tracking data for the driver; calculating, based on the gaze tracking data, an output value representing the cumulative gaze of the driver on various predetermined subregions within a time period for the corresponding predetermined sub-region of each predetermined region among multiple predetermined regions, where the multiple predetermined regions represent non-overlapping areas of interest during vehicle driving, and each predetermined region is divided into multiple non-overlapping predetermined subregions; and determining the driver’s state based on the output values of various predetermined sub-regions within each predetermined region.

[0012] According to another aspect, an apparatus for driver monitoring is provided, comprising: An obtaining unit configured to obtain gaze tracking data for the driver; a calculating unit configured to calculate, based on the gaze tracking data, an output value representing the cumulative gaze of the driver on various predetermined sub-regions within a time period for the corresponding predetermined sub-region of each predetermined region among multiple predetermined regions, where the multiple predetermined regions represent non-overlapping areas of interest during vehicle driving, and each predetermined region is divided into multiple non-overlapping predetermined sub-regions; and a determining unit configured to determine the driver’s state based on the output values of various predetermined sub-regions within each predetermined region.

[0013] According to yet another aspect, a computer-readable medium is provided, storing a computer program, wherein the computer program, when executed by a processor, causes the processor to perform the method according to various examples of the present disclosure.

[0014] According to various examples of various aspects of the present disclosure, the strictly defined regions corresponding to specific vehicle components are no longer used. Instead, areas of interest during the driver’s vehicle operation are divided into multiple non-overlapping predetermined regions, and each predetermined region is further divided into multiple nonoverlapping predetermined sub-regions. The output value representing the cumulative gaze of the driver on each predetermined sub-region over a period of time is then calculated for each predetermined sub-region. On this basis, different methods for gaze tracking of the driver are provided, allowing the driver monitoring system to interpret the gaze tracking data in different ways. This approach eliminates the need to strictly define regions corresponding to different vehicle components, potentially allowing for larger predetermined regions without the system’s performance being affected by individual differences among drivers. For example, due to varying heights of drivers and different settings of their rearview mirrors, strictly defining regions corresponding to the rearview mirrors may not be applicable to all drivers. By using the methods of various examples of the present disclosure, it is no longer necessary to strictly define regions corresponding to specific vehicle components. Larger predetermined regions can be set, which are clearly not affected by individual differences among drivers, thereby enhancing the usability of the driver monitoring function.

[0015] In one example, a template related to the driver can be obtained, and the state of the driver is determined based on the template and the calculated output values of the respective predetermined sub-regions. In one example, the template may include multiple templates corresponding to different states of the driver. By using driver-specific templates, customized state monitoring for the driver is achieved, improving the accuracy of monitoring and providing a better user experience.

[0016] Description of Accompanying Drawings

[0017] In the drawings, the examples are illustrated by way of example rather than by way of limitation, and like reference numerals in the drawings denote like elements.

[0018] FIG. 1 illustrates a region division method for driver monitoring according to the prior art;

[0019] FIG. 2 illustrates a region division method for driver monitoring according to one example of the present disclosure;

[0020] FIG. 3 illustrates a flowchart of a method for driver monitoring according to one example of the present disclosure;

[0021] FIGs. 4a-4c illustrate various predetermined sub-regions and their output values obtained according to different examples of the present disclosure;

[0022] FIG. 5 illustrates a method for determining the state of the driver based on the output values of various predetermined sub-regions according to an example of the present disclosure;

[0023] FIG. 6 illustrates a block diagram of an apparatus for driver monitoring according to one example of the present disclosure.

[0024] The various aspects and features of the examples of the present disclosure are described with reference to the above drawings. The above drawings are merely illustrative and not restrictive. In addition, not all portions of the device according to the examples of the present disclosure in the accompanying drawings above are shown with reference numerals, and only the relevant components are shown in some accompanying drawings. This does not limit the various portions merely to that shown in the accompanying drawings of the Specification.

[0025] Specific Embodiments

[0026] In the field of autonomous driving, in certain application scenarios, it is necessary to use a driver monitoring system to monitor the driver’s state in order to determine how to control the vehicle’s movement subsequently. For example, according to the automatic lane change function provided by the Society of Automotive Engineers (SAE) Level 2 of autonomous driving, after the vehicle detects that a lane change is possible, it first needs to send a lane change confirmation request to the driver. The actual lane change operation is executed only after the driver has checked the surrounding environment and confirmed that the lane change is safe. In this case, the driver monitoring system can be used to monitor whether the driver has indeed checked the surrounding environment.

[0027] In the current method, the driver monitoring system monitors whether the driver has truly checked the surrounding environment based on strictly defined rules. As shown in FIG. 1 , according to the current rules, the driver’s gaze is tracked to determine which of the areas 1 , 2, and 3 the driver has looked at and for how long, thereby monitoring the driver’s state. For example, if the driver gazes at area 1 for more than 500 milliseconds, it indicates that the driver has confirmed the execution of a left lane change. According to the current method, areas 1 , 2, and 3 are strictly defined areas corresponding to the vehicle’s left side mirror, center mirror, and right side mirror, respectively. However, different drivers have different driving habits, and the setting of mirrors may vary according to their personal circumstances. Additionally, the current gaze tracking technology has limited accuracy for small areas. Therefore, the aforementioned strictly defined rules are not applicable to all drivers and cannot ensure monitoring performance for different drivers.

[0028] According to various examples of the present disclosure, the strictly defined regions corresponding to specific vehicle components are no longer used. Instead, areas of interest during the driver’s vehicle operation are divided into multiple non-overlapping predetermined regions, and each predetermined region is further divided into multiple non-overlapping predetermined sub-regions. The output value representing the cumulative gaze of the driver on each predetermined sub-region over a period of time is then calculated for each predetermined sub-region. On this basis, different methods for gaze tracking of the driver are provided, allowing the driver monitoring system to interpret the gaze tracking data in different ways. This approach no longer requires strictly defined areas corresponding to the left side mirror, center mirror, and right side mirror (areas 1 , 2, and 3). Larger predetermined regions can be set, and the system’s performance will not be affected by individual differences among drivers.

[0029] As shown in FIG. 2, the areas of interest during the driver’s operation of the vehicle, including the front and side windows, are divided into a first predetermined region A1 , a second predetermined region A2, and a third predetermined region A3. The first predetermined region includes the area related to the vehicle’s left lane change, the second predetermined region includes the area related to lane-keeping, and the third predetermined region includes the area related to the vehicle’s right lane change. For example, the first predetermined region A1 includes the area corresponding to the left side mirror, the second predetermined region A2 includes the area corresponding to the center mirror, and the third predetermined region A3 includes the area corresponding to the right side mirror. It is understood that although the first predetermined region is described as related to the vehicle’s left lane change, this does not mean that the left lane change is solely related to the first predetermined region. For example, during a left lane change, the driver may also look at the center mirror, so the left lane change may also be related to the second predetermined region. Similarly, this does not mean that lane-keeping is solely related to the second predetermined region or that the right lane change is solely related to the third predetermined region.

[0030] For each predetermined region A1 , A2, or A3, it is further divided into multiple nonoverlapping predetermined sub-regions, such as a, b, c, and d. Although these predetermined sub-regions are shown as uniformly equal squares in FIG. 2, this is not restrictive. They can be divided in any other way, such that the shapes and / or sizes of these predetermined subregions are arbitrarily selected. Based on the division of the predetermined regions and their predetermined sub-regions, for each predetermined sub-region, an output value representing the cumulative gaze of the driver on that predetermined sub-region over a period of time can be determined. This allows for the determination of the driver’s gaze distribution across the various predetermined regions during that period, thereby enabling the determination of the driver’s state.

[0031] FIG. 3 illustrates a flowchart 100 of a method for driver monitoring according to one example of the present disclosure. As shown in FIG. 3, according to the method 100, first, in step 110, gaze tracking data for the driver is received. In one example, the gaze tracking data includes multiple frames of images collected for tracking the driver’s eye movements and may also include detection data for the driver’s head angle. In another example, the gaze tracking data includes first data representing the position of the driver’s gaze and second data representing the duration of the driver’s gaze at the corresponding position. The first and second data can be derived from the multiple frames of images collected for the driver and / or head angle detection data.

[0032] Next, in step 120, based on the pre-divided multiple predetermined regions and multiple predetermined sub-regions as described above, an output value representing the cumulative gaze of the driver on each predetermined sub-region over a period of time is calculated. For example, this output value can be calculated using a mathematical expectation.

[0033] In one example, when the received gaze tracking data includes multiple frames of images collected for the driver, the first data representing the position of the driver’s gaze can be determined for each frame. Based on this first data, it can be determined which of the multiple predetermined sub-regions were gazed at by the driver for that frame. The cumulative gaze output value for each predetermined sub-region can then be determined based on the gazed predetermined sub-regions for each frame.

[0034] For example, assume that four frames of images were collected over a period. Referring to FIG. 2, for the first frame, the determined first data indicates that the driver gazed at predetermined sub-regions a and b, so specific values, such as 1 , can be assigned to predetermined sub-regions a and b. For the second frame, the determined first data indicates that the driver gazed at predetermined sub-regions a and c, so specific values of 1 can be assigned to predetermined sub-regions a and c. For the third frame, the determined first data indicates that the driver gazed at predetermined sub-regions c and d, so specific values of 1 can be assigned to predetermined sub-regions c and d. For the fourth frame, the determined first data indicates that the driver gazed at predetermined sub-region a, so a specific value of 1 can be assigned to predetermined sub-region a. Thus, over the period for the four frames of images, the cumulative gaze output value for predetermined sub-region a is 1 +1+1 =3; for predetermined sub-regions b and d, the cumulative gaze output value is 1 each; and for predetermined sub-region c, the cumulative gaze output value is 1+1=2. Other ungazed predetermined sub-regions can be assigned an output value of, for example, 0. This way, the cumulative gaze output value for various predetermined sub-region over the period can be determined.

[0035] Although the above example describes determining the output value for each predetermined sub-region based on multiple frames of images with reference to FIG. 2, this is not restrictive. When the received gaze tracking data includes first data representing the position of the driver’s gaze and second data representing the duration of the driver’s gaze at the corresponding position, the output values for various predetermined sub-regions over a period can also be determined based on the first and second data. In one example, the second data representing the gaze duration can be, for example, the number of frames the driver gazed at the corresponding position. The cumulative gaze output value for each predetermined sub-region can be determined based on which predetermined sub-regions the driver gazed at and the number of frames gazed at each specific predetermined sub-region.

[0036] Additionally, the degree or probability of the driver’s gaze falling into the corresponding predetermined sub-region can be introduced to further refine the calculation of the cumulative gaze output value for each predetermined sub-region. Specifically, for each frame, after determining the gazed predetermined sub-regions and assigning specific values to each gazed predetermined sub-region, the specific values assigned to the predetermined subregions can be weighted according to specific factors representing the degree or probability of the driver’s gaze falling into the corresponding gazed predetermined sub-regions.

[0037] In one example, for each frame, the following operations are performed to obtain the first value for various predetermined sub-regions: Assign a first specific value to the determined gazed predetermined sub-regions and weight the first specific value with a specific factor, thereby obtaining the first value for each gazed predetermined sub-region, wherein the specific factor is determined based on the first data and represents the degree or probability of the driver’s gaze falling into the corresponding predetermined sub-region; assign a second specific value to the other predetermined sub-regions in the multiple predetermined regions that are not gazed, thereby obtaining the first value for the other predetermined sub-regions. After determining the first value for various predetermined sub-regions for each frame, the first values determined for different frames in the multiple frames of images are summed for each predetermined sub-region to obtain the final output value representing the cumulative gaze for various predetermined sub-regions.

[0038] Still referring to FIG. 2, assume that four frames of images were collected over a period. For the first frame, the determined first data indicates that the driver gazed at predetermined sub-regions a and b with probabilities of 80% and 20%, respectively. After assigning a first specific value of 1 to predetermined sub-regions a and b, they are weighted with 80% and 20%, respectively, resulting in first values of 0.8 and 0.2 for predetermined sub-regions a and b. For the second frame, the determined first data indicates that the driver gazed at predetermined sub-regions a and c with probabilities of 40% and 60%, respectively. After assigning a first specific value of 1 to predetermined sub-regions a and c, they are weighted with 40% and 60%, respectively, resulting in first values of 0.4 and 0.6 for predetermined subregions a and c. For the third frame, the determined first data indicates that the driver gazed at predetermined sub-regions c and d with probabilities of 50% each. After assigning a first specific value of 1 to predetermined sub-regions c and d, they are weighted with 50%, resulting in first values of 0.5 for both predetermined sub-regions c and d. For the fourth frame, the determined first data indicates that the driver only gazed at predetermined sub-region a, so a specific value of 1 is assigned to predetermined sub-region a, and its weighted first value remains 1. Thus, over the period for the four frames of images, the cumulative gaze output value for predetermined sub-region a is 0.8+0.4+1 =2.2; for predetermined sub-region b, the cumulative gaze output value is 0.2; for predetermined sub-region c, the cumulative gaze output value is 0.6+0.5=1.1 ; and for predetermined sub-region d, the cumulative gaze output value is 0.5. Other ungazed predetermined sub-regions can be assigned an output value of, for example, 0. This way, the cumulative gaze output value for various predetermined sub- region over the period can be determined.

[0039] At step 120 in FIG. 3, after determining the output values representing the cumulative gaze of the driver on the respective predetermined sub-regions over a period of time, the driver’s state can be determined at step 130 based on the output values of these respective predetermined sub-regions. Specifically, the driver’s state can be determined based on the magnitude and / or distribution of the output values of various predetermined sub-regions.

[0040] FIGs. 4a-4c illustrate various predetermined sub-regions and their output values obtained by calculation according to different examples of the present disclosure. These are multiple predetermined sub-regions and their output values divided for the automatic lane change function as shown in FIG. 2. Not only are the output values of various predetermined sub-regions shown in grayscale, but the first total output value for the first predetermined region A1 , the second total output value for the second predetermined region A2, and the third total output value for the third predetermined region A3 are also shown. At step 120, the output values of various predetermined sub-regions can be determined, resulting in the views shown in FIGs. 4a-4c. The output values of various predetermined sub-regions may correspond to the grayscale values of various predetermined sub-regions.

[0041] As shown in FIG. 4a, it can be determined that during this period, the driver mainly gazed at the second predetermined region A2 related to lane-keeping. Therefore, it can be determined that the driver’s state remains in the lane-keeping state. As shown in FIG. 4b, it can be determined that during this period, the driver mainly gazed at the first predetermined region A1 related to the left lane change and the second predetermined region A2 related to lane-keeping. Therefore, it can be determined that the driver performed an environment check related to the left lane change. As shown in FIG. 4c, it can be determined that during this period, the driver mainly gazed at the third predetermined region A3 related to the right lane change and the second predetermined region A2 related to lane-keeping. Therefore, it can be determined that the driver performed an environment check related to the right lane change.

[0042] If the vehicle issues an automatic lane change confirmation request during autonomous driving, the above steps 110-130 can be executed within a subsequent period (e.g., 2 seconds, which may vary for different drivers) to determine whether the driver performed the corresponding environment check. If the driver performed the corresponding environment check, the corresponding lane change operation can be executed. This period is the time period used for determining the driver’s state.

[0043] For example, if a left lane change confirmation request is issued during autonomous driving, and the output values of the respective predetermined sub-regions as shown in FIG. 4b are obtained at step 120, then at step 130, it can be confirmed that the driver performed the environment check for the left lane change. Therefore, the left lane change operation can be executed. Although, as shown in FIGs. 4a-4c, the driver’s state can be determined based on the magnitude of the cumulative gaze output values of various predetermined sub-regions and their distribution among the predetermined regions, this is not limiting. In practice, after determining the cumulative gaze output values of various predetermined sub-regions over a period in step 120, these output values can be used in different ways to determine the driver’s state in step 130, not limited to their distribution among various predetermined regions. In some examples, a template specific to the current driver can be introduced to assess the driver’s state more accurately.

[0044] In one example, a template specific to the current driver can be pre-stored / obtained and the driver’s state can be determined based on a comparison between this template and the cumulative gaze output values of various predetermined sub-regions determined in step 120.

[0045] This template can be determined based on the gaze tracking data of the driver over a previous specific period. This specific period can be the same as the period used for determining the driver’s state, allowing the template to be directly determined from the gaze tracking data of that specific period. Of course, the specific period used to determine the template can also be different from the period used for determining the driver’s state. In particular, this specific period can be longer, allowing the template to be determined using data over a longer period, thereby increasing the template’s usability. In this case, gaze tracking data for a specific period can be first obtained, and this specific period can be divided into multiple periods, each of which is the same as the period used for determining the driver’s state. The specific output values of various predetermined sub-regions can be calculated for each period, and then the specific output values of various predetermined sub-regions can be averaged over multiple periods to determine the specific output values of various predetermined sub-regions for one period. Based on these specific output values, a template specific to the current driver can be determined (this can be referred to as a general state template for the current driver, representing the general state of the current driver over a relatively long specific period, as the driver experiences various states during the period used to determine the template. Therefore, this general state template is essentially an average template).

[0046] In one example, this general state template can include specific output values for each predetermined sub-region. For such a template, the driver’s state can be determined by comparing the output values of various predetermined sub-regions determined during the driver state assessment with the specific output values of these predetermined sub-regions included in the template. For example, it can be analyzed whether the distribution of these output values matches, thereby determining the driver’s state. In another example, the general state template can include specific total output values for each predetermined region among multiple predetermined regions. In yet another example, the general state template can include the ratio of the specific total output values for each predetermined region in the sum of the specific output values of all predetermined sub-regions. For example, for the automatic lane change function shown in FIG. 2, the template can include a first specific ratio for the specific total output value of the first predetermined region A1 , a second specific ratio for the specific total output value of the second predetermined region A2, and a third specific ratio for the specific total output value of the third predetermined region A3. In this case, the period corresponding to the general state template can be different from the period used for determining the driver’s state. In addition to the proportions, the template can also include the numerical range of the total output values for the predetermined regions, such as the maximum and / or minimum values of the total output values. In the case of including numerical ranges, the total output values of various predetermined regions used for determining the driver’s state should not only conform to the specific ratios specified by the template but also fall within the numerical range specified by the template.

[0047] FIG. 5 illustrates a flowchart of a method 200 for determining the driver’s state using the general state template according to an example of the present disclosure. The method 200 is particularly described with reference to the automatic lane change function in autonomous driving as described in FIG. 2. The method 200 is executed in step 130 as shown in FIG. 3. According to the method 200, in step 210, a template (general state template) related to the current driver is obtained, which may include at least one of the following: specific output values for each predetermined sub-region, specific total output values for each predetermined region among the multiple predetermined regions, and / or ratios of specific total output values for each predetermined region in the sum of specific output values of all predetermined subregions. In this example, the template includes a first specific ratio for the first predetermined region A1 , a second specific ratio for the second predetermined region A2, and a third specific ratio for the third predetermined region A3. Other forms of templates are also conceivable.

[0048] In step 220, the output values for each predetermined sub-region determined in step 120 of FIG. 3 are received. These output values are determined for a predetermined time period after receiving a lane change confirmation request. The output values for each predetermined sub-region in the first predetermined region A1 are summed to obtain the first total output value for the first predetermined region A1 ; similarly, the output values for each predetermined sub-region in the second predetermined region A2 are summed to obtain the second total output value for the second predetermined region A2; and the output values for each predetermined sub-region in the third predetermined region A3 are summed to obtain the third total output value for the third predetermined region A3. Referring to FIG. 4a, the first total output value for the first specific region A1 is 67, the second total output value for the second specific region A2 is 1403, and the third total output value for the third specific region A3 is 78. Thus, in step 230, the first ratio, second ratio, and third ratio of the first total output value, second total output value, and third total output value to the sum of the output values of all predetermined sub-regions (e.g., 67+1403+78=1548 as shown in FIG. 4a) can be determined.

[0049] Subsequently, in step 240, the first ratio, second ratio, and third ratio are compared with the first specific ratio, second specific ratio, and third specific ratio in the template, respectively. In step 250, the driver’s state can be determined based on the results of the comparison Since the template is determined based on data obtained over a long period of time representing the driver’s cumulative gaze in a general state, when the first ratio, second ratio, and third ratio for the current time period deviate from the first specific ratio, second specific ratio, and / or third specific ratio in the template beyond a predetermined range, it indicates that the driver’s state has changed.

[0050] Specifically, for automatic lane change confirmation, if in step 240 it is determined that the first ratio for the first predetermined region A1 has increased relative to the first specific ratio in the template by more than a specific threshold, e.g., by more than 100%, since the first predetermined region A1 is related to left lane change, it is determined in step 250 that the driver has performed an environment check related to the left lane change; if it is determined in step 240 that the third ratio for the third predetermined region A3 has increased relative to the third specific ratio in the template by more than a specific threshold, e.g., by more than 100%, since the third predetermined region A3 is related to right lane change, it is determined that the driver has performed an environment check related to the right lane change. Although the above description refers to the same specific threshold, it is understood that different thresholds may be used for different predetermined regions.

[0051] The above description assumes that sufficient gaze tracking data is available for the current driver to determine accurate template data. In practice, for example, for a newly purchased vehicle or a vehicle driven by the driver for the first time, it is often difficult to obtain sufficient data to determine a template for the driver.

[0052] In such cases, one approach is to disable the method shown in FIG. 5 for determining the driver’s state until sufficient data is obtained to determine a template for the driver, e.g., by turning off the function or requiring the driver to confirm their state by other means.

[0053] Another approach is to confirm the driver’s lane change state by determining that the driver’s state does not correspond to lane-keeping. This is because, during vehicle driving, even if there is not enough data to determine the output values for the predetermined regions corresponding to left and right lane changes in the template, there is often sufficient data for lane-keeping, i.e., the second specific total output value corresponding to the second predetermined region for lane-keeping in the template is more reliable. According to this approach, in step 220, only the second total output value for the second predetermined region A2 is determined, then step 230 is omitted, and in step 240, the second total output value for the second predetermined region A2 is directly compared with the second specific total output value in the template; if it is determined that the second total output value has decreased relative to the second specific total output value by more than a specific threshold, indicating that the driver may have gazed less at the second predetermined region, it is determined in step 250 that the driver has performed an environment check related to lane change, otherwise, it indicates that the driver is still in a state corresponding to lane-keeping.

[0054] Although the method shown in FIG. 5 has been described with reference to a template including specific ratios for each predetermined region, this is merely illustrative. It is also conceivable that the template includes specific total output values of various predetermined regions, and these specific total output values are compared with the total output values calculated for the corresponding predetermined regions during the driver’s state determination process to determine the driver’s state. However, it should be noted that in this case, the specific time period for obtaining the template data and the time period for determining the driver’s state should be equal. In contrast, when using ratios, the specific time period for obtaining the template data and the time period for determining the driver’s state do not need to be equal, reducing the requirements for template data and improving the usability of the template data. Therefore, a template including ratios for each predetermined region is more desirable.

[0055] The above description has referred to a general state template, but it is also conceivable to obtain or store multiple templates corresponding to different states of the driver. For example, for the automatic lane change confirmation function in autonomous driving, three templates can be included, namely the first template, the second template, and the third template, which are determined based on data when the driver is in different states. The first template corresponds to the state of the driver performing a left lane change environment check, the second template corresponds to the state of the driver performing a lane-keeping environment check, and the third template corresponds to the state of the driver performing a right lane change environment check. The data included in these templates can be at least one of the following: specific output values for each predetermined sub-region, specific total output values for each predetermined region among the multiple predetermined regions, and / or ratios of specific total output values for each predetermined region in the sum of specific output values of all predetermined sub-regions.

[0056] After obtaining the above templates for different states, the driver’s state can be determined by judging which template the output values for each predetermined sub-region calculated in step 120 of FIG. 3 for driver state confirmation match. For example, if it is determined that the output values for each predetermined sub-region match the first template corresponding to the left lane change environment check, it is determined that the driver’s state is the left lane change environment check. Although the method of the present disclosure has been mainly described with reference to FIGS. 3 and 5, this is not limiting. For example, the method according to various examples of the present disclosure may also include steps for determining and / or updating the template in real-time. In one example, gaze tracking data for a specific time period of the driver can be obtained; and the template is determined and / or updated based on the gaze tracking data for the specific time period. In another example, gaze tracking data for a specific time period of the driver in different states can be obtained; and the corresponding template among the multiple templates is determined and / or updated based on the gaze tracking data for the specific time period in each state.

[0057] FIG. 6 illustrates a block diagram of an apparatus 10 for driver monitoring according to one example of the present disclosure. As shown in FIG. 6, the apparatus 10 includes an obtaining unit 11 , a calculating unit 12, and a determining unit 13. Optionally, the apparatus 10 may also include a storage unit that stores one or more templates.

[0058] The obtaining unit 11 acquires gaze tracking data for the driver; this gaze tracking data may consist of continuous multiple frames of images captured for the driver, and may also include first data representing the position of the driver’s gaze derived from the captured multiple frames of images and second data representing the duration of the driver’s gaze at the corresponding position.

[0059] The calculating unit 12 calculates, based on the acquired gaze tracking data, the output value representing the cumulative gaze of the driver on the respective predetermined subregions within each predetermined region over a period of time. These multiple predetermined regions represent non-overlapping areas of interest during vehicle operation, and each predetermined region is divided into multiple non-overlapping predetermined sub-regions, as shown in the division method in FIG. 2. The determining unit 13 determines the driver’s state based on the output values of various predetermined sub-regions within each predetermined region.

[0060] In one example, the gaze tracking data includes first data representing the position of the driver’s gaze and second data representing the duration of the driver’s gaze at the corresponding position. In this case, the calculating unit 12 can calculate the output value for each predetermined sub-region based on the first data and the second data.

[0061] In another example, the gaze tracking data includes multiple frames of images collected for the driver over a period of time, where the calculating unit can determine the first data representing the position of the driver’s gaze for each frame of the image and, based on this first data, determine the gazed predetermined sub-region among the multiple predetermined regions. Furthermore, based on the gazed predetermined sub-region determined for each frame of the multiple frames of images, the calculating unit determines the output values for various predetermined sub-region of the multiple frames of images. In one example, the obtaining unit 11 is further configured to obtain templates related to the driver, such as a template corresponding to the general state of the driver (mean template), which includes at least one of the following: specific output value for each predetermined subregion, specific total output value for each predetermined region among the multiple predetermined regions, and / or the ratio of the specific total output value for each predetermined region to the sum of the specific output values of all predetermined sub-regions. The determining unit 13 can determine the driver’s state based on this template and the output value of each predetermined sub-region.

[0062] In another example, the obtaining unit 11 is further configured to obtain multiple templates corresponding to different states of the driver, each template among the multiple templates includes at least one of the following: specific output value for each predetermined sub-region, specific total output value for each predetermined region among the multiple predetermined regions, and / or the ratio of the specific total output value for each predetermined region to the sum of the specific output values of all predetermined sub-regions. The determining unit 13 can determine which template among the multiple templates the output value for each predetermined sub-region conforms to, and determine the driver’s state based on the judgment result.

[0063] Although not described in detail, it is understood that the obtaining unit 11 performs the function of acquiring various data and / or templates, the calculating unit 12 performs the function of calculating the cumulative gaze output value for each predetermined sub-region over a period of time, such as step 120 in the aforementioned method 100; the determining unit 13 performs the function of determining the driver’s state based on the calculated output value of each predetermined sub-region, such as step 130 in the aforementioned method 100.

[0064] The above description of the various examples of the method and apparatus of the present disclosure with reference to FIGs. 1-6 can be combined with each other to achieve different effects, without being limited by the type of subject matter. Furthermore, the aforementioned units / steps / processes are not restrictive, and the functions of the aforementioned units / steps / processes can be merged / combined / altered / modified to achieve the corresponding effects.

[0065] The method and apparatus of the present disclosure can be executed by the vehicle’s electronic control unit. The functions of the various units of the method and system of the present disclosure can be implemented by software or corresponding hardware, or the functions of the aforementioned units can be implemented with the aid of a processor, for example, the processor can read computer programs stored in memory and run these computer programs to implement the functions of the aforementioned units. In an example, the above device of the present disclosure may also be implemented by a memory and a processor. It should be understood that the method according to the various examples of the present disclosure can be achieved by computer programs / software. These software programs can be loaded into the working memory of the processor and, when executed, perform the methods according to the various examples of the present disclosure.

[0066] The exemplary examples of the present disclosure cover both: Creating / using the computer programs / software of the present disclosure from the beginning, and updating existing programs / software to use the computer programs / software of the present disclosure.

[0067] According to another example of the present disclosure, a computer program product is provided, such as a machine (e.g., computer) readable medium like a CD-ROM, which includes computer program code that, when executed, causes the computer or processor to perform the methods according to the various examples of the present disclosure. The machine-readable medium can be, for example, an optical storage medium or solid-state medium supplied with or as part of other hardware.

[0068] The above description has been given for specific examples of the present disclosure. Other examples are within the scope of the appended Claims. In some cases, actions or steps described in the Claims can be performed in a different order than that of the examples and still achieve the desired results. Additionally, the processes depicted in the drawings do not necessarily require specific or continuous sequences to achieve the desired results. In some examples, multitasking and parallel processing are also possible or advantageous.

[0069] The present disclosure has been described with reference to specific examples, and those skilled in the art should understand that various implementations of the technical solutions of the present disclosure can be made without departing from the spirit and basic features of the present disclosure. Specific examples are merely illustrative and not limiting. Additionally, these examples can be combined in any way to achieve the purposes of the present disclosure. The scope of protection of the present disclosure is defined by the appended claims.

[0070] The term “comprising” in the specification and claims does not exclude the presence of other elements or steps, and the expressions “first,” “second,” “third,” and the order of the steps mentioned and shown in the figures do not limit their sequence or quantity. The functions of the various elements described in the specification or recited in the claims can also be separated or combined and implemented by corresponding multiple elements or a single element.

Claims

Claims1. A method for driver monitoring, comprising: obtaining gaze tracking data for the driver; calculating, based on the gaze tracking data, an output value representing the cumulative gaze of the driver on various predetermined sub-regions within a time period for the corresponding predetermined sub-region of each predetermined region among multiple predetermined regions, where the multiple predetermined regions represent non-overlapping areas of interest during vehicle driving, and each predetermined region is divided into multiple non-overlapping predetermined sub-regions; and determining the driver’s state based on the output values of various predetermined subregions within each predetermined region.

2. The method according to Claim 1 , wherein the gaze tracking data includes first data representing the position of the driver’s gaze and second data representing a duration of the driver’s gaze at the corresponding position, the first data and the second data being determined based on multiple frames of images collected for the driver, the method further comprising: calculating the output value for various predetermined sub-regions based on the first data and the second data.

3. The method according to Claim 1 , wherein the gaze tracking data includes multiple frames of images collected for the driver within the time period, the method further comprising: for each frame of image, determining first data representing the position of the driver’s gaze and determining the gazed predetermined sub-region among the multiple predetermined regions based on the first data; and determining the output value for various predetermined sub-regions of the multiple frames of images based on the gazed predetermined sub-region determined for each frame of image of the multiple frames of image.

4. The method according to Claim 3, wherein the output value for various predetermined sub-regions of the multiple frames of images is determined as follows: for each frame of image, performing the following operations to obtain a first value for various predetermined sub-regions of each predetermined region among the multiple predetermined regions: assigning a first specific value to the determined gazed predetermined sub-region and weighting the first specific value with a specific factor, thereby obtaining the first value foreach gazed predetermined sub-region, wherein the specific factor is determined based on the first data and represents the degree or probability that the driver’s gaze falls into the corresponding predetermined sub-region; and assigning a second specific value to other predetermined sub-regions of the multiple predetermined regions except the gazed predetermined sub-region, thereby obtaining the first value for the other predetermined sub-regions; and summing the first values determined for different frames of images for various predetermined sub-regions of the multiple predetermined sub-regions to obtain the output value for various predetermined sub-regions of the multiple predetermined sub-regions.

5. The method according to Claim 1 , further comprising: obtaining a template related to the driver, the template including at least one of: specific output values for each predetermined sub-region, specific total output values for each predetermined region among the multiple predetermined regions, and / or ratios of specific total output values for each predetermined region in the sum of specific output values of all predetermined sub-regions; and determining the driver’s state based on the template and the output values of each respective predetermined sub-region.

6. The method according to Claim 5, wherein the multiple predetermined regions include a first predetermined region, a second predetermined region, and a third predetermined region, the first predetermined region includes an area related to left lane change of the vehicle, the second predetermined region includes an area related to lane-keeping of the vehicle, the third predetermined region includes an area related to right lane change of the vehicle, the time period is a predetermined time period after receiving a lane change confirmation request from the vehicle, and the template includes a first specific ratio for the first predetermined region, a second specific ratio for the second predetermined region, and a third specific ratio for the third predetermined region, the method further comprising: summing the output values for various predetermined sub-regions of the first predetermined region to obtain a first total output value for the first predetermined region; summing the output values for various predetermined sub-regions of the second predetermined region to obtain a second total output value for the second predetermined region; summing the output values for various predetermined sub-regions of the third predetermined region to obtain a third total output value for the third predetermined region; respectively determining the first ratio, the second ratio, and the third ratio of the first total output value, the second total output value, and the third total output value in the sum of theoutput values of all predetermined sub-regions; comparing the first ratio, the second ratio, and the third ratio with the first specific ratio, the second specific ratio, and the third specific ratio, respectively; if it is determined that the first ratio increases beyond a specific threshold relative to the first specific ratio, determining that the driver has performed an environment check related to a left lane change; and if it is determined that the third ratio increases beyond a specific threshold relative to the third specific ratio, determining that the driver has performed an environment check related to a right lane change.

7. The method according to Claim 5, wherein the multiple predetermined regions include a first predetermined region, a second predetermined region, and a third predetermined region, the first predetermined region includes an area related to left lane change of the vehicle, the second predetermined region includes an area related to lane-keeping of the vehicle, the third predetermined region includes an area related to right lane change of the vehicle, the time period is a predetermined time period after receiving a lane change confirmation request from the vehicle, and the template includes a second specific total output value for the second predetermined region, the method further comprising: summing the output values for various predetermined sub-regions of the second predetermined region to obtain a second total output value for the second predetermined region; comparing the second total output value with the second specific total output value; and if it is determined that the second total output value decreases beyond a specific threshold relative to the second specific total output value, determining that the driver has performed an environment check related to lane change.

8. The method according to Claim 5, further comprising: obtaining multiple templates corresponding to different states of the driver, each template among the multiple templates including at least one of: specific output values for each predetermined sub-region, specific total output values for each predetermined region among the multiple predetermined regions, and / or ratios of specific total output values for each predetermined region in the sum of specific output values of all predetermined sub-regions; determining which template among the multiple templates the output values for each predetermined sub-region match; and determining the driver’s state based on the result of the determination.

9. The method according to Claim 5 or 8, further comprising:determining and / or updating the template in real-time.

10. An apparatus for driver monitoring, comprising: an obtaining unit configured to obtain gaze tracking data for the driver; a calculating unit configured to calculate, based on the gaze tracking data, an output value representing the cumulative gaze of the driver on various predetermined sub-regions within a time period for the corresponding predetermined sub-region of each predetermined region among multiple predetermined regions, where the multiple predetermined regions represent non-overlapping areas of interest during vehicle driving, and each predetermined region is divided into multiple non-overlapping predetermined sub-regions; and a determining unit configured to determine the driver’s state based on the output values of various predetermined sub-regions within each predetermined region.

11. The apparatus according to Claim 10, wherein the gaze tracking data includes first data representing the position of the driver’s gaze and second data representing the duration of the driver’s gaze at the corresponding position, the first data and the second data being determined based on multiple frames of images collected for the driver, and wherein: the calculating unit calculates the output value for various predetermined sub-regions based on the first data and the second data.

12. The apparatus according to Claim 10, wherein the gaze tracking data includes multiple frames of images collected for the driver within the time period, wherein the calculating unit is further configured to: for each frame of image, determine first data representing the position of the driver’s gaze and determining the gazed predetermined sub-region among the multiple predetermined regions based on the first data; and determining the output value for various predetermined sub-regions of the multiple frames of images based on the gazed predetermined sub-region determined for each frame of image of the multiple frames of image.

13. The apparatus according to Claim 10, wherein: the obtaining unit is further configured to obtain a template related to the driver, the template including at least one of: specific output values for each predetermined sub-region, specific total output values for each predetermined region among the multiple predetermined regions, and / or ratios of specific total output values for each predetermined region in the sum of specific output values of all predetermined sub-regions; and the determining unit determines the driver’s state based on the template and the outputvalues of each respective predetermined sub-region.

14. The apparatus according to Claim 13, wherein: the obtaining unit is further configured to obtain multiple templates corresponding to different states of the driver, each template among the multiple templates including at least one of: specific output values for each predetermined sub-region, specific total output values for each predetermined region among the multiple predetermined regions, and / or ratios of specific total output values for each predetermined region in the sum of specific output values of all predetermined sub-regions; and the determining unit is further configured to determine which template among the multiple templates the output values for each predetermined sub-region match, and determine the driver’s state based on the result of the determination.

15. A computer-readable medium storing a computer program, the computer program, when executed by a processor, causing the processor to perform the method according to any one of Claims 1-9.

Images