Driver monitoring method and system

Abstract

A method for driver monitoring is provided. This method includes acquiring driver eye-tracking data, calculating output values ​​based on the eye-tracking data that represent the driver's cumulative gaze over various predetermined sub-regions within a given time period for each of a plurality of predetermined regions, wherein the plurality of predetermined regions represent non-overlapping regions of interest during vehicle operation, and each predetermined region is divided into a plurality of non-overlapping predetermined sub-regions, and determining the driver's state based on the output values ​​of various predetermined sub-regions within each predetermined region. This improves the ease of use and performance of the driver monitoring function.

Description

Technical Field

[0001] The present disclosure relates to the field of driver monitoring, and more specifically to driver eye tracking.

Background Art

[0002] A driver monitoring system is a real-time system based on processing a driver's facial images, physiological indicators, or vehicle information to determine the driver's state. These systems can implement functions such as driver identification, fatigue monitoring, distracted driving monitoring, and monitoring of dangerous driving behaviors (e.g., drunk driving, holding a phone while driving, and drinking water while driving, etc.). Conventional driver monitoring systems incorporate eye tracking technology, which can detect the driver's eye movements and head angle to determine the driver's state and achieve real-time driver monitoring.

[0003] As one application of driver monitoring systems, eye tracking technology can be used to monitor whether a driver checks the surrounding environment and confirms a lane change after the automatic lane change system outputs an indication that a lane change is allowable. The automatic lane change function is provided at Level 2 of the Society of Automotive Engineers (SAE) autonomous driving levels. According to this function, when the vehicle detects that a lane change is possible, the vehicle first displays information to the driver indicating that a lane change is allowable. An actual lane change is only performed after the driver has checked the surrounding environment and confirmed that the lane change is appropriate. However, it is generally difficult to determine whether the driver has actually checked the surrounding environment. In some cases, the driver may excessively trust the system and not truly check whether the surrounding environment is suitable for a lane change, which may lead to irreparable damage. Therefore, in driver monitoring systems, eye tracking technology is used to detect whether the driver gazes at a predetermined area and the duration of the gaze, so as to determine whether the driver has actually confirmed the lane change.

[0004] According to the method described above, strict rules must be defined in advance, and then it is determined whether the driver has confirmed the lane change based on these rules. For example, in the case of a left lane change, the designated area may be the area corresponding to the vehicle's left rearview mirror, and in the case of a right lane change, the designated area may be the area corresponding to the vehicle's right rearview mirror, and the gaze duration threshold may be 300 to 500 milliseconds. The driver is determined to have confirmed the left lane change only if it is detected that the driver gazed at the area corresponding to the left rearview mirror for a duration exceeding the threshold. Similarly, in the case of a right lane change, the driver is determined to have confirmed the right lane change only if it is detected that the driver gazed at the area corresponding to the right rearview mirror for a duration exceeding the threshold. [Overview of the project] [Problems that the invention aims to solve]

[0005] It is desirable to provide improved driver monitoring methods and systems that do not require strictly predefined rules, are applicable to different drivers, and enhance the usefulness and performance of driver monitoring functions. [Means for solving the problem]

[0006] According to one embodiment, a method for driver monitoring is provided, which includes acquiring driver eye-tracking data, calculating an output value based on the eye-tracking data that represents the driver's cumulative gaze over various predetermined sub-regions within a given time for each of a plurality of predetermined regions, wherein the plurality of predetermined regions represent non-overlapping regions of interest during vehicle operation, and each predetermined region is divided into a plurality of non-overlapping predetermined sub-regions, and determining the driver's state based on the output values ​​of various predetermined sub-regions within each predetermined region.

[0007] In another embodiment, a device for driver monitoring is provided, comprising: an acquisition unit configured to acquire driver eye-tracking data; a calculation unit configured to calculate, based on the eye-tracking data, an output value representing the driver's cumulative gaze over various predetermined sub-regions within a given time for each of a plurality of predetermined regions, wherein the plurality of predetermined regions represent non-overlapping regions of interest during vehicle operation, and each predetermined region is divided into a plurality of non-overlapping predetermined sub-regions; and a determination unit configured to determine the driver's state based on the output values ​​of various predetermined sub-regions within each predetermined region.

[0008] In yet another embodiment, a computer-readable medium is provided which stores a computer program, and which causes the processor to carry out the methods according to various embodiments of the present disclosure when the computer program is executed by the processor.

[0009] According to various embodiments of various aspects of this disclosure, strictly defined areas corresponding to specific vehicle components are no longer used. Instead, the driver's area of ​​interest while driving the vehicle is divided into a plurality of non-overlapping predetermined areas, each predetermined area being further divided into a plurality of non-overlapping predetermined sub-areas. Next, an output value representing the driver's cumulative gaze over each predetermined sub-area over a certain period of time is calculated for each predetermined sub-area. Based on this, different methods for tracking the driver's gaze are provided, enabling the driver monitoring system to interpret the gaze tracking data in different ways. This approach eliminates the need to strictly define areas corresponding to different vehicle components and may allow for larger predetermined areas without the system's performance being affected by individual differences between drivers. For example, because drivers' heights vary and their rearview mirror settings differ, strictly defining areas corresponding to rearview mirrors may not be applicable to all drivers. By using the methods of various embodiments of this disclosure, the need to strictly define areas corresponding to specific vehicle components is eliminated. Larger predetermined areas can be set, which are not affected by individual differences between drivers, thereby increasing the usefulness of the driver monitoring function.

[0010] In one example, a template related to the driver can be obtained, and the driver's state is determined based on the templates and calculated output values ​​for each predetermined sub-region. In one embodiment, the template may consist of multiple templates corresponding to different driver states. By using driver-specific templates, customized driver state monitoring is achieved, improving monitoring accuracy and providing a better user experience.

[0011] In the drawings, embodiments are shown as examples, not as limitations, and similar reference numerals in the drawings refer to similar elements. [Brief explanation of the drawing]

[0012] [Figure 1] This shows a method for dividing the area for driver monitoring using prior art. [Figure 2] This invention illustrates a method for segmenting a region for driver monitoring according to one embodiment of this disclosure. [Figure 3] A flowchart of a method for driver monitoring according to one embodiment of the present disclosure is shown. [Figure 4] Figures 4a to 4c show various predetermined subregions and their output values ​​obtained according to different embodiments of this disclosure. [Figure 5] This disclosure describes a method for determining the driver's condition based on output values ​​in various predetermined sub-regions, according to an embodiment of this disclosure. [Figure 6] This shows a block diagram of a device for driver monitoring according to one embodiment of the present disclosure. [Modes for carrying out the invention]

[0013] Various aspects and configurations of the embodiments of this disclosure will be described with reference to the drawings above. The drawings above are for illustrative purposes only and are not limiting. Furthermore, not all parts of the apparatus according to the embodiments of this disclosure are indicated by reference numbers in the accompanying drawings above, and only relevant components are shown in some of the accompanying drawings. This does not mean that the various parts are limited to those shown in the accompanying drawings of this specification.

[0014] In the field of autonomous driving, certain application scenarios require the use of a driver monitoring system to monitor the driver's condition and then determine how to control the vehicle's actions. For example, according to the automatic lane change function provided by the Society of Automotive Engineers (SAE) Level 2 for autonomous driving, after the vehicle detects that a lane change is possible, it must first send a lane change confirmation request to the driver. The actual lane change operation is only performed after the driver has checked the surrounding environment and confirmed that the lane change is safe. In this case, a driver monitoring system can be used to monitor whether the driver has actually checked the surrounding environment.

[0015] In conventional methods, driver monitoring systems monitor whether the driver has truly checked the surrounding environment based on strictly defined rules. As shown in Figure 1, conventional methods track the driver's gaze to determine which areas 1, 2, and 3 the driver is looking at and for how long, thereby monitoring the driver's state. For example, if the driver gazes at area 1 for more than 500 milliseconds, it indicates that the driver has confirmed the execution of a left lane change. According to conventional methods, areas 1, 2, and 3 are strictly defined areas corresponding to the left mirror, center mirror, and right mirror of the vehicle, respectively. However, different drivers have different driving habits, and mirror settings may vary depending on their personal circumstances. Furthermore, conventional gaze tracking technology has limited accuracy for small areas. Therefore, the aforementioned strictly defined rules do not apply to all drivers and cannot ensure performance monitoring for different drivers.

[0016] According to various embodiments of this disclosure, strictly defined areas corresponding to specific vehicle components are no longer used. Instead, the driver's area of ​​interest while driving the vehicle is divided into a plurality of non-overlapping predetermined areas, each predetermined area being further divided into a plurality of non-overlapping predetermined sub-areas. Next, an output value representing the driver's cumulative gaze over each predetermined sub-area over a certain period of time is calculated for each predetermined sub-area. Based on this, different methods for tracking the driver's gaze are provided, and the driver monitoring system is able to interpret the gaze tracking data in different ways. This approach no longer requires strictly defined areas corresponding to the left mirror, center mirror, and right mirror (areas 1, 2, and 3). Larger predetermined areas can be set, and the performance of the system is not affected by individual differences between drivers.

[0017] As shown in Figure 2, the driver's area of ​​interest during operation of a vehicle, including the front and side windows, is divided into a first predetermined area A1, a second predetermined area A2, and a third predetermined area A3. The first predetermined area includes the area related to changing the vehicle's left lane, the second predetermined area includes the area related to lane keeping, and the third predetermined area includes the area related to changing the vehicle's right lane. For example, the first predetermined area A1 includes the area corresponding to the left mirror, the second predetermined area A2 includes the area corresponding to the center mirror, and the third predetermined area A3 includes the area corresponding to the right mirror. Although the first predetermined area is described as being related to changing the vehicle's left lane, it should be understood that this does not mean that changing the left lane is only related to the first predetermined area. For example, while changing the left lane, the driver may also look at the center mirror, so that changing the left lane may also be related to the second predetermined area. Similarly, this does not mean that lane keeping is relevant only to a second designated area, or that changing lanes to the right is relevant only to a third designated area.

[0018] Each predetermined region A1, A2, or A3 is further divided into multiple non-overlapping predetermined subregions such as a, b, c, and d. In Figure 2, these predetermined subregions are shown as uniformly equal squares, but this is not limiting. They can be divided in any other way so that the shape and / or size of these predetermined subregions can be arbitrarily selected. Based on the division of the predetermined regions and their predetermined subregions, an output value representing the driver's cumulative gaze over that predetermined subregion over a certain period can be determined for each predetermined subregion. This makes it possible to determine the driver's gaze distribution across various predetermined regions over that period, and thereby, the driver's state.

[0019] FIG. 3 shows a flowchart 100 of a method for driver monitoring according to an embodiment of the present disclosure. As shown in FIG. 3, according to the method 100, first, in step 110, gaze tracking data about a driver is received. In one example, the gaze tracking data may comprise a plurality of frames of images collected for tracking the movement of the driver's eyes, and may also be configured to include detection data for the driver's head angle. In other embodiments, the gaze tracking data includes first data representing the position of the driver's line of sight, and second data representing the duration of the driver's gaze at the corresponding position. The first data and the second data may be derived from a plurality of frames of images collected for the driver and / or head angle detection data.

[0020] Next, in step 120, based on the plurality of predetermined regions and the plurality of predetermined sub-regions divided in advance as described above, an output value representing the cumulative gaze of the driver on each predetermined sub-region over a period of time is calculated. For example, this output value can be calculated using a mathematical expected value.

[0021] In one example, when the received gaze tracking data includes a plurality of frames of images collected for the driver, first data representing the position of the driver's line of sight may be determined for each frame. Based on this first data, it can be determined which of the plurality of predetermined sub-regions is gazed by the driver for that frame. Then, the cumulative gaze output value for each predetermined sub-region can be determined based on the gazed predetermined sub-region for each frame.

[0022] For example, assume that four image frames are collected over a certain period of time. Referring to FIG. 2, for the first frame, the determined first data indicates that the driver gazes at predetermined sub-regions a and b, and a specific value such as 1 can be assigned to the predetermined sub-regions a and b. For the second frame, the determined first data indicates that the driver gazes at the predetermined sub-regions a and c, therefore, a specific value of 1 can be assigned to the predetermined sub-regions a and c. For the third frame, the determined first data indicates that the driver gazes at the predetermined sub-regions c and d, therefore, a specific value of 1 can be assigned to the predetermined sub-regions c and d. For the fourth frame, the determined first data indicates that the driver gazes at the predetermined sub-region a, therefore, a specific value of 1 can be assigned to the predetermined sub-region a. Accordingly, over the period of the four image frames, the cumulative gaze output value of the predetermined sub-region a is 1+1+1=3, for the predetermined sub-regions b and d, the cumulative gaze output values are 1 respectively, and for the predetermined sub-region c, the cumulative gaze output value is 1+1=2. Other non-gazed predetermined sub-regions may be assigned an output value of 0, for example. In this way, the cumulative gaze output values of various predetermined sub-regions over the period of time can be determined.

[0023] The above embodiment describes determining the output value of each predetermined sub-region based on a plurality of frames of an image with reference to FIG. 2, but this is not limiting. When the received gaze tracking data includes first data representing the position of the driver's line of sight at the corresponding position and second data representing the duration of the driver's gaze, the output values of various predetermined sub-regions over a period of time can also be determined based on the first and second data. In one example, the second data representing the gaze duration may be, for example, the number of frames that the driver gazes at the corresponding position. The cumulative gaze output value of each predetermined sub-region can be determined based on which predetermined sub-region the driver gazed at and the number of frames gazed at each specific predetermined sub-region.

[0024] Furthermore, the calculation of the cumulative gaze output value for each predetermined sub-region can be further refined by introducing the degree or probability of the driver's gaze entering the corresponding predetermined sub-region. Specifically, for each frame, the predetermined sub-regions that were gazed upon are determined, specific values ​​are assigned to each gazed-up predetermined sub-region, and then the specific values ​​assigned to the predetermined sub-regions are weighted according to a specific coefficient representing the degree or probability of the driver's gaze entering the corresponding gazed-up predetermined sub-region.

[0025] In one example, for each frame, the following operations are performed to obtain first values ​​for various predetermined sub-regions: assigning a first specific value to a determined gazed-on predetermined sub-region, weighting the first specific value by a specific coefficient, thereby obtaining a first value for each gazed-on predetermined sub-region, where the specific coefficient is determined based on first data and represents the degree or probability of the driver's gaze entering the corresponding predetermined sub-region; and assigning a second specific value to other predetermined sub-regions within a plurality of predetermined regions that are not gazed on, thereby obtaining first values ​​for the other predetermined sub-regions. After determining first values ​​for various predetermined sub-regions for each frame, the first values ​​determined for different frames within a plurality of frames of the image are summed for each predetermined sub-region to obtain a final output value representing cumulative gaze for the various predetermined sub-regions.

[0026] Referring further to Figure 2, let's assume that four image frames were collected over a certain period of time. For the first frame, the first data determined indicates that the driver gazed at predetermined sub-regions a and b with a probability of 80% and 20%, respectively. After assigning a first specific value 1 to the predetermined sub-regions a and b, they are weighted 80% and 20% respectively, resulting in first values ​​of 0.8 and 0.2 for the predetermined sub-regions a and b. For the second frame, the first data determined indicates that the driver gazed at predetermined sub-regions a and c with a probability of 40% and 60%, respectively. After assigning a first specific value 1 to the predetermined sub-regions a and c, they are weighted 40% and 60% respectively, resulting in first values ​​of 0.4 and 0.6 for the predetermined sub-regions a and c. For the third frame, the first data determined indicates that the driver gazed at predetermined sub-regions c and d with a probability of 50%, respectively. After assigning a first specific value of 1 to predetermined sub-regions c and d, they are weighted 50% to yield a first value of 0.5 for both predetermined sub-regions c and d. For the fourth frame, the determined first data indicates that the driver gazed only at predetermined sub-region a, and therefore a specific value of 1 is assigned to predetermined sub-region a, and its weighted first value remains 1. Thus, over the period of four frames of the image, the cumulative gaze output value for predetermined sub-region a is 0.8 + 0.4 + 1 = 2.2, for predetermined sub-region b it is 0.2, for predetermined sub-region c it is 0.6 + 0.5 = 1.1, and for predetermined sub-region d it is 0.5. Other unchanged predetermined sub-regions can be assigned an output value of, for example, 0. In this way, the cumulative gaze output values ​​for various predetermined sub-regions over a period can be determined.

[0027] In step 120 of Figure 3, after determining the output value representing the driver's cumulative gaze over each predetermined sub-region over a certain period, the driver's state can be determined in step 130 based on the output values ​​of each of these predetermined sub-regions. Specifically, the driver's state can be determined based on the magnitude and / or distribution of the output values ​​of various predetermined sub-regions.

[0028] Figures 4a to 4c show various predetermined sub-regions and their output values ​​obtained by calculations according to different embodiments of the present disclosure. These are multiple predetermined sub-regions, and their output values ​​are divided for the automatic lane changing function, as shown in Figure 2. The output values ​​of the various predetermined sub-regions are shown not only in grayscale, but also the first total output value for the first predetermined region A1, the second total output value for the second predetermined region A2, and the third total output value for the third predetermined region A3. In step 120, the output values ​​of the various predetermined sub-regions are determined, and the figures shown in Figures 4a to 4c can be obtained. The output values ​​of the various predetermined sub-regions may be configured to correspond to the grayscale values ​​of the various predetermined sub-regions.

[0029] As shown in Figure 4a, the system may be configured to determine that during this period, the driver primarily focused on a second predetermined area A2 related to lane keeping. Therefore, it can be determined that the driver's state remained in a lane-keeping state. As shown in Figure 4b, it can be determined that during this period, the driver primarily focused on a first predetermined area A1 related to left lane changes and a second predetermined area A2 related to lane keeping. Therefore, it can be determined that the driver performed an environmental check related to left lane changes. As shown in Figure 4c, it can be determined that during this period, the driver primarily focused on a third predetermined area A3 related to right lane changes and a second predetermined area A2 related to lane keeping. Therefore, it can be determined that the driver performed an environmental check related to right lane changes.

[0030] If the vehicle issues an automatic lane change confirmation request while autonomous driving is in progress, steps 110-130 above may be performed within a subsequent period (e.g., 2 seconds, which may vary for different drivers) to determine whether the driver has performed the corresponding environmental check. If the driver has performed the corresponding environmental check, the corresponding lane change operation may be performed. This period is used to determine the driver's status.

[0031] For example, if a left lane change confirmation request is issued during automated driving, and the output values ​​of each predetermined sub-region shown in Figure 4b are obtained in step 120, then in step 130, it can be confirmed that the driver has performed an environmental check for a left lane change. Therefore, the left lane change operation can be performed.

[0032] As shown in Figures 4a to 4c, the driver's state can be determined based on, but is not limited to, the magnitude of the cumulative gaze output values ​​in various predetermined sub-regions and their distribution among the predetermined regions. In practice, after determining the cumulative gaze output values ​​in various predetermined sub-regions over a period in step 120, these output values ​​can be used in different ways to determine the driver's state in step 130, not limited to their distribution among the various predetermined regions. In some embodiments, a template specific to the current driver can be introduced to more accurately assess the driver's state.

[0033] In one example, a template specific to the current driver can be saved / retrieved in advance, and the driver's status can be determined based on a comparison of this template with the cumulative gaze output values ​​of various predetermined sub-regions determined in step 120.

[0034] This template can be determined based on driver eye-tracking data over a specific period. This specific period may be the same as the period used to determine the driver's condition, allowing the template to be determined directly from the eye-tracking data for that specific period. Of course, the specific period used to determine the template may also be different from the period used to determine the driver's condition. In particular, this specific period can be longer, allowing the template to be determined using data over a longer period, thereby increasing the usefulness of the template. In this case, eye-tracking data for a specific period can be obtained first, and this specific period can be divided into multiple periods, each of which is the same as the period used to determine the driver's condition. Specific output values ​​for various predetermined sub-regions can be calculated for each period, and then the specific output values ​​for various predetermined sub-regions can be averaged over multiple periods to determine the specific output values ​​for various predetermined sub-regions for a single period. Based on these specific output values, a template specific to the current driver can be determined (this may also be called the driver's general state template, which represents the driver's general state over a relatively long period, as the driver experiences a variety of states during the period used to determine the template. Therefore, this general state template is essentially an average template).

[0035] In one embodiment, the general state template may be configured to include specific output values ​​for each predetermined sub-region. For such a template, the driver state can be determined by comparing the output values ​​of various predetermined sub-regions determined during the driver state evaluation with the specific output values ​​of these predetermined sub-regions included in the template. For example, the driver state can be determined by analyzing whether the distribution of these output values ​​matches. In another embodiment, the general state template may be configured to include specific total output values ​​for each predetermined region among a plurality of predetermined regions. In yet another embodiment, the general state template may be configured to include the ratio of the specific total output value for each predetermined region to the sum of the specific output values ​​of all predetermined sub-regions. For example, for the automatic lane change function shown in Figure 2, the template may be configured to include a first specific ratio to a specific total output value for a first predetermined region A1, a second specific ratio to a specific total output value for a second predetermined region A2, and a third specific ratio to a specific total output value for a third predetermined region A3. In this case, the period corresponding to the general state template may be different from the period used to determine the driver state. In addition to ratios, templates may also be configured to include numerical ranges of total output values ​​for a given area, such as the maximum and / or minimum total output values. If numerical ranges are included, the total output values ​​for various given areas used to determine the driver condition will not only fit the specific ratios specified by the template, but will also fall within the numerical ranges specified by the template.

[0036] Figure 5 shows a flowchart of Method 200 for determining the driver's state using a general state template, according to an embodiment of the present disclosure. Method 200 will be described in more detail with reference to the automatic lane-changing function in autonomous driving described in Figure 2. Method 200 is performed in step 130, as shown in Figure 3. According to Method 200, in step 210, a template (general state template) related to the current driver is obtained, which may be configured to include at least one of the following: a specific output value for each predetermined sub-region, a specific total output value for each predetermined region among a plurality of predetermined regions, and / or the ratio of the specific total output value for each predetermined region to the sum of the specific output values ​​for all predetermined sub-regions. In this example, the template includes a first specific ratio for a first predetermined region A1, a second specific ratio for a second predetermined region A2, and a third specific ratio for a third predetermined region A3. Other forms of templates are also conceivable.

[0037] In step 220, the output values ​​for each predetermined sub-region determined in step 120 of Figure 3 are received. These output values ​​are determined for a predetermined period after the lane change confirmation request is received. The output values ​​for each predetermined sub-region of the first predetermined region A1 are summed to obtain a first total output value for the first predetermined region A1. Similarly, the output values ​​for each predetermined sub-region of the second predetermined region A2 are summed to obtain a second total output value for the second predetermined region A2, and the output values ​​for each predetermined sub-region of the third predetermined region A3 are summed to obtain a third total output value for the third predetermined region A3. Referring to Figure 4a, the first total output value for the first specific region A1 is 67, the second total output value for the second specific region A2 is 1403, and the third total output value for the third specific region A3 is 78. Therefore, in step 230, the first, second, and third ratios of the first total output value, the second total output value, and the third total output value can be determined with respect to the sum of the output values ​​of all predetermined sub-regions (for example, 67 + 1403 + 78 = 1548 as shown in Figure 4a).

[0038] Subsequently, in step 240, the first ratio, the second ratio, and the third ratio are compared to the first specific ratio, the second specific ratio, and the third specific ratio in the template, respectively. In step 250, the driver's condition can be determined based on the results of the comparison. Since the template is determined based on data acquired over a long period representing the driver's cumulative gaze in a typical state, when the first ratio, the second ratio, and the third ratio for the current period deviate from the first specific ratio, the second specific ratio, and / or the third specific ratio in the template by a predetermined range, it indicates that the driver's condition has changed.

[0039] More specifically, in the case of an automatic lane change confirmation, if in step 240 it is determined that the first ratio of the first predetermined area A1 has increased by a specific threshold, for example, more than 100%, relative to the first specific ratio in the template, then in step 250 it is determined that the driver has performed an environmental check related to a left lane change, because the first predetermined area A1 is related to a left lane change. If in step 240 it is determined that the third ratio of the third predetermined area A3 has increased by a specific threshold, for example, more than 100%, relative to the third specific ratio in the template, then in step 240 it is determined that the driver has performed an environmental check related to a right lane change, because the third predetermined area A3 is related to a right lane change. The above explanation refers to the same specific threshold, but it is understood that different thresholds may be used for different predetermined areas.

[0040] The above explanation assumes that sufficient eye-tracking data is available for the current driver to determine accurate template data. In reality, it is often difficult to obtain sufficient data to determine the driver's template, for example, for newly purchased vehicles or vehicles being driven for the first time by a particular driver.

[0041] In such cases, one approach is to disable the method shown in Figure 5 for determining the driver's status until sufficient data is obtained to determine the driver's template, for example, by turning off the function or by requesting that the driver's status be checked by other means.

[0042] Another approach is to determine the driver's lane change status by determining whether the driver's state is appropriate for lane keeping. This is because, even when there is insufficient data to determine the output values ​​for predetermined areas corresponding to left and right lane changes in the template during vehicle operation, there is often sufficient data for lane management, i.e., a second specific total output value corresponding to a second predetermined area for lane management in the template is more reliable. According to this approach, in step 220, only the second total output value for the second predetermined area A2 is determined, then step 230 is omitted, and in step 240, the second total output value for the second predetermined area A2 is directly compared with a second specific total output value in the template, and if it is determined that the second total output value has decreased by a certain threshold relative to the second specific total output value, it indicates that the driver may not have paid enough attention to the second predetermined area, and in step 250, it is determined that the driver performed an environmental check related to lane changes, otherwise it indicates that the driver is still in a state appropriate for lane management.

[0043] The method shown in Figure 5 is illustrated with reference to a template that includes specific ratios for each given region, but this is merely illustrative. It is also conceivable that the template includes specific total output values ​​for various given regions, which are compared to total output values ​​calculated for the corresponding given regions during a driver state determination process to determine the driver's condition. However, it should be noted that in this case, the specific period for acquiring the template data and the period for determining the driver's condition should be equal. In contrast, when using ratios, the specific period for acquiring the template data and the period for determining the driver's condition do not need to be equal, reducing the requirements for the template data and improving its usefulness. Therefore, a template including ratios for each given region is preferable.

[0044] In the above explanation, we have referred to a general state template, but it is also conceivable that multiple templates corresponding to different driver states may be acquired or stored. For example, the automatic lane change confirmation function in autonomous driving can be configured to include three templates, namely a first template, a second template, and a third template, which are determined based on data when the driver is in a different state. The first template corresponds to the driver's state when performing a left lane change environment check, the second template corresponds to the driver's state when performing a lane management environment check, and the third template corresponds to the driver's state when performing a right lane change environment check. The data included in these templates may be at least one of the following: a specific output value for each predetermined sub-region, a specific total output value for each predetermined region among multiple predetermined regions, and / or the ratio of the specific total output value for each predetermined region to the sum of the specific output values ​​for all predetermined sub-regions.

[0045] After obtaining the above templates for different states, the driver state can be determined by deciding which template to use so that the output values ​​of each predetermined sub-region calculated in step 120 of Figure 3 for driver state confirmation match. For example, if it is determined that the output values ​​of each predetermined sub-region match the first template corresponding to the left lane change environment check, the driver state is determined to be the left lane change environment check.

[0046] The methods of this disclosure have been described primarily with reference to Figures 3 and 5, but are not limited thereto. For example, the methods according to various embodiments of this disclosure may include a step for determining and / or updating templates in real time. In one example, eye-tracking data can be acquired for a specific period of time of a driver, and the template is determined and / or updated based on the eye-tracking data for that specific period. In another embodiment, eye-tracking data can be acquired for specific periods of time of a driver in different states, and the corresponding template among multiple templates is determined and / or updated based on the eye-tracking data for specific periods of time of each state.

[0047] Figure 6 shows a block diagram of a driver monitoring device 10 according to one embodiment of the present disclosure. As shown in Figure 6, the device 10 comprises an acquisition unit 11, a calculation unit 12, and a determination unit 13. Optionally, the device 10 may also be configured to include a storage unit for storing one or more templates.

[0048] The acquisition unit 11 acquires driver eye-tracking data. This eye-tracking data may consist of a series of frames of images captured for the driver, or it may include first data representing the position of the driver's gaze derived from the multiple frames of the captured images, and second data representing the duration of the driver's gaze at the corresponding position.

[0049] The calculation unit 12 calculates an output value representing the driver's cumulative gaze over each predetermined sub-region within each predetermined region over a certain period, based on the acquired eye-tracking data. These multiple predetermined regions represent non-overlapping regions of interest during vehicle operation, and each predetermined region is divided into multiple non-overlapping predetermined sub-regions according to the division method shown in Figure 2. The determination unit 13 determines the driver's state based on the output values ​​of various predetermined sub-regions within each predetermined region.

[0050] In one example, the eye-tracking data includes first data representing the driver's gaze position and second data representing the duration of the driver's gaze at the corresponding position. In this case, the calculation unit 12 can calculate output values ​​for each predetermined sub-region based on the first and second data.

[0051] In other embodiments, eye-tracking data includes multiple frames of images collected from a driver over a period of time, and a computing unit determines first data representing the driver's gaze position for each frame of the images, and based on this first data, can determine a predetermined sub-region of a plurality of predetermined regions that the driver gazed upon. Furthermore, based on the predetermined sub-region of gaze determined for each frame of the plurality of images, the computing unit determines output values ​​for various predetermined sub-regions of the plurality of images.

[0052] In one embodiment, the acquisition unit 11 is further configured to acquire a template related to the driver, such as a template (average template) corresponding to the driver's general state, which includes, for example, a specific output value for each predetermined sub-region, a specific total output value for each predetermined region among a plurality of predetermined regions, and / or a ratio of the specific total output value for each predetermined region to the sum of the specific output values ​​for all predetermined sub-regions. The determination unit 13 can determine the driver's state based on this template and the output values ​​for each predetermined sub-region.

[0053] In other embodiments, the acquisition unit 11 is further configured to acquire a plurality of templates corresponding to different driver states, each of which includes a specific output value for each predetermined sub-region, a specific total output value for each predetermined region, and / or a ratio between the specific total output value for each predetermined region and the sum of the specific output values ​​for all predetermined sub-regions. The determination unit 13 can determine which template the output value for each predetermined sub-region corresponds to and determine the driver's state based on the determination result.

[0054] Although not described in detail, it will be understood that the acquisition unit 11 performs the function of acquiring various data and / or templates, the calculation unit 12 performs the function of calculating cumulative gaze output values ​​for each predetermined sub-region over a period of time, such as in step 120 of the method 100 described above, and the determination unit 13 performs the function of determining the driver's state based on the calculated output values ​​for each predetermined sub-region, such as in step 130 of the method 100 described above.

[0055] The above description of various embodiments of the methods and apparatus of the present disclosure with reference to Figures 1 to 6 can be combined with each other to achieve different effects, without being limited by the type of subject matter. Furthermore, the aforementioned units / steps / processes are not limiting, and the functions of the aforementioned units / steps / processes can be merged / combined / modified / altered to achieve the corresponding effects.

[0056] The methods and apparatus of the present disclosure may be configured to be performed by the electronic control unit of a vehicle. The functions of various units of the methods and systems of the present disclosure may be implemented by software or corresponding hardware, or the functions of the aforementioned units may be implemented using a processor, for example, the processor may read computer programs stored in memory and execute these computer programs to implement the functions of the aforementioned units. In one embodiment, the apparatus of the present disclosure may also be implemented by memory and a processor.

[0057] It will be understood that the methods according to the various embodiments of this disclosure can be achieved by computer programs / software. These software programs, once loaded into the working memory of a processor and executed, can carry out the methods according to the various embodiments of this disclosure.

[0058] The exemplary embodiments of this disclosure encompass both creating and using the computer programs / software of this disclosure from scratch, and updating existing programs / software to use the computer programs / software of this disclosure.

[0059] In other embodiments of the present disclosure, a computer program product is provided, such as a machine-readable medium (e.g., a computer) like a CD-ROM, which, when executed, contains computer program code causing a computer or processor to carry out the methods of various embodiments of the present disclosure. The machine-readable medium may be, for example, an optical storage medium or a solid-state medium supplied together with or as part of other hardware.

[0060] Specific embodiments of this disclosure have been described above. Other examples are also within the scope of the appended claims. In some cases, the operations or steps described in the claims may be performed in a different order than those in the embodiments, and the desired results may still be achieved. Furthermore, the processes illustrated in the drawings do not necessarily require a specific or sequential sequence to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or advantageous.

[0061] This disclosure is described with reference to specific embodiments, and those skilled in the art should understand that various implementations of the technical solutions of this disclosure can be made without departing from the spirit and basic features of this disclosure. Specific examples are illustrative and not limiting. Furthermore, these embodiments can be combined in any way to achieve the objectives of this disclosure. The scope of protection of this disclosure is defined by the appended claims.

[0062] In this specification and in the claims, the term “includes” does not exclude the existence of other elements or steps, and expressions such as “first,” “second,” “third,” and so on, and the order of steps referred to and shown in the drawings, do not limit their arrangement or number. The functions of the various elements described herein or enumerated in the claims may also be separated or combined and implemented by the corresponding multiple elements or a single element.

Claims

1. A method for monitoring drivers, To acquire driver eye-tracking data, Based on the aforementioned eye-tracking data, the method involves calculating an output value representing the driver's cumulative gaze over various predetermined sub-regions within a certain period of time for each of a plurality of predetermined regions, wherein the plurality of predetermined regions represent non-overlapping regions of interest during vehicle operation, and each predetermined region is divided into a plurality of non-overlapping predetermined sub-regions. The driver's state is determined based on the output values ​​of various predetermined sub-regions within each predetermined region. A method that includes this.

2. The eye-tracking data includes first data representing the location of the driver's gaze and second data representing the duration of the driver's gaze at the corresponding location, wherein the first and second data are determined based on a plurality of frames of images collected from the driver. The aforementioned method, Based on the first data and the second data, the output values ​​of various predetermined sub-regions are calculated. The method according to claim 1, further comprising:

3. The eye-tracking data includes multiple frames of images collected from the driver during the period, The aforementioned method, For each frame of the image, first data representing the position of the driver's gaze is determined, and based on the first data, the predetermined sub-region that the driver is looking at is determined from among the plurality of predetermined regions. The output values ​​for various predetermined sub-regions of the multiple frames of the image are determined based on the predetermined sub-regions that are looked at, determined for each frame of the multiple frames of the image. The method according to claim 1, further comprising:

4. The output values ​​of various predetermined subregions of multiple frames of the aforementioned image are: For each frame of the image, the following actions are performed, namely: Assigning a first specific value to the determined gazed-on predetermined sub-region, weighting the first specific value by a specific coefficient, thereby obtaining a first value for each gazed-on predetermined sub-region, wherein the specific coefficient is determined based on the first data and represents the degree or probability that the driver's gaze enters the corresponding predetermined sub-region, Assigning a second specific value to other predetermined subregions of the plurality of predetermined regions, excluding the predetermined subregion that was observed, thereby obtaining the first value for the other predetermined subregions. The output value is obtained for various predetermined subregions of the plurality of predetermined subregions by summing the first values ​​determined for different image frames for various predetermined subregions of the plurality of predetermined subregions, The method according to claim 3, which is determined by performing the operation to obtain the first value for various predetermined sub-regions of each predetermined region among the plurality of predetermined regions.

5. Obtaining a template related to the aforementioned driver, wherein the template includes at least one of the following: a specific output value for each predetermined sub-region, a specific total output value for each predetermined region among the plurality of predetermined regions, and / or the ratio of the specific total output value for each predetermined region to the sum of the specific output values ​​for all predetermined sub-regions. The state of the driver is determined based on the template and the output values ​​of each predetermined sub-region. The method according to claim 1, including the method described in claim 1.

6. The plurality of predetermined regions include a first predetermined region, a second predetermined region, and a third predetermined region, wherein the first predetermined region includes a region related to the vehicle changing to the left lane, the second predetermined region includes a region related to the vehicle maintaining its lane, and the third predetermined region includes a region related to the vehicle changing to the right lane; the period is a predetermined period after receiving a lane change confirmation request from the vehicle; and the template includes a first specific ratio for the first predetermined region, a second specific ratio for the second predetermined region, and a third specific ratio for the third predetermined region. The aforementioned method, The output values ​​for various predetermined sub-regions of the first predetermined region are summed up to obtain a first total output value for the first predetermined region, The output values ​​for various predetermined sub-regions of the second predetermined region are summed to obtain a second total output value for the second predetermined region, The output values ​​for various predetermined sub-regions of the third predetermined region are summed up to obtain a third total output value for the third predetermined region, Determine the first ratio, the second ratio, and the third ratio of the first total output value, the second total output value, and the third total output value in the sum of the output values ​​of all predetermined sub-regions, respectively. The first ratio, the second ratio, and the third ratio are compared with the first specific ratio, the second specific ratio, and the third specific ratio, respectively. If it is determined that the first ratio increases beyond a certain threshold relative to the first specific ratio, it is determined that the driver has performed an environmental check related to changing to the left lane, If it is determined that the third ratio increases beyond a certain threshold relative to the third specific ratio, it is determined that the driver has performed an environmental check related to changing lanes to the right, The method according to claim 5, further comprising:

7. The plurality of predetermined regions include a first predetermined region, a second predetermined region, and a third predetermined region, wherein the first predetermined region includes a region related to the vehicle changing to the left lane, the second predetermined region includes a region related to the vehicle maintaining its lane, and the third predetermined region includes a region related to the vehicle changing to the right lane, the period is a predetermined period after receiving a lane change confirmation request from the vehicle, and the template includes a second specific total output value for the second predetermined region. The aforementioned method, The output values ​​for various predetermined sub-regions of the second predetermined region are summed to obtain a second total output value for the second predetermined region, Comparing the second total output value with the second specific total output value, If it is determined that the second total output value decreases by a certain threshold relative to the second specific total output value, it is determined that the driver has performed an environmental check related to lane changes, The method according to claim 5, further comprising:

8. The method involves obtaining a plurality of templates corresponding to different states of the driver, wherein each of the plurality of templates includes at least one of a specific output value for each predetermined sub-region, a specific total output value for each predetermined region among the plurality of predetermined regions, and / or the ratio of the specific total output value for each predetermined region to the sum of the specific output values ​​for all predetermined sub-regions. Determining which template matches the output value for each predetermined sub-region among the aforementioned multiple templates, Based on the results of the above determination, the driver's condition is determined, The method according to claim 5, further comprising:

9. Determine and / or update the aforementioned template in real time. The method according to claim 5 or 8, further comprising:

10. A device for monitoring the driver, An acquisition unit configured to acquire driver eye-tracking data, A calculation unit configured to calculate an output value representing the cumulative gaze of the driver over various predetermined sub-regions within a certain period for each of a plurality of predetermined regions, based on the eye-tracking data, wherein the plurality of predetermined regions represent non-overlapping regions of interest during vehicle operation, and each predetermined region is divided into a plurality of non-overlapping predetermined sub-regions. A determination unit configured to determine the driver's state based on the output values ​​of various predetermined sub-regions within each predetermined region, A device equipped with the following features.

11. The eye-tracking data includes first data representing the location of the driver's gaze and second data representing the duration of the driver's gaze at the corresponding location, wherein the first and second data are determined based on a plurality of frames of images collected from the driver. The aforementioned computing unit is The apparatus according to claim 10, which calculates the output values ​​of various predetermined sub-regions based on the first data and the second data.

12. The eye-tracking data includes multiple frames of images collected from the driver during the period, The aforementioned computing unit is For each frame of the image, first data representing the position of the driver's gaze is determined, and based on the first data, the predetermined sub-region that the driver is looking at is determined from among the plurality of predetermined regions. Based on the predetermined sub-regions that are looked at, determined for each frame of the multiple frames of the image, the output values ​​for various predetermined sub-regions of the multiple frames of the image are determined. The apparatus according to claim 10, configured as described above.

13. The acquisition unit is further configured to acquire a template related to the driver, the template including at least one of a specific output value for each predetermined sub-region, a specific total output value for each predetermined region among the plurality of predetermined regions, and / or the ratio of the specific total output value for each predetermined region to the sum of the specific output values ​​for all predetermined sub-regions. The apparatus according to claim 10, wherein the determination unit determines the state of the driver based on the template and the output values ​​of each predetermined sub-region.

14. The acquisition unit is further configured to acquire multiple templates corresponding to different states of the driver, Each of the above-mentioned templates includes at least one of the following: a specific output value for each predetermined sub-region, a specific total output value for each predetermined region among the above-mentioned predetermined regions, and / or the ratio of the specific total output value for each predetermined region to the sum of the specific output values ​​for all predetermined sub-regions. The apparatus according to claim 13, wherein the determination unit is further configured to determine which template matches the output value for each predetermined sub-region among the plurality of templates, and to determine the state of the driver based on the result of the determination.

15. A computer-readable medium storing a computer program that, when executed by a processor, causes the processor to perform the method described in any one of claims 1 to 9.

Images