Distracted driving determination device and method

Abstract

To provide a technique which allows determination whether or not a driver is in his careless driving state, with considering a situation in which a drive wears something covering a part of his face.SOLUTION: A careless driving determination apparatus according to the present invention acquires a face image data of a face of a driver by using a driver monitoring camera. The apparatus calculates a distribution range of a direction of a face of the driver or a visual line angle thereof, based on a plurality of face image data acquired during a predetermined period. The apparatus determines, with respect to each of the plurality of face image data, whether or not the driver wears a mask or sun glasses. In response to the determined wearing state, the apparatus sets a threshold range. If the calculated distribution range falls within the threshold range, the apparatus determines that the driver is in its careless driving state.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to a technology for determining whether a driver is in a distracted driving state by using a driver monitor camera. [Background technology]

[0002] Patent Document 1 discloses a technology for determining whether a vehicle driver is driving absentmindedly. "Absentminded driving" means that the driver is driving the vehicle without concentrating on driving the vehicle. When a driver is driving absentmindedly, the driver may show a driving tendency to insufficiently check the surroundings. This technology acquires at least one of the driver's facial direction and line of sight. Then, if the amount of change in at least one of the acquired facial direction and line of sight of the driver is within a predetermined range for a predetermined time, it is determined that the driver is driving absentmindedly. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-86907 Summary of the Invention [Problem to be solved by the invention]

[0004] In the technology of Patent Document 1, the driver's facial orientation and gaze direction are calculated from facial image data of a driver monitor camera mounted on the vehicle. Therefore, if the driver is wearing an attachment that covers part of their face, the sensor values ​​obtained from the driver monitor camera may vary. Examples of such attachments include a mask that covers the mouth or IR-blocking glasses that block infrared rays around the eyes. If distracted driving is determined without taking into account the state of the attachment on the face, an erroneous determination may occur.

[0005] The present disclosure has been made in consideration of the above-mentioned problems, and aims to provide a technology that can accurately determine whether a driver is in a distracted driving state by taking into account the state in which the driver is wearing an attachment that covers part of the face. [Means for solving the problem]

[0006] To achieve the above object, the present disclosure is applied to a distracted driving determination device that determines whether a vehicle driver is in a distracted driving state. The distracted driving determination device includes a driver monitor camera that acquires facial image data of the driver, a memory that stores the facial image data acquired by the driver monitor camera, and a processor that calculates a distribution range of the driver's facial direction angle or gaze angle based on multiple pieces of facial image data acquired over a predetermined period of time, and determines that the driver is in a distracted driving state if the calculated distribution range falls within a threshold range. The processor is configured to determine, for each piece of facial image data, whether the driver is wearing an attachment that covers part of their face, as a first state in which the attachment is worn or a second state in which the attachment is not worn, and to set a first threshold range corresponding to the first state as the threshold range if a proportion of the facial image data determined to be in the first state is greater than a proportion of the facial image data determined to be in the second state, and to set a second threshold range corresponding to the second state as the threshold range if a proportion of the facial image data determined to be in the second state is greater than a proportion of the facial image data determined to be in the first state. The first threshold range is greater than the second threshold range.

[0007] In the present disclosure, the distribution range may be a distribution range of coordinate points obtained by projecting the driver's facial direction angle or gaze angle in multiple pieces of facial image data onto a coordinate system defined by the yaw angle direction and pitch angle direction of the vehicle. The threshold range may be a circular range of a predetermined radius centered at the center of the distribution range on the coordinate system, and the predetermined radius of the first threshold range may be larger than the predetermined radius of the second threshold range.

[0008] In the present disclosure, the processor may be configured to calculate the distribution range using facial image data corresponding to the first situation among the plurality of facial image data when the proportion of facial image data determined to be the first situation is greater than the proportion of facial image data determined to be the second situation among the wearing situations determined from each of the plurality of facial image data, and to calculate the distribution range using facial image data corresponding to the second situation when the proportion of facial image data determined to be the second situation is greater than the proportion of facial image data determined to be the first situation.

[0009] In the present disclosure, the wearable article may be a mask or infrared blocking glasses.

[0010] Furthermore, to achieve the above object, the present disclosure is applied to a method for determining whether a driver of a vehicle is in a state of distracted driving. The method for determining whether a driver of a vehicle is in a state of distracted driving includes: acquiring facial image data of the driver's face using a driver monitor camera; calculating a distribution range of the driver's facial direction angle or gaze angle based on a plurality of pieces of facial image data acquired over a predetermined period of time among the facial image data; determining, for each of the plurality of pieces of facial image data, whether the wearing situation indicating whether the driver is wearing an attachment covering part of the face is a first situation in which the attachment is worn or a second situation in which the attachment is not worn; and determining, if the proportion of the facial image data determined to be the first situation is greater than the proportion of the facial image data determined to be the second situation, whether the wearing situation corresponds to the first situation. and is greater than a second threshold range corresponding to a second situation. When the first threshold range is set as the threshold range and the proportion of face image data determined to be in the second situation is greater than the proportion of face image data determined to be in the first situation , th Two threshold ranges are set as threshold ranges, and when the calculated distribution range is included in the threshold ranges, it is determined that the driver is in a distracted driving state. [Effects of the Invention]

[0011] According to the present disclosure, even if there is variation in the wearing status of a driver's accessories over a certain period of time, the wearing status with the larger proportion is used to determine whether the driver is in a state of absentminded driving. This prevents erroneous determination of the wearing status, making it possible to accurately determine whether the driver is in a state of absentminded driving. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a block diagram showing the configuration of a distracted driving determination device 100. FIG. [Figure 2] This is a distribution diagram showing how the driver's face direction and line of sight moved within a certain period of time. [Figure 3] 1 is a distribution diagram showing how the driver's face direction and line of sight change over a certain period of time while driving absentmindedly. [Figure 4] 3 is a flowchart showing a routine of a distracted driving determination process executed by the distracted driving determination device 100 of the embodiment. [Figure 5] 5 is a flowchart showing the procedure of the attachment determination process executed in step S102 of FIG. 4. [Figure 6] 5 is a flowchart showing the flow of a threshold range setting process executed in step S106 of FIG. 4. [Figure 7] 5 is a flowchart showing a modified example of the process executed in step S104 of FIG. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. However, when the numbers, quantities, amounts, ranges, etc. of each element are mentioned in the embodiments described below, the technical idea of ​​the present disclosure is not limited to the mentioned numbers unless otherwise specified or clearly specified in principle. Furthermore, the structures, etc. described in the embodiments described below are not necessarily essential to the technical idea of ​​the present disclosure unless otherwise specified or clearly specified in principle.

[0014] 1.Configuration of the distracted driving detection device The absentminded driving determination device according to this embodiment is a device that determines whether a vehicle driver is in an absentminded driving state. A vehicle equipped with the absentminded driving determination device according to this embodiment is, for example, an autonomous vehicle. In an autonomous vehicle, the recognition, prediction, judgment, and operation that are conventionally performed by a driver in vehicle driving are performed by an autonomous driving system.

[0015] However, due to various reasons such as the driving environment or system abnormalities, the automated driving system may sometimes be unable to continue the automated driving. In such cases, the driver must immediately return to normal driving. For this reason, even during automated driving, the driver should avoid distracted driving. The distracted driving determination device according to this embodiment is a device that has the function of determining whether the driver is in a distracted driving state during automated driving of an automated vehicle.

[0016] The configuration of the absentminded driving determination device according to this embodiment will be described below with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the absentminded driving determination device 100. The absentminded driving determination device 100 is mounted on an autonomous driving vehicle. The absentminded driving determination device 100 is made up of a driver monitor camera 20 and a control device 50. The control device 50 and the driver monitor camera 20 are connected by an in-vehicle network such as a CAN (Controller Area Network).

[0017] The control device 50 is an ECU (Electronic Control Unit) that includes a processor 52 and a memory 54. Various programs and data are stored in the memory 54. The memory 54 here may include not only a memory in the narrow sense such as a RAM (Random Access Memory), but also a data storage device such as a magnetic disk such as an HDD, an optical disk such as a DVD, or a flash memory storage device such as an SSD. The memory 54 stores at least the driver status information DRS as data. The memory 54 also stores at least the distracted driving determination program CDP as a program. The program stored in the memory 54 is executed by the processor 52.

[0018] The driver monitor camera 20 is installed in a location in front of the driver's seat where it can capture an image of the driver's face, for example, above the steering column or near the interior rearview mirror. The driver monitor camera 20 constantly monitors the driver's facial expressions and stores the acquired data in the memory 54 of the control device 50. The data acquired by the driver monitor camera 20 is stored in the memory 54 as driver status information DRS. The driver status information DRS is used to determine whether the driver is driving absentmindedly by the absentminded driving determination program CDP, as will be described later. In other words, the driver monitor camera 20 functions as a absentminded driving determination device together with the control device 50.

[0019] 2. Distracted driving detection process The determination of careless driving by the careless driving determination device 100 is realized by the execution of the careless driving determination program CDP by the processor 52. Hereinafter, a method of determining careless driving by the careless driving determination program CDP will be described with reference to FIG.

[0020] Figure 2 shows distribution maps showing how the driver's facial direction and line of sight changed over a certain period of time. The top distribution map was created from data obtained under normal conditions, while the bottom distribution map was created from data obtained under distracted driving conditions. In each distribution map, the black circles indicate the facial direction angle, and the white circles indicate the line of sight angle. In addition, in each distribution map, the vertical axis P represents the angle in the vehicle pitch angle direction when the vehicle's forward direction is set to zero, and the horizontal axis Y represents the angle in the vehicle yaw angle direction when the vehicle's forward direction is set to zero. The coordinate system defined by the vertical axis P and the horizontal axis Y is hereinafter referred to as the "PY coordinate system."

[0021] As is clear from a comparison of the upper and lower distribution maps, in the distribution map for the absentminded driving state, the distribution range of facial direction angles is narrower than that for the normal state, and the distribution range of gaze angles is also narrower than that for the normal state. From this, it is thought that by examining the distribution ranges of facial direction angles and gaze angles, it is possible to distinguish whether the driver's state is normal or absentminded driving.

[0022] The distracted driving determination program CDP processes the driver status information DRS acquired by the driver monitor camera 20 and collects data on face direction angles and gaze angles over a certain period of time (for example, about 30 seconds).The distracted driving determination program CDP then calculates a distribution range RF in which face direction angles are distributed and a distribution range RG in which gaze angles are distributed.The distribution ranges RF and RG here are distribution ranges of coordinate points obtained by projecting the face direction angle data and gaze angle data onto the PY coordinate system.

[0023] A threshold range RFth is set for the distribution range RF of the facial direction angle. A threshold range RGth is also set for the distribution range RG of the gaze angle. The threshold ranges RFth and RGth are defined, for example, as circular ranges of a predetermined radius centered on the centers of the distribution ranges RF and RG on the PY coordinate. The radii of the threshold ranges RFth and RGth are determined based on data obtained in a distracted driving state. The distracted driving determination program CDP determines that the driver's state is a distracted driving state when the distribution range RF of the facial direction angle falls within the threshold range RFth or when the distribution range RG of the gaze angle falls within the threshold range RGth.

[0024] Here, if the driver is wearing an attachment on their face, variations may occur in the data of the driver's face direction angle or gaze angle processed from the driver status information DRS acquired by the driver monitor camera 20. Examples of such attachments include a mask that covers the mouth or infrared-blocking glasses (hereinafter also referred to as "sunglasses") that have a function of blocking infrared (IR). Figure 3 shows distribution maps showing how the driver's face direction and gaze shifted over a certain period of time while driving absentmindedly. The top distribution map was created from data with the driver's face bare, the middle distribution map was created from data with the driver wearing a mask, and the bottom distribution map was created from data with the driver wearing sunglasses. In each distribution map, black circles indicate the face direction angle, and white circles indicate the gaze angle.

[0025] As is clear from the comparison of the distribution diagrams shown in Figure 3, the distribution ranges RF and RG of face direction angle and gaze angle when wearing a mask have greater variation than when wearing no face. Similarly, the distribution range RF of face direction angle when wearing sunglasses has greater variation than when wearing no face. Note that when wearing sunglasses, the driver's gaze cannot be detected by the driver monitor camera 20, so data on the distribution range RG of gaze angle is not obtained.

[0026] As described above, when a driver is wearing a mask or sunglasses, the distribution range RF of the face direction angle or the distribution range RG of the gaze angle tends to vary. If the same threshold range as that for the bare face state is used to determine whether a driver is driving absentmindedly, there is a possibility that the driver may be determined to be driving normally even when the driver is driving absentmindedly.

[0027] Therefore, the distracted driving determination program CDP is configured to determine whether the driver's face is in a first state ("attached") or a second state ("not wearing an accessory") and to individually set threshold ranges depending on the determination result. Specifically, the distracted driving determination program CDP determines whether the driver is wearing a mask or sunglasses based on the driver status information DRS. This process is hereinafter referred to as the "attachment presence determination process." In the accessory presence determination process, the distracted driving determination program CDP acquires multiple pieces of face image data DT of the driver over a certain period (e.g., approximately 30 seconds). Then, the distracted driving determination program CDP calculates a proportion R1 of the acquired face image data DT1 that corresponds to the first state (i.e., wearing a mask or sunglasses) and a proportion R2 of the acquired face image data DT2 that corresponds to the second state (i.e., not wearing an accessory). If the proportion R1 of the face image data DT1 that corresponds to the first state (i.e., wearing an accessory) is greater than the proportion R2 of the face image data DT2 that corresponds to the second state (i.e., not wearing an accessory), the distracted driving determination program CDP determines that the driver is wearing an accessory. On the other hand, the distracted driving determination program CDP determines that no attachment is present when the ratio R1 of attachments present is smaller than the ratio R2 of attachments absent.

[0028] The distracted driving determination program CDP sets the threshold ranges according to the determination result of the accessory presence / absence determination process. Specifically, when it is determined that an accessory is present (first situation), the distracted driving determination program CDP sets the threshold ranges RFth and RGth to first threshold ranges RFth1 and RGth1, and when it is determined that an accessory is absent (second situation), it sets the threshold ranges RFth and RGth to second threshold ranges RFth2 and RGth2. The radius of the first threshold ranges RFth1 and RGth1 is larger than the radius of the second threshold ranges RFth2 and RGth2.

[0029] This process can accurately determine whether the driver is in a distracted driving state, regardless of whether the driver is wearing an accessory that covers part of their face. Furthermore, the accessory presence determination process determines whether the driver is wearing an accessory based on the ratio of the number of times the accessory is present and the ratio of the number of times the accessory is absent. Therefore, even if there is variation in the data on the state of the accessory over a certain period of time, it is possible to make a valid determination of whether the driver is wearing an accessory. This prevents erroneous determination of whether the driver is wearing an accessory.

[0030] 3. Specific processing for determining whether the vehicle is careless driving by the careless driving determination device 100 Next, a description will be given of the specific processing of determining whether or not the vehicle is careless driving by the careless driving determination device 100. Fig. 4 is a flowchart showing a routine of the careless driving determination processing executed by the careless driving determination device 100 of this embodiment. The routine shown in Fig. 4 is started when the processor 52 executes the careless driving determination program CDP.

[0031] In step S100, the facial image data of the driver for a certain period (for example, 30 seconds) is acquired. The facial image data acquired by the driver monitor camera 20 is constantly stored in the memory 54 as the driver status information DRS. Here, the facial image data of the driver for a certain period is acquired based on the driver status information DRS.

[0032] In step S102, the presence or absence of an attachment on the driver's face is determined using the facial image data acquired in step S100. Fig. 5 is a flowchart showing the flow of the attachment determination process executed in step S102 of Fig. 4. In step S122 shown in Fig. 5, the wearing status of an attachment is determined for each piece of facial image data acquired in step S100 using a known image recognition function.

[0033] In step S124, it is determined whether the proportion R1 of images determined to be facial image data DT1 in a first situation in which an attachment is being worn, among the facial image data DT for a certain period, is greater than the proportion R2 of images determined to be facial image data DT2 in a second situation in which an attachment is not being worn. If the determination is affirmative, the process proceeds to step S126, where it is determined that an attachment is being worn. On the other hand, if the determination is negative, the process proceeds to step S128, where it is determined that an attachment is not being worn.

[0034] After the process of step S102 is executed, the process proceeds to the next step S104. In step S104, a distribution range RF of face direction angles and a distribution range RG of gaze angles on the PY coordinates are calculated from the face image data acquired in step S100. After the process of step S104 is executed, the process proceeds to the next step S106.

[0035] In step S106, threshold ranges RFth, RGth for determining absentminded driving are set. Fig. 6 is a flowchart showing the flow of the threshold range setting process executed in step S106 of Fig. 4. In step S130 shown in Fig. 6, it is determined whether or not an attachment is present in the attachment presence determination in step S102. As a result, if it is determined that the determination is successful, the process proceeds to step S132.

[0036] In step S132, first threshold ranges RFth1, RGth1 are calculated. Here, circular ranges of first radii RFr1, RGr1 centered on the centers of the distribution ranges RF, RG calculated in step S104 are calculated as the first threshold ranges RFth1, RGth1. For the first radii RFr1, RGr1, values ​​set in advance based on past facial image data are used as thresholds for the distribution range at which a driver wearing an attachment is determined to be in a distracted driving state. Then, the calculated first threshold ranges RFth1, RGth1 are set as the threshold ranges RFth, RGth for determining distracted driving.

[0037] On the other hand, if the determination is not successful, the process proceeds to step S134. In step S134, second threshold ranges RFth2, RGth2 are calculated. Here, circular ranges of second radii RFr2, RGr2 centered on the centers of the distribution ranges RF, RG calculated in step S104 are calculated as the second threshold ranges RFth2, RGth2. The second radii RFr2, RGr2 are used as thresholds for the distribution range at which a driver without any attachments is determined to be in a distracted driving state, and values ​​set in advance based on past facial image data are used. The second radii RFr2, RGr2 are smaller than the first radii RFr1, RGr1. In other words, the second threshold ranges RFth2, RGth2 are smaller than the second threshold ranges RFth2, RGth2. The calculated second threshold ranges RFth2, RGth2 are then set as the threshold ranges RFth, RGth for determining distracted driving.

[0038] After the process of step S106 is executed, the process proceeds to the next step S108. In step S108, it is determined whether the distribution ranges RF, RG acquired in step S104 are included in the threshold ranges RFth, RGth set in step S106. As a result, if it is determined that the determination is established, the process proceeds to step S110, where it is determined that the driver is in a distracted driving state. On the other hand, if it is determined that the determination is not established, the process proceeds to step S112, where it is determined that the driver is not in a distracted driving state.

[0039] As is clear from the above description, the absentminded driving determination device 100 according to this embodiment can accurately determine whether or not the driver is wearing an object on their face. This makes it possible to perform highly accurate determination of absentminded driving. 4. Variations The absentminded driving determination device 100 according to this embodiment may be modified as follows.

[0040] 4-1. Distracted driving detection process The distribution range used in the absentminded driving determination process may be either the face direction angle distribution range RF or the line of sight angle distribution range RG. 4-2.Determining whether or not an item is attached In the attachment presence determination, the presence or absence of a mask and sunglasses may be determined separately. In this case, in step S132, separate threshold ranges RFth and RGth may be set for the case where a mask is worn and the case where sunglasses are worn.

[0041] 4-3. Threshold range RFth, RGth The threshold ranges RFth and RGth are not limited to circular ranges centered at the centers of the distribution ranges RF and RG, and may be ranges of other shapes set based on past face image data.

[0042] 4-4. Distribution range RF, RG The distribution ranges RF and RG used in the distracted driving determination process may be set depending on whether or not an accessory is present. That is, if the accessory determination process determines that an accessory is present, it is preferable not to use the facial image data DT2 determined to indicate that an accessory is absent for the distracted driving determination. Similarly, if the accessory determination process determines that an accessory is absent, it is preferable not to use the facial image data DT1 determined to indicate that an accessory is present for the distracted driving determination. Therefore, if the accessory determination process determines that an accessory is present, the distribution ranges RF and RG used in the distracted driving determination process are set based on the facial image data DT1 determined to indicate that an accessory is present. Similarly, if the accessory determination process determines that an accessory is absent, the distribution ranges RF and RG used in the distracted driving determination process are set based on the facial image data DT2 determined to indicate that an accessory is absent.

[0043] Such a modified example of the process can be realized, for example, by executing the process shown in Fig. 7 in step S104 of the distracted driving determination process shown in Fig. 4. Fig. 7 is a flowchart showing a modified example of the process executed in step S104 of Fig. 4. In step S140 shown in Fig. 7, it is determined whether or not an attachment is present in the attachment presence determination in step S102. If the determination is affirmative, the process proceeds to step S142, and if the determination is negative, the process proceeds to step S144.

[0044] In step S142, distribution ranges RF1, RG1 calculated using only the facial image data DT1 determined to have an attachment are set as distribution ranges RF, RG. On the other hand, in step S144, distribution ranges RF2, RG2 calculated using only the facial image data DT2 determined to have no attachment are set as distribution ranges RF, RG. In this way, distribution ranges RF, RG that reflect the presence or absence of attachments are set. [Explanation of symbols]

[0045] 20 Driver monitor camera 50 Control Unit (ECU) 52 processors 54 memory 100 Distracted driving detection device

Claims

1. A distracted driving determination device that determines whether a driver of a vehicle is in a distracted driving state, a driver monitor camera for acquiring face image data of the driver; a memory for storing the face image data acquired by the driver monitor camera; a processor that calculates a distribution range of the driver's face direction angle or line of sight angle based on a plurality of pieces of face image data acquired over a predetermined period of time, and determines that the driver is in a distracted driving state when the calculated distribution range falls within a threshold range; The processor: For each of the plurality of facial image data, a wearing state indicating whether the driver is wearing an attachment covering a part of the face is determined to be a first state in which the attachment is worn or a second state in which the driver is not wearing the attachment; When the proportion of the facial image data determined to be the first situation is greater than the proportion of the facial image data determined to be the second situation, a first threshold range corresponding to the first situation is set as the threshold range, and when the proportion of the facial image data determined to be the second situation is greater than the proportion of the facial image data determined to be the first situation, a second threshold range corresponding to the second situation is set as the threshold range. It is configured as follows: The first threshold range is greater than the second threshold range.

2. the distribution range is a distribution range of coordinate points obtained by projecting the face direction angle or the line of sight angle of the driver in the plurality of face image data onto a coordinate system defined by a yaw angle direction and a pitch angle direction of the vehicle, the threshold range is a circular range having a predetermined radius centered on the center of the distribution range on the coordinate system, The absentminded driving determination device according to claim 1 , wherein the predetermined radius of the first threshold range is greater than the predetermined radius of the second threshold range.

3. The processor: When the proportion of the facial image data determined to be the first situation is greater than the proportion of the facial image data determined to be the second situation among the wearing situations determined from each of the plurality of facial image data, the distribution range is calculated using the facial image data corresponding to the first situation among the plurality of facial image data, and when the proportion of the facial image data determined to be the second situation is greater than the proportion of the facial image data determined to be the first situation, the distribution range is calculated using the facial image data corresponding to the second situation.

3. The absentminded driving determination device according to claim 1 or 2, which is configured as follows:

4. The absentminded driving detection device according to claim 1 or 2, wherein the attachment is a mask or infrared blocking glasses.

5. A method for determining whether a driver of a vehicle is in a distracted driving state, comprising: Acquiring facial image data of the driver's face using a driver monitor camera; calculating a distribution range of the driver's face direction angle or line of sight angle based on a plurality of pieces of face image data acquired during a predetermined period of time among the face image data; For each of the plurality of facial image data, a wearing state indicating whether the driver is wearing an attachment covering a part of the face is determined to be a first state in which the attachment is worn or a second state in which the driver is not wearing the attachment; If the proportion of the facial image data determined to be the first situation is greater than the proportion of the facial image data determined to be the second situation, a first threshold range corresponding to the first situation and greater than a second threshold range corresponding to the second situation is set as the threshold range, and if the proportion of the facial image data determined to be the second situation is greater than the proportion of the facial image data determined to be the first situation, the second threshold range is set as the threshold range; The method for determining whether the driver is in a state of absentminded driving comprises determining that the driver is in a state of absentminded driving when the calculated distribution range is included in the threshold range.

Images