Driver monitoring device and monitoring program

Abstract

A driver monitoring device comprising: a camera (1) configured to capture images of a vehicle driver and the surroundings of the driver; a camera (2) having a narrow image capture range and configured to capture an image of a facial feature of the driver; a control unit configured to control image capture timing of the camera; and a determination unit configured to determine whether the driver is performing a driving-independent operation based on the acquired image. When a driver's negligence is detected on the basis of an image input from a camera (2) in a state in which the input from the camera (1) is stopped, a determination unit starts acquiring an image from the camera (1) and determines whether a detection target or a posture of the driver to be detected is included in the image captured by the camera (1).

Description

Technical Field

[0001] The present invention relates to a driver monitoring device and a monitoring program. Background Art

[0002] In recent years, there is a situation where, for example, due to the revision of the Road Traffic Law, new obligations are imposed on vehicle drivers and the number of new devices that must be installed on vehicles increases. For example, during the automatic driving of a vehicle, the obligation of safe driving is imposed on the driver, and therefore, it is very necessary for the system on the vehicle to correctly recognize the driver's driving condition to support the driver's safe driving, and to correctly record the actual situation of safe driving.

[0003] For example, Patent Document 1 discloses a driver monitoring device that can detect dangerous driving factors outside the image capture range of a camera. Specifically, Patent Document 1 describes detecting fatigue driving by determining the orientation of the face based on the brightness distribution on the driver's face, and determining negligent driving, fatigue driving, etc. based on the position of the pupil, the movement of the pupil, etc. In addition, Patent Document 1 describes a smartphone that can detect pupils reflected on the face or in glasses (lenses).

[0004] Reference List

[0005] Patent Literature

[0006] Patent Document 1: JP2021-43637A Summary of the invention

[0007] Technical issues

[0008] When the in-vehicle system monitors the driver's situation during driving, it needs basic functions such as detecting fatigue driving, inattentive driving, abnormal posture, etc. Therefore, it is necessary for the in-vehicle system to accurately grasp the details of the driver's eyes or face. For example, the in-vehicle system also needs the function of detecting dangerous driving conditions, such as when the driver is distracted while operating a smartphone.

[0009] However, in order to realize the above functions, it is necessary to continuously capture images in a wide range including the body of the driver sitting on the driver's seat from a position where a blind spot is unlikely to occur using a high-resolution camera, and continuously perform data processing on the captured images to monitor the detailed status regardless of the actual driving situation. Therefore, the load on the processing device that performs processing on the image data, etc. becomes large, and there is a concern of increased power consumption and increased heat generation.

[0010] As the load on the processing device becomes larger, the processing device is in a state of continuously consuming most of the resources of the processing device, and it is conceivable that the resources of the processing device cannot be used for purposes other than monitoring the driver. Therefore, it is difficult to perform design changes such as adding new functions to the existing processing device.

[0011] The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a driver monitoring device and a monitoring program that can appropriately monitor the driver in various situations and reduce the load on a processing device that processes image data and the like.

[0012] Solution to the problem

[0013] The object of the present invention is achieved by the following configuration.

[0014] A driver monitoring device comprising:

[0015] a first image capture unit configured to be able to capture an image of a vehicle driver and the driver's surroundings;

[0016] a second image capture unit having an image capture range narrower than that of the first image capture unit and configured to be capable of capturing an image of features appearing on the driver's face;

[0017] a control unit configured to control image capturing timing of the first image capturing unit and the second image capturing unit; and

[0018] a determination unit configured to determine whether the driver is performing at least an operation unrelated to the driving operation based on the images acquired from the first image capture unit and the second image capture unit, wherein

[0019] When driver inattention is detected based on the image input from the second image capture unit in a state where the input from the first image capture unit is stopped, the determination unit controls the control unit to start acquiring images from the first image capture unit, and the determination unit determines whether the detection target or the posture of the driver to be detected is included in the image acquired from the first image capture unit.

[0020] A monitoring program executable by a computer of a monitoring device, the monitoring device comprising: a first image capture unit configured to be able to capture images of a vehicle driver and the driver's surroundings; a second image capture unit having an image capture range narrower than the image capture range of the first image capture unit and configured to be able to capture images of features appearing on the driver's face; a control unit configured to control image capture timings of the first image capture unit and the second image capture unit; and a determination unit configured to determine whether the driver is performing at least an operation unrelated to a driving operation based on images acquired from the first image capture unit and the second image capture unit, the monitoring program comprising:

[0021] A process of detecting driver inattention based on a second image input from a second image capturing unit; and

[0022] When driver inattention is detected based on the second image in a state where input from the first image capture unit is stopped, a process of acquiring a first image from the first image capture unit and determining whether a detection target or a driver posture to be detected is included in the first image is started.

[0023] Advantageous Effects of the Invention

[0024] According to the driver monitoring device and monitoring program of the present invention, various conditions of the driver can be appropriately monitored and the load on the processing device that processes image data and the like can be reduced.

[0025] The present invention has been briefly described above. Further, the details of the present invention can be clarified by reading the modes for carrying out the present invention (hereinafter, referred to as "embodiments") described below with reference to the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a block diagram showing a schematic configuration of a driver monitoring device according to an embodiment of the present invention.

[0027] Figure 2 is a front view showing examples of two types of captured images captured by two cameras.

[0028] Figure 3 It is shown installed in Figure 1 Block diagram of the configuration of the main functions on the driver monitoring device.

[0029] Figure 4 is a flowchart showing an operation example of the main control unit.

[0030] Figure 5 It is shown Figure 1 A time diagram of Example 1 of the operation timing in the driver monitoring device.

[0031] Figure 6 It is shown Figure 1 2 is a time diagram of Example 2 of the operation timing in the driver monitoring device.

[0032] Figure 7 It is shown Figure 1 3 is a time diagram of Example 3 of the operation timing in the driver monitoring device. DETAILED DESCRIPTION

[0033] Specific embodiments of the present invention will be described below with reference to the accompanying drawings.

[0034] Figure 11 is a block diagram showing a schematic configuration of a driver monitoring apparatus 100 according to an embodiment of the present invention.

[0035] Figure 1 The illustrated driver monitoring apparatus 100 includes a first camera (camera 1 ) 10A, a second camera (camera 2 ) 10B, and a main control unit 20 .

[0036] The installation position and viewing angle of the first camera 10A have been adjusted in advance so that the first camera 10A can be Figure 1 That is, the first camera 10A can capture images within an image capture range A1 in a relatively wide range, including the face, upper body, arms, hands, etc. of the driver 50 sitting in the driver's seat.

[0037] For example, the first camera 10A is installed in a part of any one of an overhead module, a rearview mirror, a center portion of an instrument panel, a pillar, an instrument cover, an instrument interior, a steering column, etc. in a cabin of a vehicle.

[0038] The image capturing range A1 includes a region in which vehicle-mounted equipment such as a steering wheel 51 and a seat belt 52 of the vehicle exist.

[0039] The installation position and viewing angle of the second camera 10B have been adjusted in advance so that the second camera 10B can be Figure 1 That is, the second camera 10B can capture images only in a relatively narrower range than that of the first camera 10A, so that detailed information of a part of the face of the driver 50 sitting in the driver's seat can be captured.

[0040] The installation position of the second camera 10B may be the same as that of the first camera 10A, but it is preferable to set the second camera 10B at a position where it can capture an image from the front side of the driver 50 in order to more easily acquire information such as a line of sight.

[0041] The main control unit 20 is an electronic control unit (ECU) having a function of monitoring the driver's condition based on the images captured by the first camera 10A and the second camera 10B. The main control unit 20 can control the timing of capturing images, the timing of acquiring image data, exposure, gain, etc. for each of the first camera 10A and the second camera 10B. The main control unit 20 processes the image data acquired from at least one of the first camera 10A and the second camera 10B to acquire information indicating the driving condition, automatically records and stores the obtained information, and performs output of an alarm or the like according to the condition in order to support safe driving.

[0042] <Example of captured image>

[0043] Figure 2Specific examples of two types of captured images i1 and i2 captured by the first camera 10A and the second camera 10B, respectively, are shown.

[0044] exist Figure 2 In the captured image i1 shown, information such as the upper body, hands, and arms of the driver 50 sufficient to understand the driving situation is depicted. Figure 2 In the captured image i2 shown, information including the entire face and head of the driver 50 is depicted in an enlarged state.

[0045] Therefore, for example, by analyzing image data such as captured image i2 by the main control unit 20, the facial orientation and the line of sight direction of the driver 50 can be detected, and changes therein can be monitored. In addition, by analyzing image data such as captured image i1 by the main control unit 20, it is also possible to detect a situation in which the driver 50 is holding a steering wheel or to detect a mobile terminal (such as a smartphone) that the driver 50 may be holding in his or her hand.

[0046] <Main function configuration>

[0047] Figure 3 It is shown installed in Figure 1 1 is a block diagram of the configuration of main functions on the driver monitoring device 100.

[0048] Figure 3 Each function of the main control unit 20 shown is implemented by hardware operation of an electronic circuit, which mainly includes a microcomputer (not shown) in the main control unit 20 and a program executed by the microcomputer.

[0049] The main control unit 20 includes feature detection functions 21 and 22, a timing control function 23, a monitoring function unit 24, an alarm function 25, and a history information recording function 26. The monitoring function unit 24 includes an inattention detection function 24a, a fatigue detection function 24b, a posture collapse detection function 24c, a motion detection function 24d, a posture detection function 24e, a seat belt detection function 24f, and a steering wheel holding detection function 24g.

[0050] The feature detection function 21 detects various features of the upper body and hand regions of the driver 50 by performing image processing on the image data acquired from the first camera 10A. The feature detection function 22 detects various features of the facial region of the driver 50 by performing image processing on the image data acquired from the second camera 10B.

[0051] The timing control function 23 controls the timing at which each of the first camera 10A and the second camera 10B performs image capture (including control of illumination light), the timing at which each of the first camera 10A and the second camera 10B transmits the captured image data to the main control unit 20 (or the timing at which the main control unit 20 requests an image), etc. based on the detection status of each of the feature detection functions 21 and 22.

[0052] The monitoring function unit 24 detects the current state of each of various monitoring target items required for the vehicle.

[0053] The inattention detection function 24 a has a function of detecting whether the driver 50 is driving in a state facing a direction not related to driving based on the facial information of the driver 50 .

[0054] The fatigue detection function 24 b has a function of detecting whether the driver 50 is driving in a state where the eyes of the driver 50 are closed or in a fatigued state based on information on the face of the driver 50 .

[0055] The posture collapse detection function 24 c has a function of detecting whether the driver 50 is driving in an abnormal posture different from the normal driving posture based on information on the upper body of the driver 50 or the like.

[0056] The action detection function 24 d has a function of detecting whether the driver 50 is performing a dangerous distracted driving action (such as operating a smartphone) based on information about the upper body of the driver 50 or the like.

[0057] The posture detection function 24e has a function of detecting information such as the posture of the whole body of the driver 50, the orientation of the face in the horizontal direction and the front-back direction, the line of sight direction, the inclination of the upper body, and the like.

[0058] The seat belt detection function 24 f has a function of detecting whether the driver 50 is wearing a seat belt.

[0059] The steering wheel holding detection function 24g has a function of detecting whether the driver 50 holds the steering wheel under a condition in which the driver 50 can drive.

[0060] The alarm function 25 has a function of supporting the driver 50's safe driving by notifying the driver 50 of abnormalities, for example, when any function of the monitoring function unit 24 detects a situation in which the driver's driving condition should be warned, the driver 50 is notified of the abnormality by outputting voice or alarm sound.

[0061] The history information recording function 26 records and stores information related to the driving condition of the driver 50 and information about the current position of the vehicle detected by each function of the monitoring function unit 24 (eg, associated with the current time) as history information in a predetermined nonvolatile recording medium.

[0062] <Operation of the main control unit>

[0063] Figure 4 An example of the operation of the main control unit 20 is shown in FIG. That is, when the driver monitoring system (DMS) of the vehicle is activated, the computer of the main control unit 20 starts Figure 4 The following describes the operation Figure 4 The operation shown.

[0064] As an initialization process of each of the first camera 10A and the second camera 10B, the main control unit 20 automatically adjusts the exposure of each camera and the gain of the video processing circuit to make them appropriate, and records information indicating the state of each of the first camera 10A and the second camera 10B (S11). In the initial state, the lighting of each camera is turned off.

[0065] In the present embodiment, the main control unit 20 generally processes only the image captured by the second camera 10B to monitor the driver 50. Therefore, first, only the LED lighting built into or attached to the second camera 10B starts to turn on (S12). The brightness of the image captured by the second camera 10B is detected, and the amount of brightness change generated during a specific time is recorded. The emission wavelength of the LED lighting is in the near infrared region, so that the camera can capture images even at night.

[0066] The main control unit 20 periodically acquires image data obtained by capturing an image through the second camera 10B, processes the image data through the feature detection function 22, and periodically repeatedly performs a detection process of the face position of the driver 50 (S13).

[0067] The main control unit 20 compares the length of time during which the face position of the driver 50 can be detected with a predetermined threshold time (S14). If the length of time during which the face position can be detected is equal to or longer than the threshold time, the process proceeds to S16. If the condition in S14 is not met, the process proceeds to S17, and the main control unit 20 outputs information indicating that the face of the driver 50 cannot be detected.

[0068] The main control unit 20 compares the facial orientation and sight direction of the driver 50 with a predetermined specific range (S16). The specific range corresponds to the range of facial orientation and sight direction assumed when the driver 50 stares at the screen of a mobile terminal such as a smartphone, when the driver 50 operates the mobile terminal with his or her hands, or when the driver 50 performs an operation such as making a phone call on the mobile terminal. If the condition in S16 is met, the process proceeds to S24.

[0069] When the condition in S16 is not satisfied, the main control unit 20 identifies in S18 whether the vehicle is stopped. In addition, if the vehicle is stopped, the process proceeds to S19, and it is identified whether the facial orientation and the sight direction of the driver 50 remain stationary for a predetermined threshold time or longer. If the vehicle is not stopped, the process proceeds to S20, and it is identified whether the time when the facial orientation and the sight direction of the driver 50 are both outside the predetermined range continues for a threshold time or longer.

[0070] If the vehicle stops and the condition in S19 is satisfied, the process performed by the main control unit 20 proceeds to S21, and if the condition in S19 is not satisfied, proceeds to S13. If the vehicle does not stop and the condition in S20 is satisfied, the process proceeds to S21, and if the condition in S19 is not satisfied, the process proceeds to S13.

[0071] In S21, the main control unit 20 outputs an inadvertence alarm for the driver. In addition, the presence or absence of a predetermined alarm cancel signal is identified in S22. If the alarm cancel signal is detected, the alarm in S21 is canceled in the next S23, and then the process proceeds to S13. If the alarm cancel signal is not detected, the process proceeds to S18. For example, the alarm cancel signal for inadvertence may be generated by a predetermined button operation performed by the driver.

[0072] On the contrary, if the condition in S16 is satisfied, the main control unit 20 starts turning on the LED lighting built in or attached to the first camera 10A in S24 so that the first camera 10A can start capturing images. The main control unit 20 reflects the brightness change amount of the second camera 10B detected in S12 in the setting values ​​of exposure and gain in the first camera 10A.

[0073] The main control unit 20 periodically acquires image data obtained by capturing images through the first camera 10A, processes the image data through the feature detection function 21, and periodically and repeatedly performs detection processing on a mobile terminal (such as a smartphone) that the driver 50 may hold (S25).

[0074] For example, when the position of the driver's 50 hand is detected and tracked, and the same outline shape as that of a smartphone is detected in a rectangular area near the hand in the image to be processed, the smartphone can be considered to be detected. Alternatively, in the case where a hand appears on one side of the driver's 50 face and the same outline shape as that of a smartphone is detected in a rectangular area near the hand, such as when the driver 50 is making a phone call, the smartphone can be considered to be detected.

[0075] In the case where a smartphone is detected based on the image captured by the first camera 10A, the main control unit 20 generates a distracted driving warning related to smartphone operation in S27. In addition, it is identified in S29 whether the vehicle is stopped. If the vehicle is not stopped, the process returns to S25, and if the vehicle is stopped, the distracted driving warning is canceled (S30).

[0076] On the contrary, if the smartphone is not detected in S26, the process performed by the main control unit 20 proceeds to S28. It is identified whether the facial orientation and the direction of sight of the driver 50 are both maintained in a constant state. If the facial orientation and the direction of sight are constant, the process returns to S25, and if the facial orientation and the direction of sight change, the process proceeds to S18.

[0077] When the main control unit 20 is executing Figure 4 During the processing of S24 to S30, the image data captured by the second camera 10B is unnecessary, so the content of the processing may be changed so that the illumination of the second camera 10B is temporarily turned off and the image capturing operation or transmission of image data performed by the second camera 10B is stopped.

[0078] <Operation Timing of Equipment>

[0079] <Example 1>

[0080] Figure 5 Shows Figure 1 Example 1 of operation timing in the driver monitoring device 100 in FIG. Figure 5 The operation shown.

[0081] exist Figure 5 In the operation shown, the driver monitoring device 100 generally uses only the second camera 10B to monitor the driver at a constant period (i.e., at Figure 5 Images are repeatedly captured at times t1, t2, t3, t4, . . . ), and the images are processed each time to identify the driving condition of the driver 50.

[0082] When the state of the inattention state within the specified range is detected to continue for a certain time or longer (within the range from time t6 to time t9), as shown in FIG. Figure 5 In the case of the driver 50A shown, the main control unit 20 determines that there is a possibility of distracted driving and starts the process of detecting the smartphone. That is, image capture performed by the first camera 10A starts at time t10A, and image capture is repeated at times t11A, t12A, ... at regular time intervals.

[0083] Based on the images captured by the first camera 10A at times t10A, t11A, t12A, ..., as Figure 5In the case of the driver 50B shown in FIG. 1 , the main control unit 20 extracts a rectangular area near the hand and detects whether the smartphone is present in the rectangular area (corresponding to Figure 4 S25 and S26 in ).

[0084] exist Figure 5 In the illustrated operation, the times t10, t11, t12, ... at which the second camera 10B captures an image and the times t11A, t12A, ... at which the first camera 10A captures an image are not simultaneous and are offset from each other so that the timings do not overlap.

[0085] <Example 2>

[0086] Figure 6 Shows Figure 1 Example 2 of operation timing in the driver monitoring device 100 in FIG. Figure 6 The operation shown.

[0087] exist Figure 6 In the operation shown, the driver monitoring device 100 generally uses only the second camera 10B to monitor the driver at a constant period (i.e., at Figure 6 , . . . ) in the image capture sequence, but the timing (time t5, t10, . . . ) at which the second camera 10B does not capture an image is predetermined.

[0088] The main control unit 20 processes the image captured by the second camera 10B each time at times t1 , t2 , t3 , . . . to recognize the driving condition of the driver 50 .

[0089] like Figure 6 In the case of the driver 50A shown, when the state of the inattention state within the specified range is detected to continue for a certain time or longer (within the range from time t6 to time t9), the main control unit 20 determines that there is a possibility of distracted driving, and performs processing for detecting the smartphone. That is, the first camera 10A captures an image at time t10, and based on the image captured by the first camera 10A, the main control unit 20 extracts a rectangular area near the hand, such as Figure 6 and detects whether the smartphone is present in the rectangular area (corresponding to Figure 4 S25 and S26 in ).

[0090] <Example 3>

[0091] Figure 7 Shows Figure 1 Example 3 of the operation timing in the driver monitoring device 100 in FIG. Figure 7 The operation shown.

[0092] exist Figure 7 In the operation shown, the driver monitoring device 100 generally uses only the second camera 10B to monitor the driver at a constant period (i.e., at Figure 7 , . . . ) in the image capture sequence, but the timing (time t5, t10, . . . ) at which the second camera 10B does not capture an image is predetermined.

[0093] The main control unit 20 processes the image captured by the second camera 10B each time at times t1 , t2 , t3 , . . . to recognize the driving condition of the driver 50 .

[0094] like Figure 7 In the case of the driver 50A shown, when the state of the inattention state within the specified range is detected to continue for a certain time or longer (within the range from time t6 to time t9), the main control unit 20 determines that there is a possibility of distracted driving and starts the process for detecting the smartphone. That is, the image capture performed by the first camera 10A starts from time t10, and the image capture is repeated at times t10, t11, t12, ... predetermined at regular time intervals.

[0095] Image capturing performed by the second camera 10B stops at a timing (t11, t12, . . . ) after the first camera 10A starts capturing an image so that the timing at which the first camera 10A captures an image and the timing at which the second camera 10B captures an image do not overlap with each other.

[0096] Based on the images captured by the first camera 10A at times t10, t11, t12, ..., as Figure 7 In the case of the driver 50B shown in FIG. 1 , the main control unit 20 extracts a rectangular area near the hand and detects whether the smartphone is present in the rectangular area (corresponding to Figure 4 S25 and S26 in ).

[0097] As described above, in the driver monitoring device 100 according to the present embodiment, the main control unit 20 generally processes only the information received by the second camera 10B in the Figures 5 to 7 The images captured at times t1 to t9 in the image capture process are used as the processing targets, so the processing load in the main control unit 20 can be reduced. In particular, since the second camera 10B has a narrow image capture range, the orientation of the face and the direction of the line of sight can be sufficiently detected even in an image with a low resolution, which is effective in reducing the processing load. When the inattention of the driver 50 is detected, the image obtained by capturing a wide range using the first camera 10A is set as the processing target, so it is easy to detect a smartphone or the like held in the hand of the driver 50, and a distracted driving condition can be correctly detected. Figures 5 to 7As shown, by stopping image capture and processing performed by the second camera 10B at the timing of processing an image captured by the first camera 10A, the processing load of the main control unit 20 can be reduced.

[0098] The present invention is not limited to the above-described embodiments, and may be appropriately modified, improved, etc. In addition, the materials, shapes, sizes, numbers, arrangement positions, etc. of the components in the above-described embodiments are freely selectable and are not limited as long as the present invention can be achieved.

[0099] Here, the features of the driver monitoring device and the monitoring program according to the above-mentioned embodiments of the present invention are briefly summarized and listed in the following [1] to [5].

[0100] [1] A driver monitoring device (100), comprising:

[0101] A first image capturing unit (first camera 10A) configured to be capable of capturing images of a vehicle driver and the driver's surroundings;

[0102] a second image capturing unit (second camera 10B) having an image capturing range narrower than that of the first image capturing unit and configured to be capable of capturing an image of features appearing on the driver's face;

[0103] a control unit (main control unit 20, timing control function 23) configured to control the image capturing timing of the first image capturing unit and the second image capturing unit; and

[0104] A determination unit (main control unit 20, action detection function 24d) is configured to determine whether the driver is performing an operation unrelated to the driving operation based on the images acquired from the first image capture unit and the second image capture unit, wherein

[0105] When driver inattention is detected based on the image input from the second image capture unit in a state where the input from the first image capture unit is stopped, the determination unit controls the control unit to start acquiring images from the first image capture unit (S24, S25), and determines whether the detection target or the posture of the driver to be detected is included in the image acquired from the first image capture unit (S26).

[0106] According to the driver monitoring device having the above-mentioned [1] configuration, a condition such as distracted driving performed by a driver driving a vehicle can be easily detected. That is, by processing an image captured by a second image capture unit having a narrow image capture range, a negligent driving condition can be easily detected. By processing an image captured by a first image capture unit under a negligent driving condition, a smartphone operated by the driver can be detected or a posture of the driver staring at the smartphone screen can be detected. That is, distracted driving, such as operating a smartphone, can be identified under a negligent driving condition. In addition, by appropriately adjusting the image capture timing and processing timing of the first image capture unit and the second image capture unit, the processing load can be significantly reduced. Therefore, the heat generation and power consumption increase of the control device can be reduced, and the resource consumption of the control device can be reduced.

[0107] [2] The driver monitoring device according to [1], wherein

[0108] The determination unit detects a mobile terminal existing in an area near the driver based on the image captured by the first image capture unit (S24 to S26).

[0109] According to the driver monitoring device having the above-mentioned [2] configuration, the mobile terminal is detected based on the image captured by the first image capture unit, which captures the image within a relatively wide range, so distracted driving can be easily detected even if the driver holds and operates the mobile terminal at a distance from the driver's face.

[0110] [3] The driver monitoring device according to [1] or [2], wherein

[0111] The determination unit detects at least one of the facial orientation and the sight line direction of the driver based on the image captured by the second image capture unit ( S12 to S16 ).

[0112] According to the driver monitoring device having the above-mentioned [3] configuration, the driver's facial orientation or line of sight direction is detected, thereby detecting the driver's negligent driving condition, the driver's staring at the mobile terminal screen, etc.

[0113] [4] The driver monitoring device according to [1], wherein

[0114] The determination unit detects at least one of the driver's facial orientation and gaze direction based on the image captured by the second image capture unit (S12 to S16), and determines that the driver is inattentive when the detected facial orientation or gaze direction is within the range of the inattention state for a certain time or longer (S16, S24).

[0115] According to the driver monitoring device having the above-mentioned [4] configuration, the driver's facial orientation or line of sight direction is detected, and the length of time the same state continues is identified. Therefore, the driver's negligent driving condition, the condition in which the driver stares at the screen of the mobile terminal, etc., can be detected more accurately.

[0116] [5] A monitoring program executable by a computer (main control unit 20) of a monitoring device, the monitoring device comprising a first image capture unit, a second image capture unit, a control unit, and a determination unit, the first image capture unit being configured to be able to capture images of a vehicle driver and the driver's surroundings, the second image capture unit having an image capture range narrower than the image capture range of the first image capture unit and being configured to be able to capture images of features appearing on the driver's face, the control unit being configured to control image capture timings of the first image capture unit and the second image capture unit, the determination unit being configured to determine whether the driver is performing an operation unrelated to a driving operation based on images acquired from the first image capture unit and the second image capture unit, the monitoring program comprising:

[0117] a process of detecting driver inattention based on a second image input from a second image capturing unit (S12 to S16); and

[0118] When driver inattention is detected based on the second image while the input from the first image capture unit is stopped, the process of acquiring the first image from the first image capture unit and determining whether the detection target or the posture of the driver to be detected is included in the first image is started (S24 to S26).

[0119] By executing a monitoring program including the above-mentioned [5] program on a predetermined computer, conditions such as distracted driving performed by a driver driving a vehicle can be easily detected. That is, by processing an image captured by a second image capture unit having a narrow image capture range, a negligent driving condition can be easily detected. By processing an image captured by a first image capture unit under a negligent driving condition, a smartphone operated by the driver can be detected or a posture of the driver staring at the smartphone screen can be detected. That is, distracted driving, such as operating a smartphone, can be identified under a negligent driving condition. In addition, by appropriately adjusting the image capture timing and processing timing of the first image capture unit and the second image capture unit, the processing load can be significantly reduced. Therefore, the heat generation and power consumption increase of the control device can be reduced, and the resource consumption of the control device can be reduced.

[0120] This application is based on Japanese patent application (No. 2022-202280) filed on December 19, 2022, the contents of which are incorporated herein by reference.

[0121] Reference Symbols List

[0122] 10A: First Camera

[0123] 10B: Second Camera

[0124] 20: Main control unit

[0125] 21, 22: Feature detection function

[0126] 23: Timing control function

[0127] 24: Monitoring function unit

[0128] 24a: Negligence detection function

[0129] 24b: Fatigue detection function

[0130] 24c: Posture collapse detection function

[0131] 24d: Motion detection function

[0132] 24e: Posture detection function

[0133] 24f: Seat belt detection function

[0134] 24g: Steering wheel grip detection function

[0135] 25: Alarm function

[0136] 26: Historical information recording function

[0137] 50, 50A, 50B: Driver

[0138] 51: Steering Wheel

[0139] 52: Seat belt

[0140] 100: Driver Monitoring Device

[0141] A1, A2: Image capture range

[0142] i1, i2: captured images.

Claims

1. A driver monitoring device, comprising: a first image capture unit configured to be capable of capturing an image of a driver of a vehicle and the driver's surroundings; a second image capture unit having an image capture range narrower than that of the first image capture unit and configured to be capable of capturing an image of features appearing on the driver's face; a control unit configured to control image capturing timing of the first image capturing unit and the second image capturing unit; and a determination unit configured to determine whether the driver is performing an operation unrelated to a driving operation based on the images acquired from the first image capture unit and the second image capture unit, wherein When it is detected that the driver is inattentive based on the image input from the second image capture unit in a state where the input from the first image capture unit is stopped, the determination unit controls the control unit to start acquiring images from the first image capture unit, and the determination unit determines whether a detection target or a posture of the driver to be detected is included in the image acquired from the first image capture unit.

2. The driver monitoring device according to claim 1, wherein The determination unit detects a mobile terminal existing in an area near the driver based on the image captured by the first image capture unit.

3. The driver monitoring device according to claim 1, wherein The determination unit detects at least one of a facial orientation and a sight line direction of the driver based on the image captured by the second image capture unit.

4. The driver monitoring device according to claim 1, wherein The determination unit detects at least one of the driver's facial orientation and gaze direction based on the image captured by the second image capture unit, and determines that the driver is inattentive when the detected facial orientation or the gaze direction is within the range of an inattentive state for a certain period of time or longer.

5. A monitoring program executable by a computer of a monitoring device, the monitoring device comprising: a first image capture unit configured to be capable of capturing an image of a driver of a vehicle and the driver's surroundings; a second image capture unit having an image capture range narrower than that of the first image capture unit and configured to be capable of capturing an image of features appearing on the driver's face; a control unit configured to control image capturing timing of the first image capturing unit and the second image capturing unit; and a determination unit configured to determine whether the driver is performing an operation unrelated to a driving operation based on the images acquired from the first image capture unit and the second image capture unit, the monitoring program comprising: a process of detecting that the driver is inattentive based on a second image input from the second image capturing unit; and When the driver is detected to be inattentive based on the second image in a state where the input from the first image capture unit is stopped, a process is started to acquire a first image from the first image capture unit and determine whether a detection target or a posture of the driver to be detected is included in the first image.

Images