Driver monitoring device and monitoring program

Abstract

To reduce a load of processing a processing device for processing data such as images when monitoring a driving situation of a driver.SOLUTION: A driver monitoring device comprises: an image acquisition section for acquiring an image of a driver; and a determination section for determining from the acquired image whether or not the driver uses a detection target which is not associated with a driving operation. The determination section executes processing for extracting a first feature point included in the neck or the head of the driver on the image and a second feature point included in the arm or the hand of the driver, proceeding to S12 to S13 or S17 to S18 in a case where a difference between coordinates of the feature points is smaller than a reference value, and determining whether or not the target such as a smartphone is included on the image.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a driver monitoring device and a monitoring program. [Background technology]

[0002] In recent years, for example, due to amendments to the Road Traffic Act, new obligations are imposed on drivers who drive vehicles, and new devices that must be installed in vehicles are increasing. For example, since the driver is required to drive safely during automatic driving of a vehicle, it becomes increasingly necessary for the system on the vehicle to correctly grasp the driving situation of the driver to support the driver's safe driving, and to correctly record the actual situation of safe driving.

[0003] For example, Patent Document 1 discloses a driver monitoring device that can detect dangerous driving factors outside the imaging range of a camera. Specifically, it discloses detecting distracted driving by determining the direction of the driver's face from the luminance distribution of the driver's face, and detecting distracted driving, drowsy driving, etc. from the position and movement of the pupils. It also discloses detecting a smartphone or other object reflected in the pupils or eyeglasses of the face. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-43637 Summary of the Invention [Problem to be solved by the invention]

[0005] When an in-vehicle system monitors the driver's condition while driving, it is required to have a basic function of detecting drowsy driving, distracted driving, abnormal posture, etc. Therefore, the in-vehicle system needs to grasp detailed information of the driver's eyes and face. In addition, the in-vehicle system needs to have a function to detect dangerous driving situations, such as when the driver is distracted while driving while operating a smartphone.

[0006] However, for example, in a situation where a driver is talking on a smartphone by holding it to their ear, the smartphone will not be reflected in the driver's eyes or glasses, so the presence of the smartphone cannot be detected even using the technology in Patent Document 1.

[0007] In order to realize the above-mentioned functions, it is necessary to constantly capture images of a wide area including the body of the driver seated in the driver's seat with a high-resolution camera from a location where blind spots are unlikely to occur, regardless of the actual driving situation, and to constantly process the captured images to monitor the condition in detail. This causes a very large load on the processing device that processes the image data, etc., and there are concerns about increased power consumption and increased heat generation.

[0008] In addition, the load on the processing device will be so large that it will consume most of the processing device's resources at all times, making it impossible to use the processing device's resources for any purpose other than monitoring the driver. This makes it difficult to make design changes to add new functions to existing processing devices.

[0009] The present invention has been made in consideration of the above-mentioned circumstances, and its purpose is to provide a driver monitoring device and a monitoring program that are capable of reducing the processing load on a processing device that processes data such as images when monitoring the driver's driving condition. [Means for solving the problem]

[0010] The above object of the present invention can be achieved by the following configuration.

[0011] An image acquisition unit that acquires an image of a driver; From the acquired images, the driver Carry A determination unit that determines whether or not a mobile terminal is being used, the determination unit extracts a first feature point included in the neck or head of the driver on the image and a second feature point included in the arm or hand of the driver, and determines that the driver is operating the mobile terminal when a difference in a horizontal x coordinate between the coordinates of the first feature point and the coordinates of the second feature point is smaller than a first reference value and the mobile terminal is included in the image, and determines that the driver is talking on the phone when a difference in a vertical y coordinate between the coordinates of the first feature point and the coordinates of the second feature point is smaller than a second reference value and the mobile terminal is included in the image. Driver monitoring device.

[0012] An image acquisition unit that acquires an image of a driver; From the acquired images, the driver Carry A determination unit that determines whether or not a mobile terminal is being used, the determination unit detects at least three feature points including points of joint positions of the body of the driver on the input image, calculates angles of the joint positions from the coordinates of these feature points, and determines that the driver is on a call when the angle is equal to or smaller than a first angle and the image includes the mobile terminal, and determines that the driver is operating the mobile terminal when the angle is greater than the first angle and equal to or smaller than a second angle greater than the first angle and the image includes the mobile terminal. Driver monitoring device.

[0013] An image acquisition unit that acquires an image of a driver, and a driver recognition unit that recognizes the driver from the acquired image. Carry A monitoring program executable by a computer that controls a monitoring device having a determination unit that determines whether a mobile terminal is being used, A step of extracting a first feature point included in a neck or head of a driver and a second feature point included in an arm or a hand of the driver on a captured image; a step of executing a process of determining that the driver is operating the mobile terminal when a difference in a horizontal x coordinate between the coordinates of the first feature point and the coordinates of the second feature point is smaller than a first reference value and it is determined that the mobile terminal is included in the image, and determining that the driver is talking on the phone when a difference in a vertical y coordinate between the coordinates of the first feature point and the coordinates of the second feature point is smaller than a second reference value and it is determined that the mobile terminal is included in the image; Monitoring programs including. An image acquisition unit that acquires an image of a driver, and a driver recognition unit that recognizes the driver from the acquired image. Carry A monitoring program executable by a computer that controls a monitoring device having a determination unit that determines whether a mobile terminal is being used, A step of detecting at least three feature points including points of joint positions of the driver's body on the input image, and calculating angles of the joint positions from the coordinates of these feature points; a step of executing a process of determining that the driver is on a call when the angle is equal to or smaller than a first angle and when it is determined that the mobile terminal is included in the image, and determining that the driver is operating the mobile terminal when the angle is greater than the first angle and equal to or smaller than a second angle greater than the first angle and when it is determined that the mobile terminal is included in the image; Monitoring programs including. Effect of the Invention

[0014] According to the driver monitoring device and the monitoring program of the present invention, the processing load on a processing device that processes data such as images can be reduced when monitoring the driving condition of a driver.

[0015] The present invention has been briefly described above. Furthermore, the details of the present invention will be further clarified by reading the following description of the embodiment of the present invention (hereinafter, referred to as "embodiment") with reference to the accompanying drawings. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 is a block diagram showing a configuration of a driver monitoring device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram showing details of the main parts of the driver monitoring device shown in FIG. [Diagram 3] FIG. 3 is a front view showing two types of captured images. [Figure 4] FIG. 4 is a front view showing two types of captured images. [Diagram 5] FIG. 5 is a flowchart showing a first operational example of the main part of the driver monitoring device shown in FIG. [Figure 6] FIG. 6 is a front view showing two types of captured images. [Figure 7] FIG. 7 is a front view showing two types of captured images. [Figure 8] FIG. 8 is a flowchart showing a second operational example of the main part of the driver monitoring device shown in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Specific embodiments of the present invention will be described below with reference to the accompanying drawings.

[0018] FIG. 1 is a block diagram showing a configuration of a driver monitoring device 100 according to an embodiment of the present invention. The driver monitoring device 100 shown in FIG. 1 includes a first camera (camera 1) 10A, a second camera (camera 2) 10B, and a main control unit 20.

[0019] The installation position and angle of view of first camera 10A are adjusted in advance so that it can capture an image in shooting range A1 in Fig. 1. That is, first camera 10A can capture a relatively wide range of images, such as the face, upper body, arms, and hands, of driver 50 seated in the driver's seat.

[0020] For example, the first camera 10A is installed in any one of an overhead module, a room mirror, a center part of an instrument panel, a pillar, a meter hood, an inside of a meter, a steering column, or the like in the passenger compartment of the vehicle.

[0021] The photographing range A1 includes an area in which the vehicle's handle (steering wheel) 51, seat belt 52, and other in-vehicle accessories are present. The installation position and angle of view of second camera 10B are adjusted in advance so that it can capture an image in shooting range A2 in Fig. 1. That is, second camera 10B can capture only an image within a relatively narrow range compared to first camera 10A so as to capture detailed information of the facial parts of driver 50 seated in the driver's seat.

[0022] The installation position of second camera 10B may be the same as that of first camera 10A, but it is preferable to place second camera 10B in a position where it can capture an image from the front side of driver 50 so as to more easily obtain information such as the line of sight. The number of cameras used by the driver monitoring device 100 may be one, or may be increased to three or more as necessary.

[0023] The main control unit 20 is an electronic control unit (ECU) having a function of monitoring the driver's situation based on images captured by the first camera 10A and the second camera 10B. The main control unit 20 can control the timing of capturing images, the timing of acquiring image data, the amount of exposure, gain, and the like for each of the first camera 10A and the second camera 10B. The main control unit 20 processes the image data acquired from at least one of the first camera 10A and the second camera 10B to acquire information representing the driving situation, automatically records and saves the resulting information, and outputs an alarm or the like according to the situation to support safe driving.

[0024] <Major functional configuration> FIG. 2 is a block diagram showing details of the main parts of the driver monitoring device 100 shown in FIG.

[0025] Each function of the main control unit 20 shown in FIG. 2 is realized, for example, by the operation of hardware of an electronic circuit mainly composed of a microcomputer (not shown) incorporated in the main control unit 20 and a program executed by this microcomputer.

[0026] 2, the main control unit 20 includes feature detection functions 21 and 22, a timing control function 23, a monitoring function unit 24, an alarm function 25, a history information recording function 26, and an object detection control function 27. The monitoring function unit 24 also includes an inattentive driving detection function 24a, a drowsiness detection function 24b, a poor posture detection function 24c, an action detection function 24d, a posture detection function 24e, a seat belt detection function 24f, and a steering wheel holding detection function 24g.

[0027] Feature detection function 21 detects various features of each part of the upper body and the hand area of ​​driver 50 by image processing the image data acquired from first camera 10A. Feature detection function 22 detects various features of the face area of ​​driver 50 by image processing the image data acquired from second camera 10B.

[0028] Based on the detection states of feature detection functions 21 and 22, timing control function 23 can control the timing at which first camera 10A and second camera 10B each take pictures (including control of illumination light), the timing at which image data taken by first camera 10A and second camera 10B each is sent to main control unit 20 (or the timing at which main control unit 20 requests images), and the like.

[0029] The monitoring function unit 24 detects the current state of each of the various monitoring items required in the vehicle. The inattentiveness detection function 24a has a function of detecting whether the driver 50 is driving while facing a direction unrelated to driving, based on face information of the driver 50.

[0030] The drowsiness detection function 24b has a function of detecting whether the driver 50 is driving with his / her eyes closed or in a drowsy state, based on information on the face of the driver 50. The posture error detection function 24c has a function of detecting whether the driver 50 is driving in an abnormal posture different from a normal driving posture, based on information on the upper body of the driver 50 and the like.

[0031] The behavior detection function 24d has a function of detecting whether the driver 50 is engaging in a dangerous “distracted driving” behavior, such as operating a smartphone, based on information about the upper body of the driver 50, etc.

[0032] The posture detection function 24e has a function of detecting information such as the posture of the entire body of the driver 50, the horizontal and front-to-back orientation of the face, the direction of the line of sight, and the inclination of the upper body. The seat belt detection function 24f has a function of detecting whether or not the driver 50 is wearing a seat belt.

[0033] The steering wheel holding detection function 24g has a function of detecting whether or not the driver 50 is holding the steering wheel in a state capable of driving. The warning function 25 has a function of supporting the driver in safe driving by notifying the driver of an abnormality, for example by outputting a voice or an alarm sound, when any function of the monitoring function unit 24 detects a situation requiring a warning regarding the driving condition of the driver 50.

[0034] The history information recording function 26 records and stores information detected by each function of the monitoring function unit 24 regarding the driving state of the driver 50 in a specified non-volatile recording medium as history information, for example, in association with the current date and time and the current location of the vehicle.

[0035] The object detection control function 27 can generate a trigger to execute the image detection algorithm only when the behavior detection function 24d in the monitoring function unit 24 or the like meets a specific condition indicating a high possibility of distracted driving. The object detection control function 27 identifies whether or not a specific condition is met based on the coordinates of a plurality of feature points detected by the feature detection function 21 for the image of the driver 50, as described below.

[0036] <Example of a captured image-1> Two types of captured images i11 and i12 are shown in Figure 3. Two other types of captured images i21 and i22 are shown in Figure 4. In Figures 3 and 4, the horizontal direction is the x-axis and the vertical direction is the y-axis, and each position in the image is represented by two-dimensional coordinates of x and y.

[0037] Each of the captured images i11 and i12 shown in FIG. 3 and each of the captured images i21 and i22 shown in FIG. 4 contains sufficient information to grasp the driving situation of the driver 50, such as the upper body, both hands, and both arms.

[0038] 3, the captured image i11 shows a situation in which the driver 50 is driving safely with both hands holding the steering wheel 51. On the other hand, the captured image i12 shows a situation in which the driver 50 is holding the steering wheel 51 with the left hand, but has removed his right hand from the steering wheel 51 and is holding the smartphone held in his right hand close to the side of his face, that is, he is driving while talking on the phone.

[0039] 4, the driver 50 is holding the steering wheel 51 with both hands and driving safely. On the other hand, the driver 50 is holding the steering wheel 51 with his left hand, but his right hand is away from the steering wheel 51 and the smartphone he is holding in his right hand is placed away from his face, and he is either gazing at the smartphone screen or operating the screen while driving.

[0040] Moreover, in the captured images i11 and i12 in FIG. 3, the position of the characteristic point (A) of the wrist of the driver 50 and the position of the characteristic point (B) of the edge of the face (position of the ear) are shown, respectively. Here, it can be seen that in the captured image i11, the distance Δy in the y-axis direction between the two feature points (A, B) is relatively large, and in the captured image i12, the distance Δy in the y-axis direction between the two feature points (A, B) is relatively small.

[0041] Therefore, by examining the magnitude relationship of the distance Δy detected in the images, it is possible to easily distinguish between a safe driving state, as in the captured image i11, and a state in which there is a high possibility of driving while talking on the phone, as in the captured image i12.

[0042] On the other hand, in the captured images i21 and i22 in FIG. 4, the position of the characteristic point (A) of the wrist of the driver 50 and the position of the characteristic point (C) of the neck are shown, respectively. Here, it can be seen that in the captured image i21, the distance Δx in the x-axis direction between the two feature points (A, C) is relatively large, and in the captured image i22, the distance Δx in the x-axis direction between the two feature points (A, C) is relatively small.

[0043] Therefore, by examining the magnitude relationship of the distance Δx detected in the images, it is easy to distinguish between a safe driving state, as in the captured image i21, and a state in which there is a high possibility of driving while operating a smartphone, as in the captured image i22.

[0044] <Main control unit operation-1> An example of the operation of the main control unit 20 is shown in Fig. 5. That is, the computer of the main control unit 20 starts the operation of Fig. 5 when the driver monitoring system (DMS) of the vehicle activates a function to monitor "distracted driving" of the driver 50. The operation shown in Fig. 5 will be described below.

[0045] The main control unit 20 periodically and repeatedly performs a process for detecting the driving posture of the driver 50 at short time intervals (S11). That is, image data of each frame captured by the first camera 10A is periodically input to the main control unit 20. The feature detection function 21 detects the coordinate values ​​of each feature point (A, B, C) that represents the posture of the driver 50 for each frame.

[0046] The object detection control function 27 of the main control unit 20 obtains the coordinate values ​​of each feature point (key points: A, B) from the feature detection function 21, calculates the distance Δy in the y-axis direction between the feature points (A, B), and compares this distance Δy with a threshold value (S12).

[0047] For example, when the distance Δy is large as in the case of the photographed image i11 shown in Fig. 3, the condition of S12 is not satisfied and the process returns to S11 from S12. On the other hand, when the distance Δy is small as in the case of the photographed image i12, the condition of S12 is satisfied and the process proceeds from S12 to S13.

[0048] The main control unit 20 starts the execution of a predetermined object detection algorithm in S13 in response to a trigger output by the object detection control function 27. That is, in order to enable detection of characteristic points such as the joints of the driver 50 as well as objects such as a smartphone, detailed image processing is performed, and processing such as object pattern recognition is also performed.

[0049] In S14, the main control unit 20 identifies whether or not a smartphone has been detected in the image as a result of the object detection in S13. If the main control unit 20 detects a smartphone in S14, the process proceeds to S15, where it recognizes that the driver 50 is on a phone call, ie, is in a "distracted driving" state, and outputs a "distracted driving" warning.

[0050] Moreover, if the main control unit 20 does not detect a smartphone in S14, the process proceeds to S16, where it recognizes that even if the distance Δy is small, the driving state is not inappropriate. In addition, the object detection control function 27 of the main control unit 20 obtains the coordinate values ​​of each feature point (A, C) from the feature detection function 21, calculates the distance Δx in the x-axis direction between the feature points (A, C), and compares this distance Δx with a threshold value (S17).

[0051] For example, when the distance Δx is large as in the case of the photographed image i21 shown in Fig. 4, the condition of S17 is not satisfied, so the process returns to the process from S17 to S11. On the other hand, when the distance Δx is small as in the case of the photographed image i22, the condition of S17 is satisfied, so the process proceeds from S17 to S18.

[0052] The main control unit 20 starts the execution of a predetermined object detection algorithm in S18 in response to a trigger output by the object detection control function 27. That is, in order to enable detection of characteristic points such as the joints of the driver 50 as well as objects such as a smartphone, detailed image processing is performed, and processing such as object pattern recognition is also performed.

[0053] In S19, the main control unit 20 identifies whether or not a smartphone has been detected in the image as a result of the object detection in S18. If the main control unit 20 detects a smartphone in S19, the process proceeds to S20, where it recognizes that the driver 50 is in a "distracted driving" state, operating the smartphone, and outputs a "distracted driving" warning.

[0054] Also, if the main control unit 20 does not detect the smartphone in S19, the process proceeds to S21, where it recognizes that the driving state is not inappropriate even if the distance Δx is small. Therefore, in this case, no warning is output.

[0055] <Example of a photo taken-2> Two types of captured images i31 and i32 are shown in Fig. 6. Two other types of captured images i41 and i42 are shown in Fig. 7. In Fig. 6 and Fig. 7, the horizontal direction is the x-axis and the vertical direction is the y-axis, and each position in the image is represented by two-dimensional coordinates of x and y.

[0056] Each of the captured images i31 and i32 shown in FIG. 6 and each of the captured images i41 and i42 shown in FIG. 7 contains sufficient information to grasp the driving situation of the driver 50, such as the upper body, both hands, and both arms.

[0057] In the example of the captured image i31 shown in Fig. 6, the driver 50 is holding the steering wheel 51 with both hands, and therefore can be considered to be driving in a safe condition. On the other hand, in the example of the captured image i32, the driver 50 is holding the steering wheel 51 with the left hand, but has removed his right hand from the steering wheel 51 and has brought the smartphone held in his right hand close to the side of his face, that is, a problematic situation is shown in which the driver is driving while talking on the phone.

[0058] 7, the driver 50 is holding the steering wheel 51 with both hands and driving safely. On the other hand, the driver 50 is holding the steering wheel 51 with his left hand, but his right hand is away from the steering wheel 51 and the smartphone he is holding in his right hand is placed away from his face, and he is either gazing at the smartphone screen or operating the screen while driving.

[0059] 6, the position of the characteristic point of the right wrist of the driver 50, the position of the characteristic point corresponding to the joint of the right arm, and the position of the characteristic point of the right shoulder are shown. The angle of the joint part of the right arm that can be specified by the positions of these three characteristic points is shown as θ.

[0060] Here, it can be seen that in the photographed image i31, the angle θ of the joint part of the right arm is large, whereas in the photographed image i32, the angle θ of the joint part of the right arm is small. Therefore, by examining the magnitude relationship of the joint angle θ detected in the image, it is possible to easily distinguish between a safe driving state, as in the captured image i31, and a state in which there is a high possibility of driving while talking on the phone, as in the captured image i32.

[0061] On the other hand, the captured images i41 and i42 in Fig. 7 show the positions of the wrist feature point, the feature point corresponding to the right arm joint, and the feature point of the right shoulder of the driver 50. In addition, the angle of the joint part of the right arm that can be specified by the positions of these three feature points is shown as θ.

[0062] Here, it can be seen that in the photographed image i41, the angle θ of the joint part of the right arm is large, whereas in the photographed image i42, the angle θ of the joint part of the right arm is small. Therefore, by examining the magnitude relationship of the joint angle θ detected in the images, it is possible to distinguish between a safe driving state, as in the captured image i41, and a state in which there is a high possibility of driving while operating a smartphone, as in the captured image i42.

[0063] <Main control unit operation-2> An operation example-2 of the main control unit 20 is shown in Fig. 8. That is, the computer of the main control unit 20 starts the operation of Fig. 8 when the vehicle's driver monitoring system activates a function to monitor "distracted driving" of the driver 50. The content of the control shown in Fig. 8 is designed assuming a case where images expressing situations such as the captured images i31 and i32 shown in Fig. 6 and the captured images i41 and i42 shown in Fig. 7 are processed. The operation shown in Fig. 8 will be explained below.

[0064] The main control unit 20 periodically and repeatedly performs a process for detecting the driving posture of the driver 50 at short time intervals (S31). That is, image data of each frame captured by the first camera 10A is periodically input to the main control unit 20. The feature detection function 21 detects the coordinate values ​​of each feature point (the positions of the wrist, arm joint, and shoulder) that represents the posture of the driver 50 for each frame.

[0065] Furthermore, the object detection control function 27 of the main control unit 20 acquires the coordinate values ​​of each feature point from the feature detection function 21, and calculates the angle θ of the arm joint position in S32. Then, the main control unit 20 compares the angle θ of the joint position with predetermined threshold values ​​θ1, θ2 (S33, S34). For example, when the angle θ of the joint position is large as in the photographed image i31 shown in FIG. 6 or the photographed image i41 shown in FIG. 7, the conditions of S33 and S34 are not satisfied, so the process returns from S33 to S31.

[0066] On the other hand, when the angle θ of the joint position is equal to or smaller than the threshold θ1, as in the case of the photographed image i32, the condition of S33 is satisfied, and the process proceeds from S33 to S35. The main control unit 20 starts the execution of a predetermined object detection algorithm in S35 in response to a trigger output by the object detection control function 27. That is, in order to enable detection of characteristic points such as the joints of the driver 50 as well as objects such as a smartphone, detailed image processing is performed, and processing such as object pattern recognition is also performed.

[0067] In S36, the main control unit 20 identifies whether or not a smartphone has been detected in the image as a result of the object detection in S35. If the main control unit 20 detects a smartphone in S36, the process proceeds to S37, where it recognizes that the driver 50 is on a phone call, ie, is in a "distracted driving" state, and outputs a "distracted driving" warning.

[0068] Furthermore, if the main control unit 20 does not detect a smartphone in S36, the process proceeds to S41, where it recognizes that even if the joint position angle θ is small, the driving state is not inappropriate. On the other hand, when the angle θ of the joint position is larger than the threshold θ1 and smaller than the threshold θ2, as in the photographed image i42, the condition of S34 is satisfied, and the process proceeds from S34 to S38.

[0069] The main control unit 20 starts the execution of a predetermined object detection algorithm in S38 in response to a trigger output by the object detection control function 27. That is, in order to enable detection of characteristic points such as the joints of the driver 50 as well as objects such as a smartphone, detailed image processing is performed, and processing such as object pattern recognition is also performed.

[0070] In S39, the main control unit 20 identifies whether or not a smartphone has been detected in the image as a result of the object detection in S38. If the main control unit 20 detects a smartphone in S39, the process proceeds to S40, where it recognizes that the driver 50 is in a "distracted driving" state, operating the smartphone, and outputs a "distracted driving" warning.

[0071] Also, if the main control unit 20 does not detect a smartphone in S39, the process proceeds to S41, where it recognizes that even if the angle θ of the joint position is small, the driving state is not inappropriate. Therefore, in this case, no warning is output.

[0072] The present invention is not limited to the above-described embodiment, and can be appropriately modified, improved, etc. In addition, the material, shape, size, number, arrangement location, etc. of each component in the above-described embodiment are arbitrary as long as the present invention can be achieved, and are not limited.

[0073] For example, the "distracted driving" of the driver monitored by the driver monitoring device 100 may include driving while eating and drinking, or driving while smoking, in addition to operating a smartphone. Therefore, for example, the objects recognized in S13 and S18 of Fig. 5 and the objects recognized in S35 and S38 of Fig. 8 may be changed to include food, drink, and cigarettes in addition to a smartphone. Even when monitoring "distracted driving" caused by eating, drinking, or smoking, the influence of the difference in the type of target object on the distances Δx and Δy between feature points and the angle θ of the joints is not significantly different from that of a smartphone, so the contents of the control shown in Fig. 5 and the control shown in Fig. 8 can be used as is.

[0074] It is also possible to use a combination of the process shown in Fig. 5 and the process shown in Fig. 8. That is, the main control unit 20 may detect both the distances Δx, Δy and the joint angle θ, and when both or either one of the distances Δx or Δy and the joint angle θ satisfy a predetermined condition, control may be performed to proceed to any one of the processes of S13, S18, S35, or S38.

[0075] As described above, when the main control unit 20 of the driver monitoring device 100 executes the operation shown in FIG. 5, in a normal driving state in which the conditions of S12 or S17 are not satisfied, it is sufficient to detect the coordinates of multiple feature points (A, B, C), and there is no need to start complex image processing to detect an object such as a smartphone.

[0076] Furthermore, when the main control unit 20 executes the operation shown in FIG. 8, in a normal driving state that does not satisfy the conditions of S33 or S34, it is only necessary to detect a plurality of feature points and calculate the joint angle θ, and there is no need to start complex image processing to detect an object such as a smartphone. Therefore, in either case of implementing the control of FIG. 5 or FIG. 8, the normal processing load on the main control unit 20 can be reduced. This makes it possible to reduce the cost of the main control unit 20, while at the same time suppressing heat generation and extending the product life. Furthermore, the amount of power consumed by the main control unit 20 is also reduced.

[0077] Here, the features of the driver monitoring device and the monitoring program according to the above-mentioned embodiment of the present invention will be briefly summarized and listed in the following [1] to [5]. [1] An image acquisition unit (feature detection function 21) that acquires an image of a driver; A determination unit (action detection function 24d) that determines whether or not the driver is using a predetermined detection object unrelated to driving operation from the acquired image, The determination unit extracts a first feature point (B or C) included in the neck or head of the driver on the image and a second feature point (A) included in the arm or hand of the driver, and when a difference (Δy or Δx) between the coordinates of the first feature point and the coordinates of the second feature point is smaller than a reference value, executes a process (S12, S13, S17, S18) of determining whether or not the detection target is included in the image. Driver monitoring device.

[0078] According to the driver monitoring device having the configuration of [1] above, in a normal driving situation of the driver, it is only necessary to execute a simple process of monitoring the coordinates of two or three feature points, so that the processing load on the determination unit can be reduced. Also, only when it is detected that there is a possibility of distracted driving based on the coordinates of the feature points, a process for determining whether or not the detection object is included in the image is executed, thereby making it possible to detect the situation of distracted driving related to a smartphone or the like.

[0079] [2] The determination unit determines whether or not at least one of an x ​​coordinate and a y coordinate of a difference between the coordinates of the first feature point and the second feature point is smaller than a reference value (S12, S17). A driver monitoring device as described in [1] above.

[0080] According to the driver monitoring device having the configuration of [2] above, by comparing the difference (Δx) in the x coordinate between the first feature point and the second feature point with a reference value, it is possible to distinguish between two types of situations, for example, the captured images i21 and i22 shown in Fig. 4. Also, by comparing the difference (Δy) in the y coordinate between the first feature point and the second feature point with a reference value, it is possible to distinguish between two types of situations, for example, the captured images i11 and i12 shown in Fig. 3.

[0081] [3] An image acquisition unit (feature detection function 21) that acquires an image of a driver; A determination unit (action detection function 24d) that determines whether or not the driver is using a predetermined detection object unrelated to driving operation from the acquired image, The determination unit detects at least three feature points including points of joint positions of the driver's body on the input image, calculates an angle (θ) of the joint position from the coordinates of these feature points, and executes a process (S32, S33, S34, S35, S38) of determining whether or not a predetermined detection object is included in the image if the calculated angle satisfies a predetermined condition. Driver monitoring device.

[0082] According to the driver monitoring device having the configuration of [3] above, in a normal driving situation of the driver, it is sufficient to execute a simple process of detecting the coordinates of at least three feature points and calculating the joint angles, so that the processing load on the determination unit can be reduced. Also, only when it is detected that there is a possibility of distracted driving based on the detected joint angles, a process for determining whether or not the detection object is included in the image is executed, thereby making it possible to detect the situation of distracted driving related to a smartphone, etc.

[0083] [4] The determination unit detects a mobile terminal in the vicinity of the driver's hand as the detection object. A driver monitoring device as described in [1] above.

[0084] According to the driver monitoring device having the configuration of [4] above, since it detects a mobile device that the driver is likely to be handling in his / her hands while driving, it becomes easy to detect a situation in which normal driving is not possible, such as in the case of distracted driving, even if the driver's face or gaze is facing forward.

[0085] [5] A monitoring program executable by a computer that controls a monitoring device having an image acquisition unit that acquires an image of a driver and a determination unit that determines whether the driver is using a predetermined detection object unrelated to driving operation from the acquired image, A step (S11) of extracting a first feature point included in the neck or head of the driver and a second feature point included in the arm or hand of the driver on the captured image; a step of executing a process of determining whether or not the detection target object is included in an image when a difference between the coordinates of the first feature point and the coordinates of the second feature point is smaller than a reference value (S12, S13, S17, S18); Monitoring programs including.

[0086] According to the monitoring program having the configuration of [5] above, in a normal driving situation of a driver, it is sufficient to execute a simple process of monitoring the coordinates of two or three feature points, thereby reducing the processing load on the determination unit. Also, only when it is detected that there is a possibility of distracted driving based on the coordinates of the feature points, a process for determining whether or not the detection object is included in the image is executed, thereby making it possible to detect the situation of distracted driving related to a smartphone, etc. [Explanation of symbols]

[0087] 10A 1st Camera 10B Second Camera 20 Main control section 21,22 Feature detection function 23 Timing control function 24 Monitoring function section 24a Inattentive detection function 24b Drowsiness detection function 24c Posture correction detection function 24d Activity Detection Function 24e Attitude detection function 24f Seat belt detection function 24g Handle holding detection function 25 Alarm function 26 History information recording function 27 Object detection control function 50 Driver 51 Handle 52 Seatbelt 100 Driver monitoring device A1, A2 shooting range i11, i12, i21, i22, i31, i32, i41, i42 Images Δx,Δy distance θ Joint angle θ1,θ2 threshold

Claims

1. An image acquisition unit that acquires an image of a driver; A determination unit that determines whether or not the driver is using a mobile device from the acquired image, the determination unit extracts a first feature point included in the neck or head of the driver on the image and a second feature point included in the arm or hand of the driver, and if a difference in a horizontal x coordinate between the coordinate of the first feature point and the coordinate of the second feature point is smaller than a first reference value and it is determined that the mobile terminal is included on the image, determines that the driver is operating the mobile terminal, and if a difference in a vertical y coordinate between the coordinate of the first feature point and the coordinate of the second feature point is smaller than a second reference value and it is determined that the mobile terminal is included on the image, determines that the driver is talking on the phone. Driver monitoring device.

2. An image acquisition unit that acquires an image of a driver; A determination unit that determines whether or not the driver is using a mobile device from the acquired image, the determination unit detects at least three feature points including points of joint positions of the driver's body on the input image, calculates angles of the joint positions from the coordinates of these feature points, and determines that the driver is on a call when the angle is equal to or smaller than a first angle and the mobile terminal is included in the image, and determines that the driver is operating the mobile terminal when the angle is greater than the first angle and equal to or smaller than a second angle greater than the first angle and the mobile terminal is included in the image. Driver monitoring device.

3. A monitoring program executable by a computer that controls a monitoring device having an image acquisition unit that acquires an image of a driver and a determination unit that determines whether or not the driver is using a mobile device from the acquired image, A step of extracting a first feature point included in a neck or head of a driver and a second feature point included in an arm or a hand of the driver on a captured image; a step of executing a process of determining that the driver is operating the mobile terminal when a difference in a horizontal x coordinate between the coordinates of the first feature point and the coordinates of the second feature point is smaller than a first reference value and it is determined that the mobile terminal is included in the image, and determining that the driver is talking on the phone when a difference in a vertical y coordinate between the coordinates of the first feature point and the coordinates of the second feature point is smaller than a second reference value and it is determined that the mobile terminal is included in the image; Monitoring programs including.

4. A monitoring program executable by a computer that controls a monitoring device having an image acquisition unit that acquires an image of a driver and a determination unit that determines whether or not the driver is using a mobile device from the acquired image, A step of detecting at least three feature points including points of joint positions of the driver's body on the input image, and calculating angles of the joint positions from the coordinates of these feature points; a step of executing a process of determining that the driver is on a call when the angle is equal to or smaller than a first angle and when it is determined that the mobile terminal is included in the image, and determining that the driver is operating the mobile terminal when the angle is greater than the first angle and equal to or smaller than a second angle greater than the first angle and when it is determined that the mobile terminal is included in the image; Monitoring programs including.

Images