Driver monitoring device and monitoring program

Abstract

The driver monitoring device comprises: an image acquisition unit for acquiring an image of a driver; and a determination unit for determining whether the driver is using the detection target object irrelevant to the driving operation according to the acquired image. The determination unit extracts a first feature point included in the neck or head of the driver and a second feature point included in the arms or hands of the driver from the image, advances from S12 to S13 or from S17 to S18 if a difference between coordinates of the feature points is less than a reference value, and performs processing to determine whether or not a target object such as a smartphone is included in the image.

Description

Technical Field

[0001] The invention relates to a driver monitoring device and a monitoring program. Background Art

[0002] In recent years, there are cases where, for example, new obligations are imposed on drivers who drive vehicles due to modifications to road traffic behavior and the number of new devices that must be installed on vehicles has increased. For example, safe driving obligations are imposed on drivers during the automatic driving of vehicles, and therefore, it is highly necessary for a system on the vehicle to correctly recognize the driver's driving situation to support the driver's safe driving, and to correctly record the actual situation of safe driving.

[0003] For example, Patent Document 1 discloses a driver monitoring device capable of detecting dangerous driving elements outside the imaging range of a camera. Specifically, Patent Document 1 describes detecting careless driving by determining the orientation of the driver's face based on the brightness distribution of the face, and determining careless driving, fatigue driving, etc. based on the position of the pupil, the movement of the pupil, etc. In addition, Patent Document 1 describes a smartphone or the like that can detect reflections in the pupils or glasses (lenses) of the face.

[0004] Reference List

[0005] Patent Literature

[0006] Patent Document 1: JP2021-43637A Summary of the invention

[0007] Technical issues

[0008] When the in-vehicle system monitors the driver's condition during driving, basic functions such as detecting fatigue driving, careless driving, abnormal posture, etc. are required. Therefore, the in-vehicle system needs to accurately grasp information about the details of the driver's eyes or face. For example, the function of detecting dangerous driving situations (such as situations where the driver performs distracted driving while operating a smartphone) is also required in the in-vehicle system.

[0009] However, for example, in the case where the driver puts the smartphone to his / her ear and makes a call, the smartphone is not reflected in the driver's pupils or lenses, making it impossible to detect the presence of the smartphone even using the technology in Patent Document 1.

[0010] In order to realize the above functions, it is necessary to continuously capture a wide range of images including the body of the driver sitting on the driver's seat from a position where blind spots are unlikely to occur by using a camera with high resolution, and continuously perform data processing on the captured images to monitor the state of details regardless of the actual driving situation. Therefore, the load on the processing device that processes the image data, etc. becomes very large, and there are problems of increased power consumption and increased heat generation.

[0011] Since the load on the processing device becomes very large, most of the resources of the processing device are continuously consumed, and it is conceivable that the resources of the processing device cannot be used for purposes other than monitoring the driver. Therefore, it is difficult to change the design of adding new functions to the existing processing device.

[0012] The present invention is made in view of the above situation, and an object of the present invention is to provide a driver monitoring device and a monitoring program, which can reduce the processing load of a processing device that processes image data when monitoring the driver's driving condition.

[0013] Solutions to technical problems

[0014] The object of the present invention is achieved by the following configuration.

[0015] A driver monitoring device, comprising:

[0016] an image acquisition unit configured to acquire an image of the driver; and

[0017] a determination unit configured to determine whether the driver is using a predetermined detection target that is not related to the driving operation based on the acquired image, wherein

[0018] The determination unit extracts a first feature point included in the driver's neck or head and a second feature point included in the driver's arm or hand from the image, and when the difference between the coordinates of the first feature point and the coordinates of the second feature point is less than a reference level, performs processing to determine whether the detection target is included in the image.

[0019] A driver monitoring device, comprising:

[0020] an image acquisition unit configured to acquire an image of the driver;

[0021] a determination unit configured to determine whether the driver is using a predetermined detection target that is not related to the driving operation based on the acquired image, wherein

[0022] The determination unit detects at least three feature points including points of joint positions of the driver's body on an input image, calculates angles at the joint positions based on the coordinates of these feature points, and performs processing to determine whether a predetermined detection target is included in the image when the calculated angle satisfies a predetermined condition.

[0023] A monitoring program executable by a computer controlling a monitoring device, the monitoring device comprising: an image acquisition unit configured to acquire an image of a driver; and a determination unit configured to determine whether the driver is using a predetermined detection target unrelated to a driving operation based on the acquired image, the monitoring program comprising:

[0024] A process of extracting a first feature point included in a neck or head of a driver and a second feature point included in an arm or hand of the driver from the captured image; and

[0025] When the difference between the coordinates of the first feature point and the coordinates of the second feature point is smaller than a reference level, a process of determining whether the detection target is included in the image is performed.

[0026] Beneficial Effects of the Invention

[0027] According to the driver monitoring device and the monitoring program of the present invention, it is possible to reduce the processing load of a processing device that processes data such as images when monitoring the driving condition of a driver.

[0028] The present invention has been briefly described above. The details of the present invention can be clarified by describing the following mode for carrying out the present invention (hereinafter referred to as “embodiment”) with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a block diagram showing a configuration of a driver monitoring device according to an embodiment of the present invention.

[0030] Figure 2 It is shown Figure 1 A block diagram showing details of the main components of a driver monitoring device is shown.

[0031] Figure 3 is a front view showing two types of captured images.

[0032] Figure 4 is a front view showing two types of captured images.

[0033] Figure 5 It is shown Figure 1 A flowchart of operation example 1 of the main components of the driver monitoring device is shown.

[0034] Figure 6is a front view showing two types of captured images.

[0035] Figure 7 is a front view showing two types of captured images.

[0036] Figure 8 It is shown Figure 1 A flowchart of operation example 2 of the main components of the driver monitoring device is shown. DETAILED DESCRIPTION

[0037] Specific embodiments of the present invention will be described hereinafter with reference to the accompanying drawings.

[0038] Figure 1 is a block diagram showing the configuration of a driver monitoring device 100 according to an embodiment of the present invention.

[0039] Figure 1 The illustrated driver monitoring device 100 includes a first camera (camera 1 ) 10A, a second camera (camera 2 ) 10B, and a main control unit 20 .

[0040] The installation position and viewing angle of the first camera 10A are adjusted in advance so that the Figure 1 That is, the first camera (10A) can capture images within a wider range of the face, upper body, arms, hands, etc. of the driver (50) contained in the driver's seat.

[0041] For example, in a cabin of a vehicle, the first camera 10A is installed in a portion of any one of an overhead module, a rearview mirror, a central portion of an instrument panel, a pillar, an instrument cover, an interior of an instrument, a steering column, and the like.

[0042] The imaging range A1 includes a region in which vehicle-mounted devices such as a steering wheel 51 and a seat belt 52 of the vehicle exist.

[0043] Pre-adjust the installation position and viewing angle of the second camera 10B so that Figure 1 That is, the second camera 10B can capture images only in a range narrower than that of the first camera 10A, so that detailed information on a portion of the face of the driver 50 sitting in the driver's seat can be captured.

[0044] The installation position of the second camera 10B may be the same position as that of the first camera 10A, but it is desirable to set the second camera 10B at a position where an image can be captured from the front side of the driver 50 in order to more easily acquire information such as a line of sight.

[0045] The number of cameras used by the driver monitoring device 100 may be one, or may be increased to three or more as required.

[0046] The main control unit 20 is an electronic control unit (ECU) having a function of monitoring the driver's situation based on the images captured by the first camera 10A and the second camera 10B. The main control unit 20 can control the timing of capturing images, the timing of acquiring image data, the exposure amount, the gain, etc. for each of the first camera 10A and the second camera 10B. The main control unit (20) processes the image data acquired from at least one of the first camera (10A) and the second camera (10B) to acquire information representing the driving situation, automatically records and stores the obtained information, and performs output of an alarm, etc. according to the situation to support safe driving.

[0047] <Main function configuration>

[0048] Figure 2 It is shown Figure 1 A block diagram of the details of the main components of driver monitoring device 100 is shown.

[0049] Figure 2 Each function of the illustrated main control unit 20 is realized by the operation of hardware such as an electronic circuit mainly including a microcomputer (not shown) incorporated in the main control unit 20 and a program executed by the microcomputer.

[0050] like Figure 2 As shown, the main control unit 20 includes feature detection functions 21 and 22, a timing control function 23, a monitoring function unit 24, an alarm function 25, a history information recording function 26, and an object detection control function 27. The monitoring function unit 24 includes a carelessness detection function 24a, a fatigue detection function 24b, a posture collapse detection function 24c, a motion detection function 24d, a posture detection function 24e, a seat belt detection function 24f, and a steering wheel holding detection function 24g.

[0051] The feature detection function 21 detects various features of each part of the upper body and hand area of ​​the driver 50 by performing image processing on the image data acquired from the first camera 10A. The feature detection function 22 detects various features of the facial area of ​​the driver 50 by performing image processing on the image data acquired from the second camera 10B.

[0052] The timing control function 23 is capable of controlling the timing at which each of the first camera 10A and the second camera 10B performs imaging (including controlling the illumination light), the timing at which each of the first camera 10A and the second camera 10B transmits the captured image data to the main control unit 20 (or the timing at which the main control unit 20 requests an image), etc. based on the detection status of each of the feature detection functions 21 and 22.

[0053] The monitoring function unit 24 detects the current state of each of various monitoring target items required in the vehicle.

[0054] The carelessness detection function 24 a has a function of detecting whether the driver 50 is driving in a state facing a direction not related to driving based on information on the face of the driver 50 .

[0055] The fatigue detection function 24 b has a function of detecting whether the driver 50 is driving in a state with the eyes of the driver 50 closed or in a fatigued state based on information on the face of the driver 50 .

[0056] The posture collapse detection function 24 c has a function of detecting whether the driver 50 is driving in an abnormal posture different from the normal driving posture based on information on the upper body of the driver 50 and the like.

[0057] The action detection function 24 d has a function of detecting whether the driver 50 is performing a dangerous “distracted driving” action such as operating a smartphone based on information on the upper body of the driver 50 or the like.

[0058] The posture detection function 24e has a function of detecting information such as the posture of the entire body of the driver 50, the orientation in the horizontal direction and the front-back direction of the face, the direction of the line of sight, the inclination of the upper body, and the like.

[0059] The seat belt detection function 24 f has a function of detecting whether the driver 50 is wearing a seat belt.

[0060] The steering wheel holding detection function 24g has a function of detecting whether the driver 50 holds the steering wheel in a state where the driver 50 can drive.

[0061] The warning function 25 has a function of supporting safe driving of the driver 50 by notifying the driver 50 of abnormal situations, for example, by outputting a sound or an alarm sound when any function of the monitoring function unit 24 detects a situation that should warn the driver 50 of the driving state.

[0062] The history information recording function 26 records and stores information on the driving state of the driver 50 detected by each function of the monitoring function 24 in a predetermined nonvolatile recording medium in association with information such as the current date and time and the current position of the vehicle.

[0063] The object detection control function 27 may generate a trigger to execute the object detection algorithm only when the action detection function 24d or the like in the monitoring function unit 24 detects that a specific condition indicating a high possibility of distracted driving is satisfied. As will be described later, the object detection control function 27 identifies whether the specific condition is satisfied based on the coordinates of a plurality of feature points detected by the feature detection function 21 with respect to the image of the driver 50.

[0064] <Example 1 of captured image>

[0065] exist Figure 3 Two types of captured images i11 and i12 are shown in FIG. Figure 4 Two other types of captured images i21 and i22 are shown in FIG. Figure 3 and 4 The horizontal direction is represented by the x-axis, and the vertical direction is represented by the y-axis. Each position in the image is represented by a two-dimensional coordinate of x and y.

[0066] like Figure 3 The captured images i11 and i12 shown and Figure 4 In each of the captured images i21 and i22 shown, information sufficient to grasp the driving situation regarding the upper body, hands, arms, and the like of the driver 50 is depicted.

[0067] exist Figure 3 In the example of the captured image i11 shown, a situation is depicted in which the driver 50 holds the steering wheel 51 with both hands and drives in a safe state. In contrast, in the example of the captured image i12, a state is depicted in which the driver 50 holds the steering wheel 51 with his left hand, but the right hand is away from the steering wheel 51 and holds the smartphone to the lateral side close to the face, that is, the driver 50 drives while talking on the phone.

[0068] exist Figure 4 In one example of the captured image i21 shown, a situation is depicted in which the driver 50 holds the steering wheel 51 with both hands and drives in a safe state. In contrast, in the example of the captured image i22, a situation is depicted in which the driver 50 holds the steering wheel 51 with his left hand, but keeps his right hand away from the steering wheel 51 and holds the smartphone at a position away from the face, and the driver 50 drives while looking at the screen of the smartphone or operating the screen.

[0069] Figure 3 In the captured images i11 and i12 shown, the position of the feature point (A) in the wrist of the driver 50 and the position of the feature point (B) in the edge of the face (position of the ear) are shown.

[0070] It can be seen that in the captured image i11, the distance Δy between the two feature points (A and B) in the y-axis direction is relatively large. It can be seen that in the captured image i12, the distance Δy between the two feature points (A and B) in the y-axis direction is relatively small.

[0071] By checking the magnitude relationship regarding the magnitude of the distance Δy detected in the image, it is possible to easily distinguish between a safe driving state as in the captured image i11 and a state in which there is a high possibility of driving while talking on the phone as in the captured image i12 .

[0072] At the same time, Figure 4 In the captured images i21 and i22 shown, the position of the feature point (A) in the wrist and the position of the feature point (C) in the neck of the driver 50 are shown.

[0073] It can be seen that in the captured image i21, the distance Δx between the two feature points (A, C) in the x-axis direction is relatively large. It can be seen that in the captured image i22, the distance Δx between the two feature points (A, C) in the x-axis direction is relatively small.

[0074] By checking the magnitude relationship regarding the magnitude of the distance Δx detected in the image, it is possible to easily distinguish between a safe driving state as in the captured image i21 and a state in which there is a high possibility of driving while operating a smartphone as in the captured image i22 .

[0075] <Operation of the main control unit 1>

[0076] exist Figure 5 An example of the operation of the main control unit 20 is shown in FIG. That is, when the driver monitoring system (DMS) of the vehicle activates the function of monitoring "distracted driving" performed by the driver 50, the computer of the main control unit 20 starts Figure 5 The following describes the operation shown in Figure 5 The operations shown in .

[0077] The main control unit 20 periodically repeats the process for detecting the driving posture of the driver 50 at short time intervals (S11). That is, the image data of each frame captured by the first camera 10a is periodically input to the main control unit 20. The feature detection function 21 detects the coordinate values ​​of the feature points (A, B, and C) representing the posture of the driver 50 for each frame.

[0078] The object detection control function 27 of the main control unit 20 acquires the coordinate values ​​of the feature points (key points: A and B) from the feature detection function 21, calculates the distance Δy in the y-axis direction between the feature points (A and B), and compares the distance Δy with a threshold value (S12).

[0079] For example, if the distance Δy is large, as in Figure 3 If the distance Δy is small, as in the captured image i11 shown, the condition in S12 is not satisfied. The process thus returns from S12 to S11. In contrast, if the distance Δy is small, as in the captured image i12, the condition in S12 is satisfied, and the process thus proceeds from S12 to S13.

[0080] The main control unit 20 starts executing a predetermined object detection algorithm in S13 by being triggered by the output of the object detection control function 27, that is, in order to be able to detect objects such as smartphones in addition to feature points such as the joints of the driver 50, detailed image processing is performed, and processing such as pattern recognition of the object is also performed.

[0081] As a result of the object detection in S13 , the main control unit 20 identifies whether a smartphone is detected in the image in S14 .

[0082] If a smartphone is detected by the main control unit 20 in S14, the process proceeds to S15. In S15, the main control unit 20 recognizes that the driver 50 is in a "distracted driving" state in which the driver 50 is on the phone, and outputs an alarm for "distracted driving".

[0083] If the main control unit 20 does not detect the smartphone in S14, the process proceeds to S 16. In S16, the main control unit 20 recognizes that the driver 50 is not in an inappropriate driving state even when the distance Δy is small.

[0084] The object detection control function 27 of the main control unit 20 acquires the coordinate values ​​of the feature points (A and C) from the feature detection function 21 , calculates the distance ΔX between the feature points (A and C) in the X-axis direction, and compares the distance ΔX with a threshold value ( S17 ).

[0085] For example, if the distance Δx is large, as in Figure 4 If the distance Δx is small, as in the captured image i21 shown, the condition in S17 is not satisfied, and thus the process returns from S17 to S11. In contrast, if the distance Δx is small, as in the captured image i22, the condition in S17 is satisfied, and thus the process proceeds from S17 to S18.

[0086] The main control unit 20 starts executing a predetermined object detection algorithm in S18 by a trigger output by the object detection control function 27. That is, in order to be able to detect, in addition to feature points such as joints of the driver 50, objects such as smartphones, detailed image processing is performed, and processing such as pattern recognition of objects is also performed.

[0087] As a result of the object detection in S18 , the main control unit 20 identifies in S19 whether a smartphone is detected in the image.

[0088] If the smartphone is detected by the main control unit 20 in S19, the process proceeds to S20. In S20, the main control unit 20 recognizes that the driver 50 is in a "distracted driving" state where the driver 50 is operating a smartphone, and outputs an alarm for "distracted driving".

[0089] If the main control unit 20 in S19 does not detect the smartphone, the process proceeds to S21. In S21, the main control unit 20 recognizes that the driver 50 is not in an inappropriate driving state even when the distance Δx is small. Therefore, in this case, no alarm is output.

[0090] <Example 2 of Captured Image>

[0091] exist Figure 6 Two types of captured images i31 and i32 are shown in FIG. Figure 7 Two other types of captured images i41 and i42 are shown in FIG. Figure 6 and Figure 7 In the image, the horizontal direction is represented by the x-axis and the vertical direction is represented by the y-axis. Each position in the image is represented by a two-dimensional coordinate of x and y.

[0092] Figure 6 The captured images i31 and i32 shown in FIG. Figure 7 In each of the captured images i41 and i42 shown, information sufficient to grasp the driving situation regarding the upper body, hands, arms, etc. of the driver 50 is depicted.

[0093] exist Figure 6 In the example of the captured image i31 shown, the driver 50 holds the steering wheel 51 with both hands, so it can be considered that the driver 50 is driving in a safe state. In contrast, in the example of the captured image i32, a state is depicted in which the driver 50 holds the steering wheel 51 with his left hand, but the right hand is away from the steering wheel 51 and holds the smartphone on the lateral side close to the face, that is, a problematic situation in which the driver 50 drives while talking on the phone.

[0094] exist Figure 7In the example of the captured image i41 shown, a situation is depicted in which the driver 50 holds the steering wheel 51 with both hands and drives in a safe state. In contrast, in the example of the captured image i42, a situation is depicted in which the driver 50 holds the steering wheel 51 with his left hand, but the right hand is away from the steering wheel 51 and holds the smartphone at a position away from the face, and the driver 50 drives while looking at the screen of the smartphone or operating the screen.

[0095] Figure 6 In each of the captured images i31 and i32 shown, there are depicted the position of a feature point in the right wrist, the position of a feature point corresponding to a joint of the right arm, and the position of a feature point in the right shoulder of the driver 50. The angle at the joint portion of the right arm that can be specified by the positions of the three feature points is represented as θ.

[0096] It can be seen here that in the captured image i31, the angle θ at the joint portion of the right arm is large, while in the captured image i32, the angle θ at the joint portion of the right arm is small.

[0097] By checking the size relationship regarding the size of the angle θ at the joint portion detected in the image, it is possible to easily distinguish between a safe driving state as in captured image i31 and a state with a high possibility of driving while talking on the phone as in captured image i32.

[0098] At the same time, Figure 7 In the captured images i41 and i42 shown, the positions of feature points in the right wrist, the positions of feature points corresponding to the joints of the right arm, and the positions of feature points in the right shoulder of the driver 50 are depicted. The angle at the joint portion of the right arm that can be specified by the positions of the three feature points is represented as θ.

[0099] It can be seen here that in the captured image i41, the angle θ at the joint portion of the right arm is large, while in the captured image i42, the angle θ at the joint portion of the right arm is small.

[0100] By examining the size relationship regarding the size of the angle θ at the joint portion detected in the image, it is possible to easily distinguish a safe driving state as in captured image i41 from a state in which there is a high possibility of driving while operating a smartphone as in captured image i42.

[0101] <Operation of the main control unit 2>

[0102] exist Figure 8 2 shows an operation example 2 of the main control unit 20. That is, when the driver monitoring system of the vehicle activates the function of monitoring "distracted driving" performed by the driver 50, the computer of the main control unit 20 starts Figure 8 The operation shown. Figure 8 The control content shown in is designed to assume that representations such as Figure 6 The captured images i31 and i32 shown in FIG. Figure 7 The captured images i41 and i42 shown in FIG. 1 are processed in the following manner. Figure 8 The operations shown in .

[0103] The main control unit 20 periodically and repeatedly performs processing for detecting the driving posture of the driver 50 at short time intervals (S31). That is, image data of each frame captured by the first camera 10A is periodically input to the main control unit 20. The feature detection function 21 detects the coordinate values ​​of feature points (the position of the wrist, the position of the arm joint, and the position of the shoulder) representing the posture of the driver 50 for each frame.

[0104] The object detection control function 27 of the main control unit 20 acquires the coordinate values ​​of the feature points from the feature detection function 21 and calculates the angle θ at the joint position of the arm in S32. Then, the main control unit 20 compares the angle θ at the joint position with predetermined thresholds θ1 and θ2 (S33 and S34).

[0105] For example, if the angle θ at the joint position is large, such as Figure 6 The captured image i31 or Figure 7 As shown in the captured image i41, the conditions in S33 and S34 are not satisfied, and therefore the process returns from S33 to S31.

[0106] In contrast, if the angle θ at the joint position is equal to or smaller than the threshold θ1 in the captured image i32 , the condition in S33 is satisfied, and thus the process proceeds from S33 to S35 .

[0107] The main control unit 20 starts executing a predetermined object detection algorithm in S35 by a trigger output by the object detection control function 27. That is, in order to be able to detect an object such as a smartphone in addition to feature points such as the joints of the driver 50, detailed image processing is performed, and processing such as pattern recognition of the object is also performed.

[0108] As a result of the object detection in S35 , the main control unit 20 identifies in S36 whether a smartphone is detected in the image.

[0109] If a smartphone is detected by the main control unit 20 in S36, the process proceeds to S37. In S37, the main control unit 20 recognizes that the driver 50 is in a "distracted driving" state in which the driver 50 is on the phone, and outputs an alarm for "distracted driving".

[0110] If the main control unit 20 in S36 does not detect the smartphone, the process proceeds to S41. In S41, the main control unit 20 recognizes that the driver 50 is not in an inappropriate driving state even in the case where the angle θ of the joint position is small.

[0111] In contrast, if the angle θ at the joint position is greater than the threshold θ1 and equal to or less than the threshold θ2, as in the captured image i42, the condition in S34 is satisfied, and thus the process proceeds from S34 to S38.

[0112] The main control unit 20 starts executing a predetermined object detection algorithm in S38 by a trigger output by the object detection control function 27. That is, in order to be able to detect an object such as a smartphone in addition to feature points such as the joints of the driver 50, detailed image processing is performed, and processing such as pattern recognition of the object is also performed.

[0113] As a result of the object detection in S38 , the main control unit 20 identifies in S39 whether a smartphone is detected in the image.

[0114] If the smartphone is detected by the main control unit 20 in S39, the process proceeds to S40. In S40, the main control unit 20 recognizes that the driver 50 is in a "distracted driving" state where the driver 50 is operating a smartphone, and outputs an alarm for "distracted driving".

[0115] If the main control unit 20 does not detect the smartphone in S39, the process proceeds to S41. In S41, even if the angle θ of the joint position is small, the main control unit 20 recognizes that the driver 50 is not in an inappropriate driving state. Therefore, in this case, no alarm is output.

[0116] The present invention is not limited to the above-described embodiments, and may be appropriately modified, improved, etc. Furthermore, the materials, shapes, sizes, numbers, arrangement positions, etc. of the components in the above-described embodiments are freely selectable and are not limited as long as the present invention can be achieved.

[0117] For example, as for "distracted driving" performed by the driver monitored by the driver monitoring device 100, in addition to the operation on the smartphone, it is assumed that the driver drives while eating, drives while smoking, etc. Figure 5 The objects identified in S13 and S18 and Figure 8The objects identified in S35 and S38 in can be modified to include food, drink, and cigarettes. Even in the case of monitoring "distracted driving" caused by eating, drinking, or smoking, the difference in the type of target object does not affect the distances Δx and Δy between feature points and the angle θ at the joint much differently than in the case of a smartphone, so Figure 5 The control contents and Figure 8 The controls shown in can be used as usual.

[0118] Can be used Figure 5 The processing and Figure 8 That is, when either or both of the distance Δx or Δy and the joint angle θ satisfy predetermined conditions, the main control unit 20 can detect the distance Δx or Δy and the angle θ at the joint, and can perform control to proceed to any of the processes of S13, S18, S35, and S38.

[0119] As described above, when the main control unit 20 of the driver monitoring device 100 executes Figure 5 In the operation shown, in a normal driving state where the conditions in S12 or S17 are not satisfied, it is sufficient to detect the coordinates of multiple feature points (A, B, and C), and there is no need to initiate complex image processing to detect an object such as a smartphone.

[0120] When the main control unit 20 executes Figure 8 In the operation shown in FIG. 1 , in a normal driving state where the conditions in S33 or S34 are not satisfied, it is sufficient to detect a plurality of feature points and calculate the angle θ at the joint, and there is no need to initiate complex image processing to detect an object such as a smartphone. Figure 5 or Figure 8 In any of the control cases shown, the load of processing in the main control unit 20 under normal conditions can be reduced. This can reduce the cost of the main control unit 20, prevent heat generation, and extend the product life. In addition, the amount of power consumed by the main control unit 20 is also reduced.

[0121] Here, the features of the driver monitoring device and the monitoring program according to the embodiment of the present invention are briefly summarized and listed in [1] to [5] below.

[0122] [1] A driver monitoring device, comprising:

[0123] an image acquisition unit (feature detection function 21 ) configured to acquire an image of the driver; and

[0124] a determination unit (action detection function 24d) configured to determine whether the driver is using a predetermined detection target that is not related to the driving operation based on the acquired image, wherein

[0125] The determination unit extracts a first feature point (B or C) included in the driver's neck or head and a second feature point (A) included in the driver's arm or hand from the image, and performs processing (S12, S13, S17 and S18) to determine whether the detection target is included in the image when the difference (Δy or Δx) between the coordinates of the first feature point and the coordinates of the second feature point is less than a reference level.

[0126] According to the driver monitoring device having the configuration according to [1] above, in the case of normal driving by the driver, only a simple process of monitoring the coordinates of two feature points or three feature points needs to be performed, thereby reducing the processing load in the determination unit. Only when the possibility of distracted driving is detected based on the coordinates of the feature points, a process for determining whether the detection target is included in the image is performed, thereby detecting a case of distracted driving related to a smartphone or the like.

[0127] [2] The driver monitoring device according to [1] above, wherein

[0128] The determination unit determines whether a difference in one of an x-coordinate and a y-coordinate with respect to a difference between a coordinate of the first feature point and a coordinate of the second feature point is smaller than a reference level ( S12 and S17 ).

[0129] According to the driver monitoring device having the configuration according to the above [2], by comparing the difference (Δx) between the x-coordinate of the first feature point and the x-coordinate of the second feature point with the reference level, it is possible to identify, for example Figure 4 The two cases are shown in captured images i21 and i22. By comparing the difference (Δy) between the y coordinate of the first feature point and the y coordinate of the second feature point with the reference level, it is possible to identify, for example Figure 3 Two situations are shown in captured images i11 and i12.

[0130] [3] A driver monitoring device comprising:

[0131] an image acquisition unit (feature detection function 21 ) configured to acquire an image of the driver; and

[0132] a determination unit (action detection function 24d) configured to determine whether the driver is using a predetermined detection target that is not related to the driving operation based on the acquired image, wherein

[0133] The determination unit detects at least three feature points, calculates an angle (θ) at a joint position based on the coordinates of these feature points, and performs processing (S32, S33, S34, S35, S38) for determining whether a predetermined detection target is included in the image when the calculated angle satisfies a predetermined condition, and the at least three feature points include points at the joint positions of the driver's body on the input image.

[0134] According to the driver monitoring device having the configuration according to [3] above, in the case of normal driving by the driver, only a simple process of detecting the coordinates of the three feature points and calculating the angle at the joint is required, thereby reducing the processing load in the determination unit. Only when the possibility of distracted driving at the joint is detected based on the detected angle, a process for determining whether the detection target is included in the image is performed, thereby detecting a case of distracted driving related to a smartphone or the like.

[0135] [4] The driver monitoring device according to [1] above, wherein

[0136] The determination unit detects the mobile terminal near the driver's hand as a detection object.

[0137] According to a driver monitoring device having a configuration according to [4] above, a mobile terminal that may be held by the driver during driving is detected, and thus even when the driver's face or sight is facing forward, a situation in which normal driving is impossible, such as in the case of careless driving, is easily detected.

[0138] [5] A monitoring program executable by a computer that controls a monitoring device, the monitoring device comprising: an image acquisition unit configured to acquire an image of a driver; and a determination unit configured to determine whether the driver is using a predetermined detection target that is not related to a driving operation based on the acquired image, the monitoring program comprising:

[0139] a process of extracting a first feature point included in a neck or head of a driver and a second feature point included in an arm or hand of the driver from the captured image ( S11 ); and

[0140] When the difference between the coordinates of the first feature point and the coordinates of the second feature point is smaller than the reference level, a process of determining whether the detection target is included in the image is performed ( S12 , S13 , S17 , and S18 ).

[0141] According to the monitoring program having the configuration described in [5] above, in the case of normal driving by the driver, only a simple process of monitoring the coordinates of two feature points or three feature points needs to be performed, thereby reducing the processing load in the determination unit. Only when the possibility of distracted driving is detected based on the coordinates of the feature points, a process for determining whether the detection target is included in the image is performed, thereby detecting a case of distracted driving related to a smartphone or the like.

[0142] This application is based on Japanese patent application (No. 2022-194327) filed on December 5, 2022, the contents of which are incorporated herein by reference.

[0143] Reference numerals list

[0144] 10A: First Camera

[0145] 10B: Second Camera

[0146] 20: Main control unit

[0147] 21, 22: Feature detection function

[0148] 23: Timing control function

[0149] 24: Monitoring function unit

[0150] 24a: Carelessness detection function

[0151] 24b: Fatigue detection function

[0152] 24c: Posture collapse detection function

[0153] 24d: Motion detection function

[0154] 24e: Posture detection function

[0155] 24f: Seat belt detection function

[0156] 24g: Steering wheel hold detection function

[0157] 25: Alarm function

[0158] 26: Historical information recording function

[0159] 27: Object detection control function

[0160] 50: Driver

[0161] 51: Steering Wheel

[0162] 52: Seat belt

[0163] 100: Driver monitoring device

[0164] A1, A2: Imaging range

[0165] i11, i12, i21, i22, i31, i32, i41, i42: the captured images

[0166] Δx, Δy: distance

[0167] θ: Angle of the joint

[0168] θ1, θ2: thresholds.

Claims

1. A driver monitoring device, comprising: an image acquisition unit configured to acquire an image of the driver; a determination unit configured to determine whether the driver is using a predetermined detection target that is not related to driving operation based on the acquired image, wherein The determination unit extracts a first feature point included in the driver's neck or head and a second feature point included in the driver's arm or hand from the image, and when the difference between the coordinates of the first feature point and the coordinates of the second feature point is less than a reference level, performs processing to determine whether the detection target is included in the image.

2. The driver monitoring device according to claim 1, wherein The determination unit determines whether a difference in one of an x-coordinate and a y-coordinate with respect to a difference between a coordinate of the first feature point and a coordinate of the second feature point is smaller than the reference level.

3. A driver monitoring device, comprising: an image acquisition unit configured to acquire an image of the driver; as well as a determination unit configured to determine whether the driver is using a predetermined detection target that is not related to driving operation based on the acquired image, wherein The determination unit detects at least three feature points including points of joint positions of the driver's body on the input image, calculates angles at the joint positions based on the coordinates of these feature points, and performs processing to determine whether the predetermined detection target is included in the image when the calculated angle meets predetermined conditions.

4. The driver monitoring device according to claim 1, wherein The determination unit detects a mobile terminal near the driver's hand as the detection target.

5. A monitoring program executable by a computer controlling a monitoring device, the monitoring device comprising: an image acquisition unit configured to acquire an image of the driver; and a determination unit configured to determine whether the driver is using a predetermined detection target that is not related to a driving operation based on the acquired image, the monitoring program comprising: A process of extracting a first feature point included in the neck or head of the driver and a second feature point included in the arm or hand of the driver from the captured image; and When the difference between the coordinates of the first feature point and the coordinates of the second feature point is smaller than a reference level, a process of determining whether the detection target is included in the image is performed.

Images