Driver monitoring device, storage medium storing a driver monitoring computer program, and driver monitoring method

Abstract

Provided is a driver monitoring device, a storage medium storing a driver monitoring computer program, and a driver monitoring method, in which, in a case where it is determined based on an image obtained by capturing a vicinity of a driver's seat that a driver is not seated on the driver's seat, it is possible to determine whether or not the driver is seated on the driver's seat in a short time by immediately using a seat sensor of a seat other than the driver's seat. The driver monitoring device acquires a number of passengers who are on board a vehicle at the time when the vehicle starts traveling, determines whether or not a driver is seated on a driver's seat of the vehicle based on an image of a vicinity of the driver's seat of the vehicle captured by a capturing unit, and in a case where it is determined that the driver is not seated on the driver's seat, determines that the driver is not seated on the driver's seat in a case where a number of seated persons detected by seat sensors respectively provided in seats other than the driver's seat of the vehicle is identical to the number of passengers, and determines that the driver is seated on the driver's seat in a case where the number of seated persons is one less than the number of passengers.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a driver monitoring device, a storage medium storing a driver monitoring computer program, and a driver monitoring method. BACKGROUND

[0002] An automatic control system of a vehicle monitors a degree of driver's engagement in driving using a sensor or the like (for example, refer to Patent Literature 1). The degree of driver's engagement in driving required differs according to a level of automatic control.

[0003] The automatic control system monitors the driver and determines whether the driver is engaged in driving as required according to the level of automatic control. As a means of performing the above monitoring, for example, a seat sensor of a driver's seat, a buckle sensor that detects whether the driver wears a seat belt of the driver's seat, and a monitoring camera that captures an image for monitoring a state of the driver can be cited.

[0004] The automatic control system of the vehicle controlled at the automatic control level 3 requires that the driver wears the seat belt of the driver's seat and that the state in which the driver can take over the driving operation can be detected. Therefore, the automatic control system of the vehicle controlled at the automatic control level 3 preferably has the buckle sensor of the driver's seat, the seat sensor of the driver's seat, and the monitoring camera.

[0005] PRIOR ART DOCUMENTS

[0006] Patent Literature 1: International Publication No. 2017 / 158726 SUMMARY

[0007] PROBLEMS TO BE SOLVED BY THE INVENTION

[0008] In consideration of the case where a vehicle controlled at the automatic control level 3 is manufactured based on a vehicle that can be controlled at the automatic control level 2 or less, it is advantageous in terms of cost to utilize components such as sensors already mounted on the vehicle. The buckle sensor, the seat sensor of a seat other than the driver's seat, and the monitoring camera are mounted on the vehicle controlled at the automatic control level 2 or less.

[0009] However, in driving of the vehicle that can be controlled at the automatic control level 2 or less, it is determined that the driver is seated on the driver's seat and can operate the vehicle, and the driver's seat is a prerequisite, so the seat sensor is not mounted on the driver's seat.

[0010] By monitoring a camera to take an image of the vicinity of the driver's seat, a predetermined part of the driver is recognized within the image, whereby it is possible to determine whether the driver is seated on the driver's seat. For example, it is possible to determine that the driver is seated on the driver's seat in a case where the driver's face is recognized within the image. However, when the face of the driver seated on the driver's seat is oriented sideways, the driver's face is not recognized within the image. Then, even when the driver's face is not recognized within the image, the driver seated on the driver's seat can be oriented to the front, and thus, it is necessary to wait for a predetermined time period to determine whether the driver's face is recognized within the image. Thus, there is a problem that a certain length of time is required to determine whether the driver is seated based on the image.

[0011] Thus, an object of the present disclosure is to provide a driver monitoring device that, in a case where it is determined based on an image taken of the vicinity of a driver's seat that a driver is not seated on the driver's seat, can determine whether the driver is seated on the driver's seat in a short time by immediately using a seat sensor of a seat other than the driver's seat.

[0012] Technical solution for solving the problem

[0013] According to one embodiment, a driver monitoring device is provided. The driver monitoring device has: an acquisition unit that acquires the number of passengers who are on board a vehicle at the time when the vehicle starts traveling; a first determination unit that determines whether a driver is seated on a driver's seat of the vehicle based on an image of the vicinity of the driver's seat taken by a camera; and a second determination unit that, in a case where it is determined by the first determination unit that the driver is not seated on the driver's seat, determines that the driver is not seated on the driver's seat in a case where the number of people seated is consistent with the number of passengers, and determines that the driver is seated on the driver's seat in a case where the number of people seated is one less than the number of passengers, the number of people seated indicating the number of people seated detected by seat sensors respectively provided in seats other than the driver's seat in the vehicle.

[0014] In addition, in the driver monitoring device, it is preferable to further have a control unit that, in a case where it is determined by the second determination unit that the driver is not seated on the driver's seat, notifies the driver to sit on the driver's seat via a notification unit.

[0015] In addition, in the driver monitoring device, it is preferable to further have a third determination unit that, in a case where it is determined by the second determination unit that the driver is seated on the driver's seat, determines whether the driver is seated on the driver's seat based on a plurality of images taken by the camera during a predetermined time period after the determination by the second determination unit.

[0016] Further, in the driver monitoring device, preferably, a control section is further provided, which notifies the driver to sit on the driver's seat via a notification section in a case where the driver is determined not to sit on the driver's seat by the third determination section.

[0017] Further, in the driver monitoring device, preferably, a control section is further provided, which notifies the driver to sit on the driver's seat via a notification section in a case where the number of persons sitting and the number of persons riding are not consistent, and the number of persons sitting is not one less than the number of persons riding.

[0018] Further, in the driver monitoring device, preferably, a fourth determination section is further provided, which compares the number of persons sitting and the number of persons riding in a case where the driver is determined to sit on the driver's seat by the first determination section, and obtains a value obtained by subtracting the number of persons riding from the number of persons sitting and adding one as the number of objects placed on seats other than the driver's seat in a case where the number of persons sitting is more than a value obtained by subtracting one from the number of persons riding, and the second determination section determines that the driver is not sitting on the driver's seat in a case where a value obtained by subtracting the number of objects from the number of persons sitting is consistent with the number of persons riding after the number of objects is obtained by the fourth determination section in a case where the driver is determined not to sit on the driver's seat by the first determination section, and determines that the driver is sitting on the driver's seat in a case where the value obtained by subtracting the number of objects from the number of persons sitting is one less than the number of persons riding.

[0019] According to another embodiment, a storage medium storing a driver monitoring computer program is provided. The driver monitoring computer program causes a processor to execute processing including: acquiring a number of persons riding in a vehicle at the time when the vehicle starts traveling; determining whether a driver is sitting on a driver's seat of the vehicle on the basis of an image of a vicinity of the driver's seat of the vehicle captured by a capturing section; and determining that the driver is not sitting on the driver's seat in a case where the number of persons sitting and the number of persons riding are consistent when it is determined that the driver is not sitting on the driver's seat, and determining that the driver is sitting on the driver's seat in a case where the number of persons sitting is one less than the number of persons riding, the number of persons sitting indicating a number of persons sitting detected by a sitting sensor provided for each seat other than the driver's seat in the vehicle.

[0020] According to another embodiment, a driver monitoring method is provided. In the driver monitoring method, a driver monitoring device performs: acquiring the number of passengers who get on a vehicle when the vehicle starts traveling; determining whether a driver is seated on a driver's seat based on an image of the vicinity of the driver's seat of the vehicle captured by a camera; and when it is determined that the driver is not seated on the driver's seat, determining that the driver is not seated on the driver's seat when the number of seated persons and the number of passengers are consistent, and determining that the driver is seated on the driver's seat when the number of seated persons is one less than the number of passengers, the number of seated persons indicating the number of persons seated detected by a seat sensor provided for each seat other than the driver's seat in the vehicle.

[0021] Effects of Invention

[0022] The driver monitoring device according to the present disclosure can determine whether a driver is seated on a driver's seat in a short time by using a seat sensor for a seat other than the driver's seat when it is determined that the driver is not seated on the driver's seat based on an image obtained by capturing the vicinity of the driver's seat. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 (A) of FIG. 1 is a diagram illustrating one example of a state of seats in a vehicle cabin and a driver.

[0024] Figure 1 (B) of FIG. 1 is a diagram illustrating one example of a state of seats in a vehicle cabin and a driver.

[0025] Figure 2 (A) of FIG. 1 is a diagram illustrating one example of a state of seats in a vehicle cabin and a driver.

[0026] Figure 2 (B) of FIG. 1 is a diagram illustrating one example of a state of seats in a vehicle cabin and a driver.

[0027] Figure 2 (C) of FIG. 1 is a diagram illustrating one example of a state of seats in a vehicle cabin and a driver.

[0028] Figure 3 is a diagram illustrating a schematic configuration of a vehicle on which a monitoring system including the driver monitoring device according to the present embodiment is installed.

[0029] Figure 4 is one example of a flowchart of actions related to a driver monitoring process of the driver monitoring device according to the present embodiment.

[0030] Figure 5 (A) of FIG. 1 is a diagram illustrating one example of a state of seats in a vehicle cabin and a driver.

[0031] Figure 5 (A) is a diagram showing one example of the state of the seat and the driver in the vehicle cabin (first).

[0032] Figure 6 is an action flowchart showing Modification Example 1 related to the driver monitoring process of the driver monitoring device of the present embodiment.

[0033] Figure 7 is an action flowchart showing Modification Example 2 related to the driver monitoring process of the driver monitoring device of the present embodiment.

[0034] Figure 8 (A) is a diagram showing one example of the state of the seat and the driver in the vehicle cabin (second).

[0035] Figure 8 (B) is a diagram showing one example of the state of the seat and the driver in the vehicle cabin (second).

[0036] Figure 8 (C) is a diagram showing one example of the state of the seat and the driver in the vehicle cabin (second).

[0037] Explanation of Reference Signs

[0038] 1 monitoring system

[0039] 2 monitoring camera

[0040] 3a, 3b, 3c seat sensor

[0041] 4 user interface

[0042] 4a display device

[0043] 10 vehicle

[0044] 11 driver monitoring device

[0045] 21 communication interface

[0046] 22 memory

[0047] 23 processor

[0048] 230 acquisition section

[0049] 231 determination section

[0050] 232 control section

[0051] 12 in-vehicle network DETAILED DESCRIPTION

[0052] Figure 1 (A) andFigure 1 (B) is a summary of the operation of the driver monitoring device 11 of the present embodiment, Figure 1 (A) is a view showing the vehicle 10, Figure 1 (B) is a view showing the driver 30 photographed by the monitoring camera 2. Figure 2 (A) ~ Figure 2 (C) is a view showing one example of the state of the seats and the driver 30 in the vehicle cabin 10A.

[0053] As Figure 1 (A) shows, the vehicle 10 has the driver monitoring device 11. As Figure 1 (B) shows, in the vehicle cabin 10A, the monitoring camera 2 capable of photographing the driver 30 seated on the driver's seat 40 is disposed on the steering column 50.

[0054] As Figure 2 (A) ~ Figure 2 (C) shows, in the vehicle cabin 10A of the vehicle 10, the driver's seat 40 on which the driver 30 is seated, the passenger's seat 41, and the rear seats 42, 43 are disposed. The seat sensors 3a to 3c are disposed on the seats 41 to 43 other than the driver's seat, respectively. The seat sensors 3a to 3c detect the seating of the occupant and output a seat signal indicating that the occupant is seated to the driver monitoring device 11. The driver monitoring device 11 can acquire the number of seated persons in a short time on the basis of the seat signals input from the seat sensors 3a to 3c. In addition, it is preferable that a buckle sensor (not shown) that detects the state in which the occupant wears a seat belt is also disposed on each seat.

[0055] The driver monitoring device 11 acquires the number of occupants, which indicates the number of persons who get on the vehicle 10 at the time when the vehicle 10 starts running. For example, the driver monitoring device 11 acquires the number of occupants input by the driver via the user interface 4 (refer to Figure 3 ), and acquires the number of occupants input by the driver via the user interface 4 (refer to Figure 2 (A) ~ Figure 2 (C) shows, the number of occupants is only one person.

[0056] The driver monitoring device 11 determines whether the driver 30 is seated on the driver's seat 40 on the basis of the monitoring image photographed by the monitoring camera 2 at the time when the vehicle 10 starts running. The driver monitoring device 11 can determine whether the driver 30 is seated on the driver's seat 40 in a short time by using a recognizer such as a neural network (CNN) to determine whether the face of the driver is represented in the monitoring image.

[0057] In Figure 2In the example shown in (A), the driver 30 is seated in the driver's seat 40 facing the front of the vehicle 10. The driver's face is shown in the monitoring image as facing straight ahead with respect to the monitoring camera 2. The driver monitoring device 11 determines, based on the monitoring image, that the driver 30 is seated in the driver's seat 40.

[0058] On the other hand, in the example shown in (B), the driver 30 is seated in the rear seat 42. The driver's face is not shown in the monitoring image in a recognizable manner. The driver monitoring device 11 determines, based on the monitoring image, that the driver 30 is not seated in the driver's seat 40. The driver monitoring device 11 further determines whether the driver 30 is seated in the driver's seat 40 based on the detection results of the seat sensors 3a to 3c in the case where it is determined based on the monitoring image that the driver 30 is not seated in the driver's seat 40. Figure 2

[0059] The driver monitoring device 11 acquires, based on the seat signals output from the seat sensors 3a to 3c, a seat number indicating the number of persons seated detected by the seat sensors 3a to 3c respectively arranged in the seats other than the driver's seat in the vehicle 10. The driver monitoring device 11 compares the seat number with the occupant number.

[0060] In the example shown in (A), the seat number is one and the occupant number is also one. Since the seat number and the occupant number coincide, the driver monitoring device 11 determines that the driver 30 is not seated in the driver's seat 40. Figure 2

[0061] In the example shown in (B), the seat number is zero and the occupant number is one. The driver monitoring device 11 determines that the driver 30 is seated in the driver's seat 40 in the case where the seat number is one less than the occupant number. Figure 2 In the example shown in (C), the driver 30 is seated in the driver's seat 40 but facing the passenger's seat 41, not facing the front of the vehicle 10. The driver's face is not shown in the monitoring image as facing straight ahead with respect to the monitoring camera 2. The driver monitoring device 11 determines, based on the monitoring image, that the driver 30 is not seated in the driver's seat 40.

[0062] Figure 2 In the example shown in (C), the seat number is zero and the occupant number is one. The driver monitoring device 11 determines that the driver 30 is seated in the driver's seat 40 in the case where the seat number is one less than the occupant number.

[0063] As explained above, the driver monitoring device 11 of the present embodiment is able to determine whether the driver 40 is seated in the driver's seat 40 in a short time by immediately using the seat sensors 3a to 3c of the seats other than the driver's seat in the case where it is determined based on the image taken of the vicinity of the driver's seat that the driver is not seated in the driver's seat.

[0064] Figure 3 ​​​This is a schematic diagram of a vehicle 10 equipped with a monitoring system 1 that includes the driver monitoring device 11 of this embodiment. The monitoring system 1 includes a monitoring camera 2, seating sensors 3a, 3b, and 3c, a user interface (UI) 4, and the driver monitoring device 11, etc.

[0065] The surveillance camera 2, seating sensors 3a, 3b, 3c, UI4, and driver monitoring device 11 are connected in a communicable manner via an in-vehicle network 12 that follows a standard controller area network.

[0066] Surveillance camera 2 is configured inside the vehicle compartment 10A to capture surveillance images including the face of the driver of vehicle 10. Surveillance camera 2 is an example of a camera unit. The surveillance images show the situation near the driver's seat. Figure 1 As shown in (B), the monitoring camera 2 is, for example, mounted on the steering column 50. Alternatively, the monitoring camera 2 can also be mounted on the steering wheel, rearview mirror, instrument panel, instrument cover, etc., inside the vehicle.

[0067] The surveillance camera 2 captures a surveillance image containing the driver's face at a predetermined period of time. The surveillance camera 2 has a two-dimensional detector composed of an array of photoelectric conversion elements sensitive to infrared light, such as CCD or C-MOS, and an imaging optical system that images the area to be photographed onto the two-dimensional detector. The predetermined period can be, for example, set to 0.1 to 0.5 seconds. The surveillance camera 2 preferably has a projector along with the two-dimensional detector. The projector is an LED (light-emitting diode), for example, two near-infrared LEDs arranged on both sides of the imaging optical system. By illuminating the driver with near-infrared light, the driver's face can be photographed without causing discomfort, even in low-light conditions such as at night. Alternatively, a bandpass filter to remove light components other than near-infrared wavelengths can be provided inside the imaging optical system, and a visible light cutoff filter to remove light other than near-infrared light emitted from the near-infrared LEDs can be provided in front of the projector.

[0068] The seat occupancy sensors 3a, 3b, 3c are respectively provided in seats other than the driver's seat 40 in the vehicle 10. The seat occupancy sensor 3a is provided in the front passenger's seat 41, the seat occupancy sensor 3b is provided in the right rear seat 42, and the seat occupancy sensor 3c is provided in the left rear seat 43. The seat occupancy sensors 3a to 3c detect the occupancy of an occupant and output a seat occupancy signal indicating that a seat is occupied to the driver monitoring device 11. The seat occupancy sensors 3a to 3c preferably continue to output the seat occupancy signal to the driver monitoring device 11 during a period in which the occupancy of an occupant is detected. As the seat occupancy sensors 3a to 3c, for example, pressure-sensitive sensors can be used. The pressure-sensitive sensors output a seat occupancy signal when a pressure of a predetermined reference pressure or more is detected and do not output a seat occupancy signal in a case where a pressure less than the reference pressure is detected.

[0069] The UI 4 is an example of a notification section. The UI 4 is controlled by the driver monitoring device 11 and notifies the driver of a seat notification and the like that requests the driver to sit in a manner that enables the driver to be imaged by the monitoring camera 2. The UI 4 has a display device 4a such as a liquid crystal display or a touch panel in order to display the seat notification and the like. Information displayed on the display device 4a is preferably also visually recognized by an occupant sitting in the rear seats 42, 43 of the vehicle cabin 10A. In addition, the UI 4 can also have an audio output device (not shown) for notifying the driver of the seat notification and the like. In addition, the UI 4 has, for example, a touch panel or an operation button as an input device through which the driver inputs operation information for the vehicle 10. As the operation information, for example, the number of occupants can be cited. The UI 4 outputs the operation information input thereto to the driver monitoring device 11 via the in-vehicle network 12.

[0070] The driver monitoring device 11 performs a retrieval process, a determination process, and a control process. Therefore, the driver monitoring device 11 has a communication interface (IF) 21, a memory 22, and a processor 23. The communication interface 21, the memory 22, and the processor 23 are connected via a signal line 24. The communication interface 21 has an interface circuit for connecting the driver monitoring device 11 to the in-vehicle network 12.

[0071] The memory 22 is an example of a storage section and has, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. The memory 22 stores a computer program of an application used in information processing performed by the processor 23 and various data. The monitoring image input from the monitoring camera 2 is stored in the memory 22 in association with the monitoring image imaging time.

[0072] All or a part of the functions of the driver monitoring device 11 is, for example, a function module realized by a computer program operating on the processor 23. The processor 23 has an acquisition section 230, a determination section 231, and a control section 232. The determination section 231 is one example of the first determination section, the second determination section, the third determination section, and the fourth determination section. Alternatively, the function module that the processor 23 has can also be a dedicated arithmetic circuit provided to the processor 23. The processor 23 has one or a plurality of CPUs (Central Processing Unit) and peripheral circuits thereof. The processor 23 can also have other arithmetic circuits such as a logic arithmetic unit, a numerical value arithmetic unit, or a graphics processing unit. The driver monitoring device 11 is, for example, an electronic control device (Electronic Control Unit: ECU). In addition, the driver monitoring device 11 can also estimate the degree of participation in driving of the driver based on the monitoring image. Details of the operation of the driver monitoring device 11 will be described later.

[0073] Figure 4 is one example of an operation flowchart relating to the driver monitoring processing of the driver monitoring device 11 of the present embodiment. Next, the following describes the driver monitoring processing of the driver monitoring device 11 with reference to Figure 4 The driver monitoring processing of the monitoring system 1 will be described.

[0074] First, the acquisition section 230 acquires the number of occupants, which indicates the number of people who get on the vehicle 10 at the time when the vehicle 10 starts traveling (step S101). For example, the driver monitoring device 11 acquires the number of occupants input by the driver 30 via the UI 4. In addition, the acquisition section 230 can acquire, as the number of occupants, a number obtained by adding 1 to the number of people detected by the seat occupancy sensors 3a to 3c, in a case where it is determined based on the monitoring image captured by the monitoring camera 2 that the driver 30 is seated on the driver's seat 40. In addition, the acquisition section 230 can acquire, as the number of occupants, a number obtained by adding 1 to the number of people detected by the seat occupancy sensors 3a to 3c, after the driver 30 is notified of a notification that requires the driver 30 to be seated on the driver's seat 40. In this case, as the number of occupants, the number of cargo placed on the seat is sometimes included, but as long as the state in which the cargo is placed on the seat continues, it does not have an influence on the driver monitoring processing.

[0075] The time when the vehicle 10 starts traveling is, for example, a point in time before the vehicle 10 starts traveling based on the navigation route or a point in time when the occupant gets on the vehicle 10. In addition, the point in time at which the acquisition section 230 acquires the number of occupants can be regarded as the time when the vehicle 10 starts traveling.

[0076] Further, the determination section 231 executes the processing of steps S102 to S111 every time the monitoring image is input.

[0077] The determination section 231 inputs a monitoring image from the monitoring camera 2 (step S102). The monitoring camera 2 outputs the monitoring image and a monitoring image capturing time at which the monitoring image was captured to the driver monitoring device 11 via the in-vehicle network 12 each time the monitoring image is captured. The determination section 231 stores the monitoring image in association with the monitoring image capturing time in the memory 22.

[0078] Next, the determination section 231 determines whether or not the face of the driver 30 is detected based on the monitoring image (step S103). The determination section 231 includes an identifier that is learned so as to input a monitoring image to detect a face of a person. The identifier inputs the monitoring image, detects a face included in the monitoring image, and outputs a reliability of the detected face. The reliability is represented by a value between 0.0 and 1.0, for example. The determination section 231 determines that the face of the driver 30 is detected in a case where the reliability is equal to or higher than a predetermined reference reliability (0.7, for example). On the other hand, the determination section 231 determines that the face of the driver 30 is not detected in a case where the reliability is lower than the predetermined reference reliability.

[0079] In a case where the face of the driver 30 is detected (step S103 - Yes), the determination section 231 determines that the driver 30 is seated on the driver seat 40 (step S104). Further, the determination section 231 can determine that the driver 30 is seated on the driver seat 40 in a case where a predetermined portion of the driver 30 is detected.

[0080] Next, the determination section 231 determines whether or not the vehicle 10 has reached a destination position (step S110). In a case where the vehicle 10 has not reached the destination position, the process returns to step S102. On the other hand, in a case where the vehicle 10 has reached the destination position, a series of processes ends. For example, the determination section 231 inputs the current position of the vehicle 10 and the destination position of a navigation route from a navigation device not shown, and determines whether or not the vehicle 10 has reached the destination position.

[0081] On the other hand, in a case where the face of the driver 30 is not detected, the determination section 231 determines that the driver 30 is not seated on the driver seat 40 (step S105).

[0082] Next, the determination section 231 acquires the number of occupants (step S106). The determination section 231 acquires the number of occupants based on the occupancy signals input from the occupancy sensors 3a to 3c respectively disposed in the seats other than the driver seat 40 in the vehicle 10.

[0083] Next, the determination section 231 compares the number of occupants and the number of passengers, and determines whether or not the number of occupants and the number of passengers coincide (step S107).

[0084] In the case where the number of occupants and the number of passengers are consistent (step S107 - Yes), the determination section 231 determines that the driver 30 is not seated on the driver's seat 40 (step S108). It is considered that the driver 30 moves to a seat other than the driver's seat 40.

[0085] Next, the control section 232 notifies the driver 30 of a seating notification that requires the driver 30 to be seated on the driver's seat 40 in a manner that enables the driver 30 to be captured by the monitoring camera 2 via the UI 4 (step S109). The seating notification is displayed on the display device 4a, for example. In addition, the seating notification can be notified to the driver as a sound. In addition, the seating notification can be displayed on the display device 4a and notified to the driver as a sound.

[0086] Next, the determination section 231 determines whether the vehicle 10 has reached the destination position (step S110). In the case where the vehicle 10 has not reached the destination position, the process returns to step S102. On the other hand, in the case where the vehicle 10 has reached the destination position, the series of processes ends.

[0087] On the other hand, in the case where the number of occupants and the number of passengers are inconsistent (step S107 - No), the determination section 231 determines whether the number of occupants is one less than the number of passengers (step S111).

[0088] In the case where the number of occupants is one less than the number of passengers (step S111 - Yes), the determination section 231 determines that the driver 30 is seated on the driver's seat 40 (step S112). Since it is considered that the driver 30 is not seated on a seat other than the driver's seat based on the number of occupants, it is presumed that the driver 30 is seated on the driver's seat 40. As a reason why the driver's face is not detected from the monitoring image, it is considered that the driver's face is not directed toward the front of the vehicle 10, and thus the driver's face is not detected in the monitoring image.

[0089] Next, the determination section 231 determines whether the vehicle 10 has reached the destination position (step S110). In the case where the vehicle 10 has not reached the destination position, the process returns to step S102. On the other hand, in the case where the vehicle 10 has reached the destination position, the series of processes ends.

[0090] On the other hand, in the case where the number of occupants is not one less than the number of passengers (step S111 - No), the determination section 231 determines that the driver 30 is not seated on the driver's seat 40 (step S108). As a situation in the vehicle cabin 10A, it is considered that a cargo or the like that is not included in the number of passengers is placed on a seat other than the driver's seat.

[0091] Next, the control section 232 notifies the driver 30 of a seating notification that requires the driver 30 to be seated on the driver's seat 40 in a manner that enables the driver 30 to be captured by the monitoring camera 2 via the UI 4 (step S109).

[0092] Next, the determination unit 231 determines whether the vehicle 10 has reached the destination (step S110). If the vehicle 10 has not reached the destination, the process returns to step S102. On the other hand, if the vehicle 10 has reached the destination, the series of processes ends.

[0093] The following is for reference Figure 2 (A)~ Figure 2 (C) and Figure 5 (A) and Figure 5 (B) A specific example of driver monitoring processing of driver monitoring device 11 will be explained.

[0094] exist Figure 2 In the example shown in (A), the number of passengers is 1, and the driver 30 is seated in the driver's seat 40 facing the front of the vehicle 10. The surveillance image shows the driver 30's face facing forward relative to the surveillance camera 2. If the driver 30's face is oriented within a predetermined range relative to the optical axis of the surveillance camera 2, the determination unit 231 can detect the face contained in the surveillance image (high reliability). Based on the surveillance image, the determination unit 231 determines that the driver 30 is seated in the driver's seat 40.

[0095] exist Figure 2 In the example shown in (B), the number of passengers is 1, and the driver 30 is seated in the rear seat 42. When driving a vehicle with automatic control level 3, the driver 30 is not required to always be in a state capable of maneuvering the vehicle 10; therefore, the driver 30 may sometimes move to the more spacious rear seats 42 and 43. In the monitoring image, the driver 30's face is not represented in a detectable manner (low reliability). Based on the monitoring image, the driver monitoring device 11 determines that the driver 30 is not seated in the driver's seat 40. A seating signal is input to the determination unit 231 from the seating sensor 3b located in the rear seat 42. The number of passengers is 1. Since the number of passengers and the number of seated passengers are the same, the determination unit 231 determines that the driver 30 is not seated in the driver's seat 40.

[0096] Upon being notified to take their seat, the driver 30 moves from the rear seat 42 to the driver's seat 40, facing forward of the vehicle 10, and sits in the driver's seat 40. Based on the monitored image, the driver monitoring device 11 determines that the driver 30 is seated in the driver's seat 40. Thus, the driver monitoring device 11 can increase the degree of driver 30's involvement in driving.

[0097] exist Figure 2In the example shown in (C), the number of passengers is 1, and the driver 30 sits in the driver's seat 40, but facing the passenger seat 41, not towards the front of the vehicle 10. Additionally, the driver may sometimes recline the seat to adopt a relaxed posture. The monitoring image does not show the driver 30's face facing forward relative to the monitoring camera 2. Because the driver 30's face is facing a direction outside a predetermined range relative to the optical axis of the monitoring camera 2, the monitoring image does not show the driver 30's face in a detectable manner (low reliability). Based on the monitoring image, the determination unit 231 determines that the driver 30 is not sitting in the driver's seat 40.

[0098] Since no one is seated in any of the seats other than the driver's seat, no seating signal is input to the determination unit 231. The number of seated persons is zero, and the number of passengers is 1.1. Since the number of seated persons is one less than the number of passengers, the driver monitoring device 1 determines that the driver 30 is seated in the driver's seat 40. Furthermore, the control unit 232 notifies the driver 30 of the seating notification requesting that he / she be seated in the driver's seat 40 in a manner that can be captured by the monitoring camera 2.

[0099] Upon being notified to take their seat, the driver 30 moves towards the front of the vehicle 10 and takes their seat. Based on the monitoring image, the determination unit 231 determines that the driver 30 is seated in the driver's seat 40. As a result, the driver monitoring device 11 can increase the degree of driver 30's involvement in driving.

[0100] exist Figure 5 In the example shown in (A), the number of passengers is 1, and the driver 30 is seated in the driver's seat 40, but facing the cargo placed in the passenger seat 41, not facing the front of the vehicle 10. Since the driver 30's face is facing a direction outside the predetermined range relative to the optical axis of the monitoring camera 2, the monitoring image does not show the driver 30's face in a detectable manner (low reliability). Based on the monitoring image, the determination unit 231 determines that the driver 30 is not seated in the driver's seat 40.

[0101] Furthermore, since cargo 31 is placed in the front passenger seat 41 and cargo 32 is placed in the rear seat 43, seating signals are input to the determination unit 231 from the seating sensor 3a located in the front passenger seat 41 and the seating sensor 3c located in the rear seat 43. The number of seated passengers is 2, and the number of passengers is 1. Since the number of seated passengers is inconsistent with the number of passengers, and it is not that the number of seated passengers is one less than the number of passengers, the determination unit 231 determines that the driver 30 is not seated in the driver's seat 40. Furthermore, the control unit 232 notifies the driver 30 of a seating notification requesting that he / she sit in the driver's seat 40 in a manner that can be captured by the monitoring camera 2.

[0102] The driver 30 is notified of the seat occupancy notification, moves from the rear seat 42 to the driver's seat 40, and sits in the driver's seat 40 facing the front of the vehicle 10. Thus, the determination unit 231 determines, based on the monitoring image, that the driver 30 is seated in the driver's seat 40. Thus, the driver monitoring device 11 can improve the degree of participation of the driver 30 in driving.

[0103] In Figure 5 the example shown in (B), the number of occupants is one, and the driver 30 is seated in the rear seat 42. The face of the driver 30 is not indicated in a detectable manner in the monitoring image. The determination unit 231 determines, based on the monitoring image, that the driver 30 is not seated in the driver's seat 40.

[0104] In addition, since the driver 30 is seated in the rear seat 42 and the cargo 31 is placed in the front passenger seat 41, the determination unit 231 is inputted with a seat occupancy signal from the seat occupancy sensor 3a arranged in the front passenger seat 41 and the seat occupancy sensor 3b arranged in the rear seat 42. The number of occupants is two, and the number of occupants is one. Since the number of occupants and the number of occupants do not match, and the number of occupants is not one less than the number of occupants, the determination unit 231 determines that the driver 30 is not seated in the driver's seat 40. Also, the control unit 232 notifies the driver 30 of the seat occupancy notification requiring the driver 30 to sit in the driver's seat 40 in a manner that can be captured by the monitoring camera 2.

[0105] The driver 30 is notified of the seat occupancy notification, moves from the rear seat 42 to the driver's seat 40, and sits in the driver's seat 40 facing the front of the vehicle 10. Thus, the determination unit 231 determines, based on the monitoring image, that the driver 30 is seated in the driver's seat 40. Thus, the driver monitoring device 11 can improve the degree of participation of the driver 30 in driving.

[0106] As described above, the driver monitoring device of the present embodiment can determine whether the driver is seated in the driver's seat in a short time by immediately using the seat occupancy sensor of the seat other than the driver's seat in a case where it is determined based on the image captured in the vicinity of the driver's seat that the driver is not seated in the driver's seat.

[0107] Next, the following describes Modification Example 1 and Modification Example 2 of the driver monitoring device 11 of the above-described present embodiment with reference to the accompanying drawings. Figure 6 is an action flowchart showing Modification Example 1 related to the driver monitoring process of the driver monitoring device of the present embodiment.

[0108] In the present modification example, the seat occupancy monitoring process (Process A) shown in Figure 4 is executed between step S112 and step S110 of the action flowchart shown in Figure 6 .

[0109] First, after determining that the driver 30 is seated on the driver's seat 40 (step S112), the determination section 231 inputs the monitoring image from the monitoring camera 2 (step S201). During the period in which the sitting monitoring processing of the present modification example is executed, the processing shown in Figure 4 is not performed.

[0110] Next, the determination section 231 determines whether or not the driver 30 is seated on the driver's seat 40 on the basis of the monitoring image (step S202).

[0111] In the case where the driver 30 is seated on the driver's seat 40 (step S202 - Yes), the processing proceeds to step S110.

[0112] On the other hand, in the case where the driver 30 is not seated on the driver's seat 40 (step S202 - No), the control section 232 notifies the driver 30 of the sitting notification that requires the driver 30 to be seated on the driver's seat 40 in a manner that enables the photographing by the monitoring camera 2 via the UI 4 (step S203).

[0113] Next, the determination section 231 determines whether or not a predetermined reference time has elapsed after determining that the driver 30 is seated on the driver's seat 40 (step S112) (step S204).

[0114] In the case where the predetermined reference time has elapsed, the processing proceeds to step S110. On the other hand, in the case where the predetermined reference time has not elapsed, the processing proceeds to step S201.

[0115] As a reason why the driver 30 is not seated on the driver's seat 40 in step S105, it is considered that the face of the driver 30 is not oriented toward the front of the vehicle 10. Then, in the present modification example, in the case where the state in which the face of the driver 30 is not oriented toward the front of the vehicle 10 continues during the period of the predetermined reference time after determining that the driver 30 is seated on the driver's seat 40 (step S112), the sitting notification is notified to the driver 30. Thereby, the driver 30 is seated on the driver's seat 40 with the face oriented toward the front of the vehicle 10 by being notified of the sitting notification, and thus the driver monitoring device 11 determines that the driver 30 is seated on the driver's seat 40 on the basis of the monitoring image. Thereby, the driver monitoring device 11 can improve the degree of participation in driving by the driver 30.

[0116] Figure 7 is an action flowchart showing a modification example 2 related to the driver monitoring processing of the driver monitoring device 11 of the present embodiment.

[0117] In the present modification example, the object number determination processing (processing B) shown in Figure 4 is executed between step S104 and step S110 of the action flowchart shown in Figure 7 .

[0118] First, after determining that the driver 30 is sitting in the driver's seat 40 (step S104), the determination unit 231 obtains the number of seats (step S301).

[0119] Next, the determination unit 231 compares the number of people sitting and the number of passengers detected by the seating sensors 3a to 3c, which indicate the number of people sitting, and determines whether the number of people sitting is more than the value obtained by subtracting 1 from the number of passengers (step S302).

[0120] If the number of seated persons is greater than the value obtained by subtracting 1 from the number of passengers (step S302 - Yes), the determination unit 231 calculates the value obtained by subtracting the number of passengers from the number of seated persons and adding 1 as the number of objects placed in seats other than the driver's seat (step S303), and then causes the process to proceed to step S110.

[0121] On the other hand, if the number of seated passengers is not greater than the value obtained by subtracting 1 from the number of passengers (step S302 - No), the process proceeds to step S110.

[0122] After determining the number of objects (step S302), in the above... Figure 4 In the processing shown, if it is determined that the driver 30 is not seated in the driver's seat 40 (step S105), the determination unit 231 compares the number of seats and the number of passengers in step S107. If the value obtained by subtracting the number of objects from the number of seats matches the number of passengers, it is determined that the driver 30 is not seated in the driver's seat 40. Furthermore, after calculating the number of objects (step S302), in the aforementioned... Figure 4 In the process shown, if it is determined that the driver 30 is not sitting in the driver's seat 40 (step S105), the determination unit 231 determines that the driver 30 is sitting in the driver's seat 40 if the value obtained by subtracting the number of objects from the number of people sitting is one less than the number of passengers in step S111.

[0123] The following is for reference Figure 8 The processing of the driver monitoring device 11 in this variant example will be explained.

[0124] First of all, Figure 4 In the driver monitoring process shown, such as Figure 8 As shown in (A), the number of passengers is 1, and the driver 30 is seated in the driver's seat 40 facing the front of the vehicle 10. Cargo 31 is placed in the passenger seat 41. Based on the monitoring image, the determination unit 231 determines that the driver 30 is seated in the driver's seat 40 (step S104).

[0125] Next, in Figure 7In the object count determination process shown, a seating signal is input to the determination unit 231 from the seating sensor 3a located in the front passenger seat 41. The seating count is 1, and the number of passengers is 1. Since the number of seated passengers (1) is more than the value (0) obtained by subtracting 1 from the number of passengers (1), the determination unit 231 calculates the value (1) obtained by subtracting the number of passengers (1) from the number of seated passengers (1) and adding 1 as the number of objects placed in seats other than the driver's seat.

[0126] Next, after determining the number of objects, refer to the following... Figure 8 (B) and Figure 8 (C) of Figure 4 The driver monitoring process shown is explained below.

[0127] exist Figure 8 In the example shown in (B), the driver 30 is seated in the driver's seat 40, but facing the cargo 31 placed in the passenger seat 41, not towards the front of the vehicle 10. Based on the surveillance image, the determination unit 231 determines that the driver 30 is not seated in the driver's seat 40.

[0128] The seating sensor 3a located in the passenger seat 41 inputs a seating signal to the determination unit 231. Since the value (0) obtained by subtracting the number of objects (1) from the number of seated persons (1) is one less than the number of passengers (1), the determination unit 231 determines that the driver 30 is seated in the driver's seat 40.

[0129] exist Figure 8 In the example shown in (C), the driver 30 is seated in the rear seat 42. Based on the surveillance image, the determination unit 231 determines that the driver 30 is not seated in the driver's seat 40.

[0130] The seating sensor 3a located in the front passenger seat 41 and the seating sensor 3b located in the rear seat 42 input seating signals to the determination unit 231. Since the value (1) obtained by subtracting the number of objects (1) from the number of seated persons (2) is consistent with the number of passengers (1), the determination unit 231 determines that the driver 30 is not seated in the driver's seat 40.

[0131] exist Figure 8 (B) and Figure 8 In the example shown in (C), the control unit 232 notifies the driver 30 of a seating arrangement in the driver's seat 40 in a manner that can be captured by the surveillance camera 2.

[0132] Upon being notified to take their seat, the driver 30 moves towards the front of the vehicle 10 and sits in the driver's seat 40. Based on the monitored image, the determination unit 231 determines that the driver 30 is seated in the driver's seat 40. Therefore, the driver monitoring device 11 can increase the degree of driver 30's involvement in driving.

[0133] Sometimes, after the vehicle 10 starts running, a cargo is placed at a seat other than the driver's seat. According to the present modification, the number of objects such as the cargo placed at the seat other than the driver's seat is corrected with respect to the number of seats, and thus, even in a case where the cargo is placed at the seat other than the driver's seat, the driver can be more accurately monitored.

[0134] In the present disclosure, the driver monitoring device, the driver monitoring computer program, and the driver monitoring method of the above-described embodiments can be appropriately changed without departing from the gist of the present disclosure. In addition, the technical scope of the present disclosure is not limited to those embodiments, and pertains to the invention described in the claims and equivalents thereof.

[0135] For example, in a case where it is determined that the driver is not seated on the driver's seat, the determination unit can also use an identifier that has been learned so as to recognize a person's hand, arm, or torso to investigate whether or not the driver's body is represented in the image captured by the monitoring camera, and determine whether or not the driver is seated on the driver's seat in a state where the seat position is displaced in a reclined manner. In a case where it is determined that the driver is seated on the driver's seat in a state where the seat position is displaced in a reclined manner, the control unit can also notify the driver of the seating notification via the notification unit.

Claims

1. A driver monitoring device, characterized in that, have: The acquisition department acquires the number of passengers in the vehicle when the vehicle begins to travel; The first determination unit determines whether the driver is sitting in the driver's seat based on an image of the area near the driver's seat captured by the imaging unit. as well as The second determination unit determines that the driver is not sitting in the driver's seat when the first determination unit determines that the number of people sitting in the driver's seat is the same as the number of passengers, and determines that the driver is sitting in the driver's seat when the number of people sitting in the driver's seat is one less than the number of passengers. The number of people sitting in the driver's seat refers to the number of people sitting detected by the seating sensors of the seats other than the driver's seat respectively arranged in the vehicle.

2. The driver monitoring device according to claim 1, It also has a control unit, which, when determined by the second determination unit that the driver is not seated in the driver's seat, notifies the driver via a notification unit to sit in the driver's seat.

3. The driver monitoring device according to claim 1, It also has a third determination unit, which, when the second determination unit determines that the driver is sitting in the driver's seat, determines whether the driver is sitting in the driver's seat based on multiple images captured by the imaging unit during a predetermined time period after the determination by the second determination unit.

4. The driver monitoring device according to claim 3, It also has a control unit, which, when determined by the third determination unit that the driver is not seated in the driver's seat, notifies the driver via a notification unit to sit in the driver's seat.

5. The driver monitoring device according to claim 1, It also has a control unit, which, when the number of seated persons is inconsistent with the number of passengers and is not one less than the number of passengers, notifies the driver via a notification unit to sit in the driver's seat.

6. The driver monitoring device according to any one of claims 1 to 5, It also includes a fourth determination unit, which, when determined by the first determination unit to be that the driver is seated in the driver's seat, compares the number of seated persons with the number of passengers. If the number of seated persons is greater than the value obtained by subtracting 1 from the number of passengers, the fourth determination unit calculates the value obtained by subtracting the number of passengers from the number of seated persons and adding 1, and uses this value as the number of objects placed in seats other than the driver's seat. After the fourth determination unit calculates the number of objects, if the first determination unit determines that the driver is not sitting in the driver's seat, the second determination unit determines that the driver is not sitting in the driver's seat if the value obtained by subtracting the number of objects from the number of seated persons is the same as the number of passengers, and determines that the driver is sitting in the driver's seat if the value obtained by subtracting the number of objects from the number of seated persons is one less than the number of passengers.

7. A non-transitory storage medium storing a driver monitoring computer program readable by a computer, the driver monitoring computer program causing a processor to perform processes including: Obtain the number of passengers in the vehicle when the vehicle begins to travel; Based on images of the area near the driver's seat captured by the camera unit, it is determined whether the driver is seated in the driver's seat; and When it is determined that the driver is not seated in the driver's seat, the driver is not seated in the driver's seat if the number of seated persons is the same as the number of passengers, and the driver is seated in the driver's seat if the number of seated persons is one less than the number of passengers. The number of seated persons represents the number of people sitting as detected by the seating sensors of the seats other than the driver's seat in the vehicle.

8. A driver monitoring method, characterized in that, Driver monitoring device operation: Obtain the number of passengers in the vehicle when the vehicle begins to travel; Based on the image of the area near the driver's seat of the vehicle captured by the camera unit, it is determined whether the driver is sitting in the driver's seat; as well as When it is determined that the driver is not seated in the driver's seat, the driver is not seated in the driver's seat if the number of seated persons is the same as the number of passengers, and the driver is seated in the driver's seat if the number of seated persons is one less than the number of passengers. The number of seated persons represents the number of people sitting as detected by the seating sensors of the seats other than the driver's seat in the vehicle.

Images