Determination device

Abstract

To provide a determination device capable of accurately determining whether a driver is making conversation or the driver has a smiling face.SOLUTION: A determination device comprises: a first determination part for determining whether or not a driver is making conversation using a conversation-determining reference, or determining whether or not the driver has a smiling face using a smiling face-determining reference; and a change part for changing the conversation-determining reference in such a manner that it is easily determined that the driver is making conversation, or changing the smiling face-determining reference in such a manner that it is easily determined that the driver has a smiling face, when an occupant is detected by a detector for detecting the occupant riding in a vehicle, according to a determination content of the first determination part.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present disclosure relates to a determination device.

Background Art

[0002] Conventionally, a vehicle monitoring device has been determining and monitoring the state of a driver. For example, the monitoring device determines whether the driver is dozing off.

[0003] The monitoring device examines the eye-opening degree based on an image of the driver's face captured using a monitoring camera, and determines whether the driver is dozing off or not. When a state with a low eye-opening degree (closed-eye state) is continuously detected for a predetermined time, the monitoring device determines that the driver is in a dozing state and issues a warning.

[0004] In addition, a driver may drive a vehicle while chatting with a passenger who is riding in the vehicle together with the driver. And a technique for detecting a passenger riding in the vehicle has been proposed (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] Even when the driver is talking or has a smiling face, a state with a low eye-opening degree may be continuously detected for a predetermined time. Thus, there has been a problem that a warning that the driver is dozing off by mistake is notified to a driver who is not dozing off.

[0007] Therefore, it is considered that by correctly determining whether the driver is talking or has a smiling face, for example, it is possible to prevent the driver from being erroneously determined to be dozing off.

[0008] Therefore, an object of the present disclosure is to provide a determination device that can accurately determine whether a driver is talking or has a smiling face.

Means for Solving the Problems

[0009] (1) According to one embodiment, a determination device is provided. This determination device includes a first determination unit that determines whether the driver is talking using a conversation determination criterion or determines whether the driver has a smiling face using a smiling face determination criterion, and when a passenger riding in the vehicle is detected by a detection unit that detects the passenger, according to the determination content of the first determination unit, a change unit that changes the conversation determination criterion so that it is easier to determine that the driver is talking or changes the smiling face determination criterion so that it is easier to determine that the driver has a smiling face. It is characterized by having.

[0010] (2) In the determination device of (1), it has a second determination unit that determines whether the passenger is talking. When the second determination unit determines that the passenger is talking, the change unit changes the conversation determination criterion so that it is easier to further determine that the driver is talking, and changes the smiling face determination criterion so that it is easier to further determine that the driver has a smiling face. This is preferable.

[0011] (3) In the determination device of (1) or (2), it has a third determination unit that determines whether the passenger has a smiling face. When the third determination unit determines that the passenger has a smiling face, the change unit changes the conversation determination criterion so that it is easier to further determine that the driver is talking, and changes the smiling face determination criterion so that it is easier to further determine that the driver has a smiling face. This is preferable.

[0012] (4) In any of the determination devices according to (1) to (3), there is a fourth determination unit that determines whether or not the passenger is asleep. When the fourth determination unit determines that the passenger is asleep, it is preferable that the change unit changes the conversation determination criterion so that it is difficult to determine that the driver is having a conversation, and changes the smiling face determination criterion so that it is difficult to determine that the driver has a smiling face.

[0013] (5) In any of the determination devices according to (1) to (4), there is a fifth determination unit that determines whether or not the driver is dozing off. It is preferable that the fifth determination unit does not determine whether or not the driver is dozing off when the driver is having a conversation or has a smiling face.

Advantages of the Invention

[0014] The determination device according to the present disclosure can accurately determine whether the driver is having a conversation or has a smiling face.

Brief Description of the Drawings

[0015]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Modes for Carrying Out the Invention

[0016] FIG. 1(A) and FIG. 1(B) illustrate an overview of the operation of the monitoring device according to the first embodiment. FIG. 1(A) is a diagram showing a driver who is driving, and FIG. 1(B) is a diagram (part 1) showing the interior of the vehicle cabin.

[0017] As shown in FIG. 1(A), the vehicle 10 has a monitoring device 11. Inside the vehicle cabin 10A of the vehicle 10, a monitoring camera 2a capable of acquiring an image representing the face of the driver 30 seated in the driver's seat 40 is disposed above the steering column 50. The monitoring device 11 is an example of a determination device. The vehicle 10 may be an autonomous vehicle.

[0018] As shown in FIG. 1(B), inside the vehicle cabin 10A, a driver's seat 40 on which the driver 30 is seated, a passenger seat 41, and rear seats 42 and 43 are arranged. Occupancy sensors 3a to 3c are respectively arranged on each of the seats 41 to 43 other than the driver's seat. The occupancy sensors 3a to 3c detect occupancy other than the driver 30 and output an occupancy signal indicating that a passenger is seated to the monitoring device 11. The monitoring device 11 can determine the presence or absence of a passenger based on the occupancy signals input from the occupancy sensors 3a to 3c.

[0019] The monitoring device 11 detects the presence or absence of a passenger when the ignition is on and / or at a predetermined cycle. In the example shown in FIG. 1(B), a passenger is seated in the passenger seat 41.

[0020] The monitoring device 11 determines whether the driver 30 is talking using a predetermined conversation determination criterion based on the monitoring image acquired by the monitoring camera 2a. Further, the monitoring device 11 determines whether the driver 30 has a smiling face using a predetermined smiling face determination criterion based on this monitoring image.

[0021] The monitoring device 11 can, for example, also monitor whether the driver 30 is dozing off. Based on the monitoring image, the monitoring device 11 obtains the eye-opening degree of the driver 30. When a state with a low eye-opening degree (closed eyes state) is continuously detected for a predetermined time, it is determined that the driver is dozing off and a warning is notified.

[0022] However, even when the driver 30 is talking or has a smiling face, a state with a low eye-opening degree may be continuously detected for a predetermined time. To prevent misjudging that the driver is dozing off, it is desirable to be able to correctly determine whether the driver is talking or has a smiling face.

[0023] Here, when there is a passenger riding in the vehicle 10 together with the driver 30, the driver 30 often drives the vehicle 10 while chatting with the passenger.

[0024] By using the information that there is a passenger riding in the vehicle 10 for conversation determination and smiling face determination, it is considered that the accuracy of determining whether the driver 30 is talking or whether the driver 30 has a smiling face can be improved.

[0025] Therefore, since the monitoring device 11 detects a passenger sitting in the passenger seat 41 by the seating sensor 3a that detects passengers riding in the vehicle 10, the conversation determination criterion is changed so that it is easier to determine that the driver 30 is talking, and the smiling face determination criterion is changed so that it is easier to determine that the driver 30 has a smiling face.

[0026] In the monitoring device 11, when there is a passenger, it is easier to determine that the driver 30 is talking and it is easier to determine that the driver 30 has a smiling face than when there is no passenger.

[0027] The monitoring device 11 determines whether the driver 30 is talking by using the changed conversation determination criterion, or / and determines whether the driver 30 has a smiling face by using the changed smiling face determination criterion.

[0028] As a result, it is possible to correctly determine that the driver 30 is in a state where the driver 30 is actually having a conversation, which was erroneously determined not to be the case before. Also, it is possible to correctly determine that the driver 30 has a smiling face, which was erroneously determined not to be the case before.

[0029] As described above, the monitoring device 11 of the present embodiment can accurately determine whether the driver 30 is having a conversation or whether the driver 30 has a smiling face.

[0030] FIG. 2 is a schematic configuration diagram of a vehicle 10 in which a monitoring system 1 including the monitoring device 11 of the present embodiment is implemented. The monitoring system 1 includes monitoring cameras 2a, 2b, 2c, 2d, seating sensors 3a, 3b, 3c, a user interface (UI) 4, the monitoring device 11, and the like.

[0031] The monitoring cameras 2a, 2b, 2c, 2d, the seating sensors 3a, 3b, 3c, the UI 4, and the monitoring device 11 are communicably connected via an in-vehicle network 12 compliant with a standard such as a controller area network.

[0032] The monitoring camera 2a is disposed in the passenger compartment 10A so as to be able to acquire a monitoring image including the face of the driver 30 driving the vehicle 10. The monitoring camera 2a is an example of an image acquisition unit. The monitoring image represents the situation in the vicinity of the driver's seat. As shown in FIG. 1(A), the monitoring camera 2a is disposed, for example, on the steering column 50. Note that the monitoring camera 2a may be disposed on a steering wheel, a rearview mirror, a meter panel, a meter hood, or the like in the passenger compartment 10A.

[0033] The monitoring camera 2b is arranged in the vehicle compartment 10A so as to be able to acquire a monitoring image including the face of the passenger sitting in the passenger seat 41. The monitoring camera 2b is arranged, for example, on the dashboard in front of the passenger seat 41 so as to be able to acquire the vicinity of the passenger seat 41. The monitoring camera 2c is arranged in the vehicle compartment 10A so as to be able to acquire a monitoring image including the face of the passenger sitting in the rear seat 42. The monitoring camera 2c is arranged, for example, at a portion facing the rear seat 42 of the driver's seat 40 so as to be able to acquire the vicinity of the rear seat 42. The monitoring camera 2d is arranged in the vehicle compartment 10A so as to be able to acquire a monitoring image including the face of the passenger sitting in the rear seat 43. The monitoring camera 2d is arranged, for example, at a portion facing the rear seat 43 of the passenger seat 41 so as to be able to acquire the vicinity of the rear seat 43.

[0034] Each of the monitoring cameras 2a to 2d acquires a monitoring image, for example, at a monitoring image acquisition time having a predetermined period. Each of the monitoring cameras 2a to 2d outputs the monitoring image and the monitoring image acquisition time to the monitoring device 11 via the in-vehicle network 12 each time a monitoring image is acquired.

[0035] The monitoring cameras 2a to 2d each have a two-dimensional detector composed of an array of photoelectric conversion elements sensitive to infrared rays, such as a CCD or a C-MOS, and an imaging optical system that forms an image of the area to be imaged on the two-dimensional detector. As the predetermined period, for example, it can be 0.1 to 0.5 seconds. The monitoring camera 2a preferably has a projector together with the two-dimensional detector. The projector is an LED (light emitting diode), for example, two near-infrared LEDs arranged on both sides of the imaging optical system. By irradiating the driver with near-infrared light, the face of the driver can be imaged without giving discomfort to the driver even in low illumination conditions such as at night.

[0036] The seating sensors 3a, 3b, and 3c are respectively arranged on each of the seats in the vehicle 10 other than the driver's seat 40. The seating sensors 3a, 3b, and 3c are an example of a detection unit that detects passengers riding in the vehicle 10. The seating sensors 3a to 3c detect the seating of passengers other than the driver 30 and output a seating signal indicating that they are seated to the monitoring device 11. While the seating sensors 3a to 3c are detecting the seating of a passenger, it is preferable that they continue to output the seating signal to the monitoring device 11. As the seating sensors 3a to 3c, for example, pressure sensors can be used. When the pressure sensor detects a pressure equal to or higher than a predetermined reference pressure, it outputs a seating signal, and when it detects a pressure lower than the reference pressure, it does not output a seating signal. Other sensors may be used as the sensors for detecting passengers. For example, a buckle sensor (not shown) that detects that a passenger is wearing a seatbelt may be used to detect the passenger.

[0037] UI4 is an example of a notification unit. UI4 is controlled by the monitoring device 11 and notifies the driver of a participation request or the like that requests an increase in the degree of involvement in the operation of the vehicle 10. UI4 has a display device 4a such as a liquid crystal display or a touch panel in order to display the participation request or the like. Further, UI4 may have an acoustic output device (not shown) for notifying the driver of the participation request or the like. Further, UI4 has, as an input device for inputting operation information from the driver to the vehicle 10, for example, a touch panel or operation buttons. UI4 outputs the input operation information to the monitoring device 11 via the in-vehicle network 12.

[0038] The monitoring device 11 executes a determination process and a change process. For this purpose, the monitoring device 11 has a communication interface (IF) 21, a memory 22, and a processor 23. The communication interface 21, the memory 22, and the processor 23 are connected via a signal line 24. The communication interface 21 has an interface circuit for connecting the monitoring device 11 to the in-vehicle network 12.

[0039] The memory 22 is an example of a storage unit and includes, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. The memory 22 stores a computer program of an application and various data used in information processing executed by the processor 23. In the memory 22, surveillance images input from the surveillance cameras 2a to 2d are stored in association with the surveillance image acquisition times.

[0040] All or part of the functions of the surveillance device 11 are, for example, functional modules realized by a computer program operating on the processor 23. The processor 23 includes a determination unit 231 and a modification unit 232. The determination unit 231 is an example of a first determination unit, a second determination unit, a third determination unit, and a fifth determination unit. Alternatively, the functional module of the processor 23 may be a dedicated arithmetic circuit provided in the processor 23. The processor 23 includes one or more CPUs (Central Processing Units) and their peripheral circuits. The processor 23 may further include other arithmetic circuits such as a logical arithmetic unit, a numerical arithmetic unit, or a graphic processing unit. The surveillance device 11 is, for example, an electronic control unit (ECU).

[0041] Based on a surveillance image including the most recent series of driver seats 40, the determination unit 231 determines whether the driver 30 is talking. The determination unit 231 detects the opening degree of the driver 30 based on the surveillance image acquired by the surveillance camera 2a, and determines whether the driver 30 is talking based on the opening degree. For example, the determination unit 231 detects feature points representing the upper lip and the lower lip of the mouth from the surveillance image. Then, the determination unit 231 calculates the distance between the feature points of the upper lip and the lower lip as the opening degree. As the most recent series of surveillance images, for example, images for the most recent 10 seconds can be used.

[0042] The determination unit 231 examines the change in the opening degree of the driver 30, and if the frequency of changes indicating a change equal to or greater than the reference change amount is equal to or greater than the reference frequency, it determines that the driver 30 is engaged in conversation. The reference change amount and the reference frequency are determined in advance using a series of time-series face images or the like when an unspecified number of people or the driver are engaged in conversation. The reference change amount and the reference frequency are an example of conversation determination criteria.

[0043] Also, the determination unit 231 uses an identifier learned to identify whether a human smiling face is represented in the image by machine learning to determine whether the driver has a smiling face. The determination unit 231 inputs the monitoring image acquired by the monitoring camera 2a into the identifier and obtains the reliability indicating that a human smiling face is represented in the image output from the identifier. The determination unit 231 determines that the driver has a smiling face when the reliability is equal to or greater than the reference reliability. The reference reliability is an example of smiling face determination criteria.

[0044] FIG. 3 is an example of an operation flowchart regarding the driver monitoring process of the monitoring device 11 of the present embodiment. Next, the driver monitoring process of the monitoring device 11 will be described below with reference to FIG. 3.

[0045] First, the determination unit 231 determines whether a passenger has been detected by the seating sensors 3a, 3b, 3c that detect passengers boarding the vehicle 10 (step S101). The determination unit 231 determines that a passenger has been detected when a seating signal is input from any one of the seating sensors 3a, 3b, 3c. On the other hand, the determination unit 231 determines that no passenger has been detected when no seating signal is input from any of the seating sensors 3a, 3b, 3c.

[0046] If no passenger is detected (step S101 - No), the process proceeds to step S103. On the other hand, if a passenger is detected (step S101 - Yes), the change unit 232 changes the conversation determination criteria so that it is easier to determine that the driver 30 is engaged in conversation, and changes the smiling face determination criteria so that it is easier to determine that the driver 30 has a smiling face (step S102).

[0047] When a passenger is detected, the change unit 232 reduces the reference change amount and / or reduces the reference frequency for the conversation determination criteria. For example, the change unit 232 reduces the reference change amount to 80% of the initial value. Also, the change unit 232 reduces the reference frequency to 80% of the initial value. Further, when a passenger is detected, the change unit 232 reduces the reference reliability for the smiling face determination criteria. For example, the change unit 232 reduces the reference reliability to 80% of the initial value.

[0048] Next, the determination unit 231 determines whether the driver 30 is talking using the conversation determination criteria (step S103). If the conversation determination criteria have been changed in step S102, the determination unit 231 determines whether the driver 30 is talking using the conversation determination criteria changed by the change unit 232.

[0049] If it is determined that the driver 30 is not talking (step S103 - No), the determination unit 231 determines whether the driver 30 has a smiling face using the smiling face determination criteria (step S104), and ends the series of processes. If the smiling face determination criteria have been changed in step S102, the determination unit 231 determines whether the driver 30 has a smiling face using the smiling face determination criteria changed by the change unit 232.

[0050] When there is a passenger, the determination unit 231 makes it easier to determine that the driver 30 is talking and makes it easier to determine that the driver 30 has a smiling face than when there is no passenger.

[0051] Thereby, it is possible to correctly determine that the driver 30 is talking in a state where it was erroneously determined that the driver 30 was not talking. Also, it is possible to correctly determine that the driver 30 has a smiling face in a state where it was erroneously determined that the driver 30 did not have a smiling face.

[0052] On the other hand, when it is determined that the driver 30 is talking (step S103 - Yes), a series of processes is terminated.

[0053] According to the monitoring device of the present embodiment described above, it is possible to accurately determine whether the driver is talking or whether the driver has a smiling face.

[0054] Next, a modified example of the monitoring device of the present embodiment described above will be described below with reference to FIGS. 4 and 5. FIG. 4 is an example of an operation flowchart regarding the reference change process of the monitoring device of the present embodiment. The determination unit 231 may perform the reference change process shown in FIG. 4 between step S102 and step S103 in the driver monitoring process described above.

[0055] In the reference change process of this modified example, first, the determination unit 231 determines whether a passenger is talking. The determination unit 231 determines whether a passenger sitting in the passenger seat 41 is talking based on a series of latest monitoring images including the passenger seat 41 acquired by the monitoring camera 2b (step S201). Also, the determination unit 231 determines whether a passenger sitting in the rear seat 42 is talking based on a series of latest monitoring images including the rear seat 42 acquired by the monitoring camera 2c. Further, the determination unit 231 determines whether a passenger sitting in the rear seat 43 is talking based on a series of latest monitoring images including the rear seat 43 acquired by the monitoring camera 2d. The monitoring images acquired by the monitoring cameras 2b, 2c, and 2d are an example of first information representing the state of the passengers.

[0056] Note that the determination unit 231 may determine whether a passenger sitting in a seat where a passenger is detected by the seating sensors 3a, 3b, and 3c is talking.

[0057] When any of the passengers is talking, the determination unit 231 determines that the passengers are talking. On the other hand, when none of the passengers is talking, the determination unit 231 determines that the passengers are not talking.

[0058] When it is determined that the passenger is talking (step S201 - Yes), the change unit 232 changes the conversation determination criteria so that it becomes easier to further determine that the driver is talking, and changes the smiling face determination criteria so that it becomes easier to further determine that the driver has a smiling face (step S203), and ends a series of processes.

[0059] The change unit 232 further reduces the criterion change amount and / or further reduces the criterion frequency for the conversation determination criteria. For example, the change unit 232 reduces the criterion change amount to 60% of the initial value. Also, the change unit 232 reduces the criterion frequency to 60% of the initial value. Further, when a passenger is detected, the change unit 232 reduces the criterion reliability for the smiling face determination criteria. For example, the change unit 232 reduces the criterion reliability to 60% of the initial value.

[0060] When the passenger is talking, it is presumed that the driver 30 is also talking or the driver 30 has a smiling face. Therefore, in this modification, when the passenger is talking, the conversation determination criteria are changed so that it becomes easier to further determine that the driver 30 is talking, and the smiling face determination criteria are changed so that it becomes easier to further determine that the driver 30 has a smiling face.

[0061] On the other hand, when it is determined that the passenger is not talking (step S202 - No), the determination unit 231 determines whether the passenger has a smiling face (step S202). The determination unit 231 inputs each of the monitoring images acquired by the monitoring cameras 2b, 2c, 2d to the discriminator, and determines whether each of the passengers has a smiling face.

[0062] Note that the determination unit 231 may determine whether a passenger sitting in the seat detected by the seating sensors 3a, 3b, 3c has a smiling face.

[0063] When any of the passengers has a smiling face, the determination unit 231 determines that the passenger has a smiling face. On the other hand, when none of the passengers has a smiling face, the determination unit 231 determines that the passenger does not have a smiling face.

[0064] When it is determined that the passenger has a smiling face (step S202 - Yes), the change unit 232 changes the conversation determination criterion so that it is more likely to be further determined that the driver is having a conversation, and changes the smiling face determination criterion so that it is more likely to be further determined that the driver has a smiling face (step S203), and ends a series of processes.

[0065] When a passenger has a smiling face, it is presumed that the driver 30 also has a smiling face or the driver 30 is having a conversation. Therefore, in this modification, when a passenger has a smiling face, the conversation determination criterion is changed so that it is more likely to be further determined that the driver 30 has a smiling face, and the smiling face determination criterion is changed so that it is more likely to be further determined that the driver 30 is having a conversation.

[0066] On the other hand, when the passenger does not have a smiling face (step S202 - No), a series of processes are ended.

[0067] According to the monitoring device 11 of this modification described above, it is possible to more accurately determine whether the driver is having a conversation or the driver has a smiling face.

[0068] FIG. 5 is another example of an operation flowchart regarding the reference change process of the monitoring device of the present embodiment. The determination unit 231 may perform the reference change process shown in FIG. 5 between step S102 and step S103 in the driver monitoring process described above.

[0069] In the reference change process of this modification example, first, the determination unit 231 determines whether the passenger is sleeping (step S301). The determination unit 231 obtains the eye-opening degree of the passenger based on each of the monitoring images acquired by the monitoring cameras 2b, 2c, and 2d. The determination unit 231 determines that the passenger is sleeping when the eye-opening degree of the passenger is less than or equal to the reference eye-opening degree and exceeds the first reference time, or when the total time in a state where the eye-opening degree of the passenger within a predetermined determination period is less than or equal to the reference eye-opening degree exceeds the second reference time.

[0070] Note that the determination unit 231 may determine whether a passenger sitting on the seat detected by the seating sensors 3a, 3b, and 3c is sleeping.

[0071] When all passengers are sleeping, the determination unit 231 determines that the passengers are sleeping. On the other hand, when any passenger is not sleeping, the determination unit 231 determines that the passengers are not sleeping.

[0072] When the passenger is sleeping (step S301 - Yes), the change unit 232 changes the conversation determination criterion so that it is difficult to determine that the driver is having a conversation, and changes the smiling face determination criterion so that it is difficult to determine that the driver has a smiling face (step S302), and ends a series of processes.

[0073] The change unit 232 increases the reference change amount and / or the reference frequency for the conversation determination criterion to be greater than the current value. The change unit 232 increases the reference change amount to 1.1 to 1.3 times the current value. Also, the change unit 232 increases the reference frequency to 1.1 to 1.3 times the current value. Further, when a passenger is detected, the change unit 232 reduces the reference reliability for the smiling face determination criterion. For example, the change unit 232 increases the reference reliability to 1.1 to 1.3 times the current value.

[0074] Note that it is preferable that the conversation determination criteria be set such that it is easier to determine that the driver is engaged in conversation than the initial value. Also, it is preferable that the smiling face determination criteria be set such that it is easier to determine that the driver has a smiling face than the initial value.

[0075] On the other hand, if the passenger is not sleeping (step S301 - Yes), the series of processes is terminated.

[0076] If the passenger is sleeping, it is presumed that the driver 30 is also sleepy. If the passenger is sleeping and the driver 30 is sleepy, it is presumed that the driver 30 is not engaged in conversation or does not have a smiling face. Therefore, in this modified example, when the passenger is sleeping, the conversation determination criteria are changed so that it is difficult to determine that the driver 30 is engaged in conversation, and the smiling face determination criteria are changed so that it is difficult to determine that the driver 30 has a smiling face.

[0077] As a result, it is possible to correctly determine that the driver 30 is not engaged in conversation, instead of erroneously determining that the driver 30 is engaged in conversation. Also, it is possible to correctly determine that the driver 30 does not have a smiling face, instead of erroneously determining that the driver 30 has a smiling face.

[0078] According to the monitoring device 11 of the present modified example described above, it is possible to more accurately determine whether the driver is engaged in conversation or whether the driver has a smiling face.

[0079] Next, the monitoring device of the second embodiment disclosed in this specification will be described below with reference to FIG. 6. Regarding the monitoring device of this embodiment, the description of the monitoring device of the first embodiment described above is appropriately applied to points not particularly described.

[0080] FIG. 6 is another example of an operation flowchart regarding the driver monitoring process of the monitoring device 11 according to the second embodiment. The processes in steps S401, 402, 403, and 404 shown in FIG. 6 are the same as the processes in steps S101, 102, 103, and 104 in FIG. 3 described above.

[0081] First, the determination unit 231 determines whether a passenger has been detected by the seating sensors 3a, 3b, and 3c that detect passengers boarding the vehicle 10 (step S401).

[0082] When a passenger is detected (step S401 - Yes), the change unit 232 changes the conversation determination criterion so that it is easier to determine that the driver 30 is having a conversation, and changes the smiling face determination criterion so that it is easier to determine that the driver 30 has a smiling face (step S402).

[0083] Next, the determination unit 231 determines whether the driver 30 is having a conversation using the conversation determination criterion (step S403).

[0084] When it is determined that the driver 30 is not having a conversation (step S403 - No), the determination unit 231 determines whether the driver 30 has a smiling face using the smiling face determination criterion (step S404).

[0085] When it is determined that the driver 30 is having a conversation (step S403 - Yes), or when it is determined that the driver 30 has a smiling face (step S404 - Yes), a series of processes ends.

[0086] On the other hand, when no passenger is detected (step S401 - No), the determination unit 231 determines whether the conversation determination criterion or the smiling face determination criterion has been changed (step S405). The determination unit 231 determines that the conversation determination criterion has been changed when the conversation determination criterion has been changed from the initial value. Also, the determination unit 231 determines that the smiling face determination criterion has been changed when the smiling face determination criterion has been changed from the initial value.

[0087] When the conversation determination criterion or the smiling face determination criterion has been changed (step S405 - Yes), the changing unit 232 returns the conversation determination criterion or / and the smiling face determination criterion to the initial value, and the process proceeds to step S403. Since the passenger may get off from the vehicle 10, even if a passenger is once determined to be present, the passenger may not be present thereafter. If there is no passenger, when monitoring the state of the driver 30 using the changed conversation determination criterion or the smiling face determination criterion, there is a possibility that the correct state of the driver 30 cannot be monitored. Therefore, the changing unit 232 returns the changed conversation determination criterion or the smiling face determination criterion to the initial value. Thereby, the monitoring device 11 can accurately determine whether the driver is having a conversation or the driver has a smiling face.

[0088] Also, when the driver 30 does not have a smiling face (step S404 - No), the determination unit 231 determines whether the driver 30 is sleeping or not using the monitoring image acquired by the monitoring camera 2a (step S407). The monitoring image acquired by the monitoring camera 2a is an example of the second information representing the state of the driver.

[0089] When the driver 30 is sleeping (step S407 - Yes), the determination unit 231 notifies a warning to wake up the driver 30 using the UI4 (step S408), and ends a series of processes. For example, the determination unit 231 may display a blinking display on the display device 4a or output a loud warning sound using the UI4.

[0090] On the other hand, when the driver 30 is not sleeping (step S407 - No), the determination unit 231 determines whether the driver 30 has drowsiness (step S409). Although the driver 30 is not sleeping, the driver 30 may have drowsiness. For example, the determination unit 231 obtains the opening degree of the driver 30 using the monitoring image acquired by the monitoring camera 2a.

[0091] When the driver's 30 eye opening degree is less than or equal to the reference eye opening degree and exceeds a third reference time shorter than the first reference time, or when the total time in a state where the driver's 30 eye opening degree is less than or equal to the reference eye opening degree within a predetermined determination period exceeds a fourth reference time shorter than the second reference time, the determination unit 231 determines that the driver 30 is sleepy.

[0092] When the driver 30 is sleepy (step S409 - Yes), the determination unit 231 notifies a participation request for increasing the degree of participation in the driving of the vehicle 10 (step S410) and ends a series of processes. For example, the determination unit 231 may display on the display device 4a to concentrate on driving, or output a loud warning sound using the UI4.

[0093] According to the monitoring device of the present embodiment described above, when the driver is talking or smiling, it is not determined that the driver is dozing off. Therefore, it is possible to prevent the state where the driver is talking or smiling from being erroneously determined as the driver being dozing off, and it is possible to correctly determine that the driver is dozing off.

[0094] In the present disclosure, the determination device of the above-described embodiment can be appropriately changed without departing from the gist of the present disclosure. Further, the technical scope of the present disclosure is not limited to those embodiments, but extends to the invention described in the claims and its equivalents.

[0095] For example, when a fellow passenger riding in the vehicle is detected by a detection unit that detects the fellow passengers according to the determination content of the determination unit, if the change unit changes the conversation determination criterion so that it becomes easier to determine that the driver is having a conversation, or changes the smiling face determination criterion so that it becomes easier to determine that the driver has a smiling face. When the determination unit only determines whether the driver is having a conversation using the conversation determination criterion, the change unit may only change the conversation determination criterion. Further, when the determination unit only determines whether the driver has a smiling face using the smiling face determination criterion, the change unit may only change the smiling face determination criterion. Furthermore, when the determination unit determines whether the driver is having a conversation using the conversation determination criterion and determines whether the driver has a smiling face using the smiling face determination criterion, the change unit changes the conversation determination criterion and changes the smiling face determination criterion.

[0096] In the above-described embodiment, when the driver is having a conversation, whether the driver has a smiling face is not determined, but even when the driver is having a conversation, it may be determined whether the driver has a smiling face.

[0097] Also, the above-described method for determining the conversation and smiling face of the driver or fellow passenger is an example, and other methods may be used. Also, the above-described method for determining that the driver or fellow passenger is sleeping or sleepy is an example, and other methods may be used. Further, the present invention can be used without particular limitation in the process of determining the state of the driver based on the degree of eye opening of the driver, other than the drowsiness determination of the driver.

Explanation of Signs

[0098] 1 Monitoring system 2a, 2b, 2c, 2d Monitoring cameras 3a, 3b, 3c Seating sensors 4 User interface 4a Display device 10 Vehicle 11 Monitoring device 21 Communication interface 22 Memory 23 Processor 231 Determination Unit 232 Modification Unit 12 In-vehicle Network

Claims

1. A first determination unit that determines whether a driver is engaged in conversation using conversation determination criteria or determines whether a driver has a smiling face using smiling face determination criteria, and a modification unit that, when a passenger riding in the vehicle is detected by a detection unit that detects the passenger according to the determination content of the first determination unit, modifies the conversation determination criteria so that it is easier to determine that the driver is engaged in conversation, or modifies the smiling face determination criteria so that it is easier to determine that the driver has a smiling face. A determination device, characterized by comprising the above.

2. having a second determination unit that determines whether a passenger is engaged in conversation, and when the second determination unit determines that the passenger is engaged in conversation, the modification unit modifies the conversation determination criteria so that it is even easier to determine that the driver is engaged in conversation, and modifies the smiling face determination criteria so that it is even easier to determine that the driver has a smiling face. The determination device according to claim 1.

3. having a third determination unit that determines whether a passenger has a smiling face, and when the third determination unit determines that the passenger has a smiling face, the modification unit modifies the conversation determination criteria so that it is even easier to determine that the driver is engaged in conversation, and modifies the smiling face determination criteria so that it is even easier to determine that the driver has a smiling face. The determination device according to claim 1.

4. having a fourth determination unit that determines whether a passenger is asleep, and when the fourth determination unit determines that the passenger is asleep, the modification unit modifies the conversation determination criteria so that it is more difficult to determine that the driver is engaged in conversation, and modifies the smiling face determination criteria so that it is more difficult to determine that the driver has a smiling face. The determination device according to claim 1.

5. having a fifth determination unit that determines whether a driver is dozing off, and the fifth determination unit does not determine whether the driver is dozing off when the driver is engaged in conversation or when the driver has a smiling face. The determination device according to any one of claims 1 to 4.

Images