Monitoring device, storage medium, and monitoring method

Abstract

The present application provides a kind of monitoring device, storage medium and monitoring method, according to the surrounding condition of vehicle, the reference of monitoring driver changes, can properly monitor driver.Monitoring device detects the surrounding object of specified type existing in the specified area in the surrounding of vehicle based on environment information indicating the environment of vehicle surrounding, determines the degree of surrounding monitoring of driver of vehicle based on information indicating the state of driver, in the case where the number of surrounding object detected in the specified area is less than object quantity threshold value, the inattention threshold value that becomes the standard of the degree of surrounding monitoring of driver is set to the first value, in the case where the number of surrounding object is more than or equal to object quantity threshold value, the inattention threshold value is set to the second value smaller than the first value, determine whether the degree of surrounding monitoring of driver exceeds inattention threshold value, in the case where it is determined that the degree of surrounding monitoring of driver exceeds inattention threshold value, the driver is notified of warning via notification part, to pay attention to the surrounding of vehicle.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a monitoring device, a storage medium storing a computer program for monitoring, and a monitoring method. BACKGROUND

[0002] Conventionally, a monitoring device mounted on a vehicle monitors a state of a driver and monitors whether or not the driver pays attention to a periphery of the vehicle. For example, a monitoring device mounted on a vehicle monitors whether or not a driver gazes at a front of the vehicle, and notifies the driver of a warning in a case where the driver looks aside.

[0003] For example, Patent Literature 1 proposes a vehicle-mounted system in which a degree of improper driving such as looking aside while driving (improper driving level) is measured, and a warning is given when the improper driving level reaches a coping level. In the vehicle-mounted system, the coping level is lowered at a travel site where improper driving is likely to occur, as compared with a travel site where improper driving is not likely to occur. Thus, in the vehicle-mounted system, improper driving is prevented.

[0004] PRIOR ART DOCUMENTS

[0005] PATENT LITERATURE

[0006] Patent Literature 1: Japanese Patent Application Publication No. 2013-65246 SUMMARY

[0007] PROBLEMS TO BE SOLVED BY THE INVENTION

[0008] As described above, it is proposed that a criterion for monitoring a driver is changed according to a travel site where improper driving is likely to occur. In addition, a moving object such as a vehicle or a pedestrian is likely to approach the vehicle. Furthermore, the number of the moving objects is likely to dynamically change. Therefore, it can be considered that it is preferable that the criterion for monitoring the driver is changed according to a periphery condition of the vehicle.

[0009] An object of the present disclosure is to provide a monitoring device capable of appropriately monitoring a driver by changing a criterion for monitoring the driver according to a periphery condition of a vehicle.

[0010] MEANS FOR SOLVING THE PROBLEMS

[0011] According to an embodiment, a monitoring device is provided. The monitoring device has: a detection section that detects a surrounding object of a prescribed kind present within a prescribed area of a periphery of a vehicle based on environmental information representing an environment of the periphery of the vehicle; a decision section that decides a degree of periphery monitoring by a driver of the vehicle based on information representing a state of the driver; a setting section that sets an inattentiveness threshold value that is a criterion of the degree of periphery monitoring by the driver in a case where a number of surrounding objects detected by the detection section within the prescribed area is less than an object number threshold value, and sets the inattentiveness threshold value to a first value in a case where the number of surrounding objects is equal to or more than the object number threshold value; a determination section that determines whether or not the degree of periphery monitoring by the driver decided by the decision section exceeds the inattentiveness threshold value; and a warning section that notifies the driver of a warning via a notification section in a case where it is determined by the determination section that the degree of periphery monitoring by the driver exceeds the inattentiveness threshold value, so as to cause the driver to pay attention to the periphery of the vehicle.

[0012] Further, in the monitoring device, preferably, the decision section decides the degree of periphery monitoring by the driver of the vehicle so that the greater the degree to which the driver pays attention to the periphery of the vehicle, the greater a value of the degree of periphery monitoring by the driver of the vehicle has, and the warning section notifies the driver of a warning in a case where it is determined by the determination section that the degree of periphery monitoring by the driver is less than the inattentiveness threshold value.

[0013] Further, in the monitoring device, preferably, the decision section decides the degree of periphery monitoring by the driver based on a duration of inattentiveness of the driver.

[0014] Further, in the monitoring device, preferably, the decision section decides the degree of periphery monitoring by the driver based on a number of times of inattentiveness of the driver within a prescribed time.

[0015] Further, in the monitoring device, preferably, the decision section decides the degree of periphery monitoring by the driver based on a cumulative time of inattentiveness of the driver within a prescribed time.

[0016] Further, in the monitoring device, preferably, the decision section decides the degree of periphery monitoring by the driver of the vehicle so that the greater the degree to which the driver pays attention to the periphery of the vehicle, the greater a value of the degree of periphery monitoring by the driver of the vehicle has, and the warning section notifies the driver of a warning in a case where it is determined by the determination section that the degree of periphery monitoring by the driver is less than the inattentiveness threshold value.

[0017] Further, in the monitoring device, preferably, the monitoring device further has a first changing section that changes the inattentiveness threshold value set by the setting section based on a relative relationship between a direction of the surrounding object detected by the detection section and a direction in which the driver faces.

[0018] Further, in the monitoring device, it is preferable that the monitoring device further has a second changing unit that changes the inattentive threshold value set by the setting unit based on the speed of the vehicle.

[0019] According to another embodiment, a non-transitory storage medium storing a monitoring computer program is provided. The monitoring computer program causes a processor to execute a process including: detecting a periphery object of a prescribed kind existing within a prescribed region of a periphery of a vehicle based on environment information representing an environment of the periphery of the vehicle; deciding a degree of periphery monitoring by a driver of the vehicle based on information representing a state of the driver; setting an inattentive threshold value that is a criterion of the degree of periphery monitoring by the driver to a first value in a case where a number of the periphery objects detected within the prescribed region is less than an object number threshold value, and setting the inattentive threshold value to a second value smaller than the first value in a case where the number of the periphery objects is equal to or more than the object number threshold value; determining whether the degree of periphery monitoring by the driver exceeds the inattentive threshold value; and notifying the driver of a warning via a notification unit in a case where it is determined that the degree of periphery monitoring by the driver exceeds the inattentive threshold value, so as to cause the driver to pay attention to the periphery of the vehicle.

[0020] According to still another embodiment, a monitoring method is provided. The monitoring method is executed by a monitoring device, and includes: detecting a periphery object of a prescribed kind existing within a prescribed region of a periphery of a vehicle based on environment information representing an environment of the periphery of the vehicle; deciding a degree of periphery monitoring by a driver of the vehicle based on information representing a state of the driver; setting an inattentive threshold value that is a criterion of the degree of periphery monitoring by the driver to a first value in a case where a number of the periphery objects detected within the prescribed region is less than an object number threshold value, and setting the inattentive threshold value to a second value smaller than the first value in a case where the number of the periphery objects is equal to or more than the object number threshold value; determining whether the degree of periphery monitoring by the driver exceeds the inattentive threshold value; and notifying the driver of a warning via a notification unit in a case where it is determined that the degree of periphery monitoring by the driver exceeds the inattentive threshold value, so as to cause the driver to pay attention to the periphery of the vehicle.

[0021] Effects of Invention

[0022] The monitoring device of the present disclosure can appropriately monitor a driver by changing a criterion for monitoring the driver in accordance with a periphery situation of a vehicle. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 (A) of FIG. 1 is a diagram illustrating an outline of an action of the monitoring device of the present embodiment, and is a diagram showing a situation in which one other vehicle is traveling in a periphery of a vehicle.

[0024] Figure 1(B) is a diagram that illustrates an outline of the operation of the monitoring device of the present embodiment, and is a diagram that shows a case where a plurality of other vehicles are traveling in the periphery of the vehicle.

[0025] Figure 1 (C) is a diagram that illustrates an outline of the operation of the monitoring device of the present embodiment, and is a diagram that shows inside the vehicle cabin.

[0026] Figure 2 is a diagram that shows a schematic configuration of a vehicle on which the monitoring system of the present embodiment is installed.

[0027] Figure 3 is an example of an action flowchart related to the monitoring processing of the monitoring device of the present embodiment.

[0028] Figure 4 is a diagram that illustrates the front region and the side view region.

[0029] Figure 5 is a diagram that illustrates an example of determining the degree of the driver's monitoring of the periphery.

[0030] Figure 6 is an example of an action flowchart related to the threshold value setting processing of the monitoring device of the present embodiment.

[0031] Figure 7 is a diagram that illustrates Modification 1.

[0032] Figure 8 is an example of an action flowchart that illustrates Modification 2. DETAILED DESCRIPTION

[0033] Figure 1 A ~ Figure 1 C is a diagram that illustrates an outline of the operation of the monitoring device 12 of the present embodiment. Figure 1 A is a diagram that shows a case where one other vehicle 60 is traveling in the periphery of the vehicle 10, Figure 1 B is a diagram that shows a case where a plurality of other vehicles 61, 62, 63 are traveling in the periphery of the vehicle, Figure 1 C is a diagram that shows inside the vehicle cabin 30. Hereinafter, an outline of the operation related to the monitoring processing of the monitoring device 12 disclosed in the present specification will be described with reference to Figure 1 A ~ Figure 1 C.

[0034] The vehicle 10 has an object detection device 11 and a monitoring device 12. The object detection device 11 uses the front camera 2 and the LiDAR sensor 4 (refer to Figure 2The object detection device 11 detects other objects present in a region in front of the periphery of the vehicle 10 using the front camera 2 and the LiDAR sensor 4. The region in front of the vehicle 10 in which the other objects are detected using the front camera 2 and the LiDAR sensor 4 is an example of a prescribed region of the periphery of the vehicle 10. The monitoring device 12 monitors the state of the driver.

[0035] In Figure 1 In the example illustrated in FIG. 1, the vehicle 10 is traveling on a road 50. Another vehicle 60 is traveling in a region in front of the vehicle 10. The object detection device 11 detects the other vehicle 60 in the region in front of the vehicle 10 using the front camera 2 and the LiDAR sensor 4.

[0036] The monitoring device 12 estimates the orientation of the line of sight of the driver 40 using the monitoring image captured by the monitoring camera 3. The monitoring device 12 determines whether the orientation of the line of sight of the driver 40 is included in the front region S (non-side view region) or in the side view region N. The front region S is expressed in a prescribed range of angles in the horizontal direction to the left and right from a direction toward the front with the driver's seat 31 in the passenger compartment 30 as a starting point. The side view region N is expressed in a range of angles other than the range expressing the front region S in the horizontal direction to the left and right with respect to the direction toward the front with the driver's seat 31 in the passenger compartment 30 as a starting point.

[0037] In the case where the orientation of the line of sight of the driver 40 is included in the front region S, it can be considered that the driver 40 is monitoring the region in front of the vehicle 10. On the other hand, in the case where the orientation of the line of sight of the driver 40 is included in the side view region N, it can be considered that the driver 40 is not monitoring the region in front of the vehicle 10.

[0038] Therefore, the monitoring device 12 determines that side view is not being performed in the case where the orientation of the line of sight of the driver 40 is included in the front region S, and determines that side view is being performed in the case where the orientation of the line of sight of the driver 40 is included in the side view region N.

[0039] In addition, the monitoring device 12 decides the degree to which the driver 40 is monitoring the periphery based on the monitoring image. The monitoring image is an example of information indicating the state of the driver. Specifically, the monitoring device 12 decides the duration of time for which the driver is side viewing as the degree to which the driver 40 is monitoring the periphery.

[0040] In Figure 1In the example shown in A, since the number (1) of other vehicles 60 detected by the object detection device 11 in the region in front of the vehicle 10 is less than an object number threshold value (for example, 2), the monitoring device 12 sets an inattentive threshold value, which is a criterion of the degree of the periphery monitoring by the driver 40 of the vehicle 10, to a first value (for example, 3 seconds). The other vehicles 60 are an example of the periphery objects of a prescribed kind. The other vehicles 60 are likely to approach the vehicle 10.

[0041] Since the degree of the periphery monitoring by the driver does not exceed the inattentive threshold value, the monitoring device 12 does not issue a warning to the driver 40.

[0042] On the other hand, in Figure 1 In the example shown in B, other vehicles 61, 62, 63 are traveling in the region in front of the vehicle 10. Since the number (3) of the other vehicles 61, 62, 63 detected by the object detection device 11 in the region in front of the vehicle 10 is equal to or more than an object number threshold value (for example, 2), the monitoring device 12 sets an inattentive threshold value, which is a criterion of the degree of the periphery monitoring by the driver 40 of the vehicle 10, to a second value (for example, 2 seconds) that is smaller than the first value.

[0043] When the number of the other vehicles detected in the region in front of the vehicle 10 is large, the driver 40 is required to pay attention to each of the other vehicles. Therefore, the driver 40 is required to have a higher degree of the periphery monitoring.

[0044] Since the degree of the periphery monitoring by the driver exceeds the inattentive threshold value, the monitoring device 12 notifies the driver 40 of a warning via a user interface (UI) 7 so as to make the driver 40 pay attention to the front of the vehicle 10. The user interface 7 is an example of a notification section.

[0045] As described above, the monitoring device 12 can appropriately monitor the driver 40 by changing the criterion of monitoring the driver 40 in accordance with the periphery situation of the vehicle 10.

[0046] Figure 2 is a schematic configuration diagram of the vehicle 10 on which the monitoring system 1 of the present embodiment is installed. The vehicle 10 has a front camera 2, a monitoring camera 3, a LiDAR sensor 4, a vehicle speed sensor 5, a seat belt drive device 6, a user interface (UI) 7, an object detection device 11, and a monitoring device 12, and the like. The monitoring system 1 has at least the monitoring device 12.

[0047] The front camera 2, the monitoring camera 3, the LiDAR sensor 4, the vehicle speed sensor 5, the seat belt drive device 6, the user interface (UI) 7, the object detection device 11, and the monitoring device 12 are communicably connected via an in-vehicle network 13 in accordance with a specification such as a controller area network.

[0048] The front camera 2 is an example of an imaging section provided to the vehicle 10. The front camera 2 is installed to the vehicle 10 in a manner facing the front of the vehicle 10. The front camera 2 captures a camera image showing the environment of a prescribed region in front of the vehicle 10, for example, at a camera image capturing timing set at a prescribed cycle. In the camera image, a road feature such as a road and a lane division line on the road surface of the road included in the prescribed region in front of the vehicle 10 can be shown. The camera image is an example of environmental information representing the environment of the surroundings of the vehicle 10. The front camera 2 has a two-dimensional detector composed of an array of photoelectric conversion elements having sensitivity to visible light such as a CCD or a C-MOS, and an imaging optical system imaging an image of a region to be captured on the two-dimensional detector. It is preferable that the field of view of the front camera 2 include the front region S.

[0049] The front camera 2 outputs the camera image and the camera image capturing timing to the object detection device 11 and the like via the in-vehicle network 13 each time the camera image is captured. The camera image is used in the processing of detecting other objects in the surroundings of the vehicle 10 in the object detection device 11.

[0050] The monitoring camera 3 is arranged in the passenger compartment 30 so as to be able to capture a monitoring image including the vicinity of the driver's seat. In the monitoring image, the face of the driver 40 driving the vehicle 10 can be included. The monitoring camera 3 is an example of an imaging device that captures a monitoring image including the face of the driver 40. The monitoring camera 3 can be arranged, for example, in a not-shown steering column, a rearview mirror in the passenger compartment, an instrument panel, an instrument cover, and the like.

[0051] The monitoring camera 3 captures the monitoring image, for example, at a monitoring image capturing timing set at a prescribed cycle. The monitoring camera 3 has a two-dimensional detector composed of an array of photoelectric conversion elements having sensitivity to infrared rays such as a CCD or a C-MOS, and an imaging optical system imaging an image of a region to be captured on the two-dimensional detector. It is preferable that the monitoring camera 3 have the two-dimensional detector and a spotlight capable of irradiating the driver. The spotlight is an LED (Light Emitting Diode), for example, two near-infrared LEDs arranged on both sides of the imaging optical system. By irradiating the driver with near-infrared light, even at low illuminance such as at night, the face of the driver can be captured without giving the driver an unpleasant feeling. The monitoring camera 3 outputs the monitoring image and the monitoring image capturing timing to the monitoring device 12 and the like via the in-vehicle network 13 each time the monitoring image is captured.

[0052] The LiDAR sensor 4 is mounted to the outer surface of the vehicle 10, for example, in a manner facing the front of the vehicle 10. The LiDAR sensor 4 emits laser light in a manner scanning the front of the vehicle 10 at a reflection wave information acquisition timing set at a prescribed cycle, and receives a reflection wave reflected by a reflection object. The time required for the reflection wave to return has distance information between the other object located in the direction of the irradiated laser light and the vehicle 10. Preferably, the range in which the LiDAR sensor 4 scans the front of the vehicle 10 includes the front region S. The LiDAR sensor 4 outputs the reflection wave information including the direction of the irradiated laser light and the time required for the reflection wave to return to the object detection device 11 together with the reflection wave information acquisition timing after the laser light is emitted, via the in-vehicle network 13. The reflection wave information is used in the processing for detecting the other object in the periphery of the vehicle 10 in the object detection device 11. The reflection wave information is an example of the environment information indicating the environment in the periphery of the vehicle 10.

[0053] The vehicle speed sensor 5 acquires speed information indicating the speed of the vehicle 10, and outputs the speed information to the monitoring device 12 via the in-vehicle network 13. The speed information indicates the rotational speed of a tire (not shown), for example. The speed information is used in the monitoring device 12 to calculate the speed of the vehicle 10.

[0054] The seat belt drive device 6 is controlled by the monitoring device 12, for example, to adjust the tension of the seat belt 32 worn by the driver 40. The monitoring device 12 is able to notify the driver 40 of a warning by changing the tension of the seat belt 32, so as to cause the driver 40 to pay attention to the front of the vehicle 10.

[0055] The UI 7 is an example of a notification section. The UI 7 is controlled by the monitoring device 12 or the like to notify the driver 40 of a warning or the like requiring attention to the front of the vehicle 10. In order to display the warning or the like, the UI 7 has a display device 7a such as a liquid crystal display or a touch panel. In addition, the UI 7 can have a sound output device (not shown) for notifying the driver of the warning or the like. In addition, the UI 7 generates an operation signal corresponding to the operation of the vehicle 10 by the driver. As an input device for inputting operation information to the vehicle 10 by the driver, the UI 7 has a touch panel or an operation button, for example. The UI 7 outputs the input operation information to the monitoring device 12 or the like via the in-vehicle network 13.

[0056] The object detection device 11 detects other objects and their kinds in front of the periphery of the vehicle 10 based on the camera image. The other objects include moving objects such as pedestrians and other vehicles moving in the periphery of the vehicle 10. The object detection device 11 has, for example, an identifier that identifies a moving object shown in the image by inputting the camera image to the identifier. As the identifier, for example, a deep neural network (DNN) that is pre-learned to detect an object shown in an image according to the input image can be used. The object detection device 11 can also use an identifier other than the DNN. As the identifier, the object detection device 11 can also use, for example, a support vector machine (SVM) that is pre-learned to input a feature amount (for example, a Histogram of Oriented Gradients (HOG)) calculated according to a window set on an image and output a confidence factor indicating an object to be a detection target to the window. Alternatively, the object detection device 11 can also detect an object region by performing template matching between a template showing an object to be a detection target and an image.

[0057] In addition, the object detection device 11 can also detect other objects in front of the periphery of the vehicle 10 based on the reflection wave information. In addition, the object detection device 11 can also calculate the orientation of the other object with respect to the vehicle 10 based on the position of the other object in the camera image, and calculate the distance between the other object and the vehicle 10 based on the orientation and the reflection wave information output from the LiDAR sensor 4. The object detection device 11 estimates the position of the other object shown in, for example, a vehicle coordinate system with the vehicle as the origin based on the current position of the vehicle 10 and the distance and the orientation with respect to the vehicle 10 to the other object. The object detection device 11 outputs object detection information including information showing the kind of the detected other object (for example, a pedestrian, a vehicle) and information showing the position thereof to the monitoring device 12 or the like.

[0058] The monitoring device 12 performs a detection process, a setting process, a decision process, a determination process, a warning process, and a change process. Therefore, the monitoring device 12 has a communication interface (IF) 21, a memory 22, and a processor 23. The communication interface 21, the memory 22, and the processor 23 are connected via a signal line 24. The communication interface 21 has an interface circuit for connecting the monitoring device 12 with the in-vehicle network 13.

[0059] The memory 22 is an example of a storage unit, and has, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. The memory 22 stores a computer program of an application program used in information processing performed by the processor 23 and various data.

[0060] All or part of the functions of the monitoring device 12 are implemented, for example, by a computer program running on the processor 23. The processor 23 has a detection unit 231, a setting unit 232, a decision unit 233, a judgment unit 234, a warning unit 235, and a modification unit 236. Alternatively, the functional modules of the processor 23 may be dedicated arithmetic circuits provided in the processor 23. The processor 23 has one or more CPUs (Central Processing Units) and their peripheral circuits. Alternatively, the processor 23 may also have other arithmetic circuits such as logic operation units, numerical operation units, or graphics processing units. The operation of the monitoring device 12 will be described later.

[0061] The object detection device 11 and the monitoring device 12 are, for example, electronic control units (ECUs). Figure 2 In this document, the monitoring device 12 is described as a different device, but all or part of these devices may also constitute a single device.

[0062] Figure 3 This is an example of an operation flowchart related to the monitoring processing of the monitoring device 12 in this embodiment. Hereinafter, refer to... Figure 3 The monitoring process of the monitoring device 12 will be explained. The monitoring device 12, during monitoring at predetermined intervals, will... Figure 3 The flowchart shown is used to perform monitoring processing.

[0063] First, the decision unit 233 determines the degree of surrounding surveillance by the driver 40 of the vehicle 10 based on the camera images (step S101). For example, the decision unit 233 determines the degree of surrounding surveillance by the driver of the vehicle 10 such that the greater the degree to which the driver 40 fails to pay attention to the surroundings of the vehicle 10, the greater the value of the degree of surrounding surveillance by the driver of the vehicle 10. Details of the process by which the decision unit 233 determines the degree of surrounding surveillance by the driver will be described later.

[0064] Next, the determination unit 234 determines whether the driver 40's level of monitoring of the surroundings exceeds the negligence threshold that constitutes a standard for a decrease in the driver 40's level of monitoring of the surroundings (step S102). The threshold setting process for setting the negligence threshold will be described later.

[0065] In a case where the degree of the driver's 40 periphery monitoring exceeds the inattentive threshold (step S102 - "Yes"), the warning section 235 notifies the driver 40 of a warning requiring attention to the front of the vehicle 10 via the UI 7 (step S103), and ends the series of processes. The warning can also be displayed using the display device 7a. In addition, the warning can also be notified in a sound manner using the sound output device of the UI 7. Furthermore, the warning section 235 can notify the driver 40 of the warning by periodically changing the tension of the seat belt 32 via the seat belt driving device 6, so as to cause the driver 40 to pay attention to the front of the vehicle 10. Furthermore, the method of notifying the warning is not limited to the above-described method, and other methods can also be used.

[0066] Here, in a case where the driver 40 does not direct the line of sight to the front region S even if the warning for the prescribed time is performed, the warning section 235 can decide to stop the vehicle 10.

[0067] On the other hand, in a case where the degree of the driver's periphery monitoring does not exceed the inattentive threshold (step S102 - "No"), the series of processes is ended.

[0068] Next, the process of the decision section 233 deciding the degree of the driver's periphery monitoring will be described below with reference to Figure 4

[0069] Figure 4 is a diagram illustrating the front region S and the side view region N. The front region S is indicated in a prescribed range of angle in the horizontal direction to the left and right from the direction toward the front with the driver's seat 31 in the vehicle cabin 30 as a starting point. The prescribed range in the horizontal direction can be set, for example, to a range from the right end of the front window of the vehicle 10 toward the left side until 2 / 3 of the front window with respect to the direction toward the front with the driver's seat 31 in the vehicle cabin 30 as a starting point. The side view region N is indicated in a range of angle other than the range indicating the front region S in the horizontal direction to the left and right with respect to the direction toward the front with the driver's seat 31 in the vehicle cabin 30 as a starting point. It is preferable that the region of the front of the vehicle 10 in which the object detection device 11 detects other objects include the front region S.

[0070] The decision section 233 estimates the direction of the line of sight of the driver 40 using the monitoring image captured by the monitoring camera 3. For example, the decision section 233 detects the direction of the line of sight based on the monitoring image using the corneal reflection method or the eye center method. The direction of the line of sight of the driver 40 is indicated in an angle of the horizontal direction and an angle of the vertical direction with respect to the direction toward the front of the vehicle 10. The decision section 233 estimates the direction of the line of sight of either one of the eyes of the driver 40. Alternatively, the decision section 233 estimates the average of the directions of the lines of sight of the two eyes of the driver 40 as the direction of the line of sight of the driver 40.​

[0071] The determination unit 233 determines whether the orientation of the horizontal direction of the line of sight of the driver 40 is included in the front region S (non-side view region) or in the side view region N. In the case where the orientation of the horizontal direction of the line of sight of the driver 40 is included in the front region S, it can be considered that the driver 40 is monitoring the region in front of the vehicle 10. On the other hand, in the case where the orientation of the horizontal direction of the line of sight of the driver 40 is included in the side view region N, it can be considered that the driver 40 is not monitoring the region in front of the vehicle 10.

[0072] Therefore, in the case where the orientation of the horizontal direction of the line of sight of the driver 40 is included in the front region S, the determination unit 233 determines that side view is not being performed, and in the case where the orientation of the horizontal direction of the line of sight of the driver 40 is included in the side view region N, the determination unit 233 determines that side view is being performed.

[0073] The determination unit 233 determines the degree of the surrounding monitoring by the driver 40 on the basis of the duration of the side view by the driver 40. Specifically, in the case where the driver 40 is performing side view, the determination unit 233 determines the duration of the side view by the driver 40 as the degree of the surrounding monitoring by the driver 40. The determination unit 233 calculates the time during which the driver 40 continuously casts the line of sight to the side view region N without looking at the front region S. The greater the degree to which the driver 40 does not pay attention to the front of the vehicle 10, the longer the duration of the side view by the driver 40 becomes (the greater the value of the time).

[0074] In addition, the determination unit 233 can determine the degree of the surrounding monitoring by the driver 40 using other methods. For example, the determination unit 233 can determine the degree of the surrounding monitoring by the driver 40 on the basis of the number of times of the side view by the driver 40 within a prescribed time. Specifically, the determination unit 233 can determine the number of times of the side view by the driver 40 within a prescribed determination time as the degree of the surrounding monitoring by the driver 40. As the determination time, for example, 30 to 60 seconds can be set. The greater the degree to which the driver 40 does not pay attention to the front of the vehicle 10, the greater the number of times of the side view by the driver 40 becomes.

[0075] In addition, the determination unit 233 can determine the degree of the surrounding monitoring by the driver 40 on the basis of the accumulated time of the side view by the driver 40 within a prescribed time. Specifically, the determination unit 233 can determine the accumulated time of the side view by the driver 40 within a prescribed time as the degree of the surrounding monitoring by the driver 40. Figure 5 This is a diagram that illustrates an example of determining the degree of the surrounding monitoring by the driver 40. As the determination time, for example, 30 to 60 seconds can be set. The greater the degree to which the driver 40 does not pay attention to the front of the vehicle 10, the greater the accumulated time of the side view by the driver 40 within the prescribed time becomes.

[0076] In Figure 5 In the example shown, the driver 40 repeatedly performs the following actions: performs 2 seconds of side vision, directs the line of sight forward for 1 second, performs 2 seconds of side vision, and directs the line of sight forward for 1 second. In this way, the driver 40 repeatedly performs side vision in the form of a glance. On the other hand, with respect to the duration of the side vision of the driver 40, in a case where the inattentive threshold is set to 3 seconds, the side vision of such a driver 40 cannot be detected. Therefore, in the present embodiment, the cumulative time of the side vision of the driver 40 within a prescribed time is set to the degree of the monitoring of the surroundings by the driver 40. One or more of the above-described plurality of processes of determining the degree of the monitoring of the surroundings by the driver can be used.

[0077] Next, the following, with reference to Figure 6 The threshold setting process of setting the inattentive threshold to the monitoring device 12 will be described. Figure 6 is an example of an action flowchart related to the threshold setting process of the monitoring device of the present embodiment. The monitoring device 12 executes the threshold setting process in accordance with the action flowchart shown in Figure 6 at a threshold setting time having a prescribed period. It is preferable that the period of the threshold setting time be equal to or less than the period of the monitoring time.

[0078] First, the detection unit 231 detects a prescribed kind of surrounding object existing within a prescribed region of the surroundings of the vehicle 10 based on the object detection information (step S201). In the present embodiment, the prescribed region is a region in front of the surroundings of the vehicle 10. In addition, the prescribed kind of surrounding object is a moving object. The moving object has the potential to approach the vehicle 10. Specifically, as the moving object, other vehicles or pedestrians can be cited. The detection unit 231 detects the moving object located in the region in front of the vehicle 10 based on the object detection information input from the object detection device 11. Furthermore, the detection unit 231 can also have the function of the object detection device 11.

[0079] Next, the setting unit 232 determines whether the number of the surrounding objects detected within the prescribed region of the surroundings of the vehicle 10 is less than the object number threshold (step S202). Specifically, the setting unit 232 determines whether the number of other vehicles and pedestrians detected within the region in front of the vehicle 10 is less than the object number threshold.

[0080] The object number threshold value can be set in accordance with the number of objects that the driver 40 is able to direct his or her attention to in the line of sight normally. For example, in the case where the surrounding objects are vehicles or pedestrians, the object number threshold value can be set to 2 or 3. In the case where the surrounding objects are one, the driver 40 is able to direct his or her attention to the surrounding object only, but in the case where the surrounding objects are two or more, the driver 40 sometimes has difficulty in directing his or her attention to the line of sight dividedly. In particular, as the distance between the two surrounding objects becomes longer, the driver 40 has more difficulty in directing his or her attention to the line of sight dividedly.

[0081] For example, in the case where a plurality of surrounding objects are detected, the object number threshold value can be changed in accordance with the minimum adjacent distance between two of the plurality of surrounding objects. For example, in the case where the minimum adjacent distance is longer than a prescribed reference distance, the object number threshold value can be made smaller than in the case where the minimum adjacent distance is the prescribed reference distance or less.

[0082] In the case where the number of surrounding objects is less than the object number threshold value (step S202 - "Yes"), the setting section 232 sets the inattention threshold value to a first value (step S203), and ends the series of processing. On the other hand, in the case where the number of surrounding objects is the object number threshold value or more (step S202 - "No"), the setting section 232 sets the inattention threshold value to a second value that is smaller than the first value (step S203), and ends the series of processing.

[0083] For example, in the case where the degree to which the driver 40 monitors the surroundings is the duration for which the driver 40 looks aside, the setting section 232 sets the inattention threshold value with respect to the duration for which the driver 40 looks aside. For example, the first value can be set to 3 seconds, and the second value can be set to 2 seconds.

[0084] In addition, in the case where the degree to which the driver 40 monitors the surroundings is the number of times that the driver 40 looks aside within a prescribed time, the setting section 232 sets the inattention threshold value with respect to the number of times that the driver 40 looks aside within the prescribed time. For example, with respect to the number of times that the driver 40 looks aside within 30 seconds, the first value can be set to 4 times, and the second value can be set to 2 times.

[0085] Furthermore, in the case where the degree to which the driver 40 monitors the surroundings is the cumulative time for which the driver 40 looks aside within a prescribed time, the setting section 232 sets the inattention threshold value with respect to the cumulative time for which the driver 40 looks aside within the prescribed time. For example, with respect to the cumulative time for which the driver 40 looks aside within 30 seconds, the first value can be set to 10 seconds, and the second value can be set to 5 seconds.

[0086] When the number of other vehicles detected in the area in front of the vehicle 10 is large, the driver 40 is required to pay attention to each of the other vehicles. Therefore, the driver 40 is required to have a higher degree of peripheral monitoring. Therefore, the monitoring device 12 sets the inattentive threshold in accordance with the number of peripheral objects detected in the prescribed area around the vehicle 10. Thus, the inattentive threshold is set to be likely to notify the driver 40 of a warning in a case where the number of peripheral objects is large, as compared with a case where the number of peripheral objects is small.

[0087] As explained above, the monitoring device of the present embodiment is able to appropriately monitor the driver by changing the criterion for monitoring the driver in accordance with the situation around the vehicle.

[0088] Next, a modification of the monitoring device 12 of the above-described embodiment will be explained below with reference to Figure 7 and Figure 8 .

[0089] Figure 7 is a view for explaining Modification 1. In the present modification, the changing section 236 changes the inattentive threshold set by the setting section 232 based on the relative relationship between the direction of the peripheral object detected by the detecting section 231 and the direction in which the driver 40 is facing.

[0090] In the example shown in Figure 7 , the vehicle 10 is making a right turn from the road 51 to the road 52 at the intersection 53 where the road 51 and the road 52 cross. The orientation P1 of the line of sight of the driver 40 is oriented in the right turn direction toward the pedestrian 64 walking on the pedestrian crosswalk 66 on the road 52. As the direction in which the driver 40 is facing P1, the orientation of the line of sight of the driver 40 can be used.

[0091] Here, the other vehicle 65 is traveling on the road 51 toward the intersection 53 from the opposite side of the vehicle 10. Since the direction P1 in which the driver 40 is facing is farther away than the prescribed reference angle in the horizontal direction with respect to the direction P2 of the other vehicle 65 detected by the detecting section 231, the changing section 236 changes the inattentive threshold so as to make the inattentive threshold smaller as compared with a case where the direction P1 in which the driver 40 is facing is the prescribed reference angle or less with respect to the direction P2 of the other vehicle 65 detected. Specifically, when the inattentive threshold is the first value, the changing section 236 sets the inattentive threshold to the third value which is smaller than the first value. In addition, when the inattentive threshold is the second value, the changing section 236 sets the inattentive threshold to the fourth value which is smaller than the second value.

[0092] Thus, in a case where the direction of the peripheral object detected by the detecting section 231 is farther away than the direction in which the driver 40 is facing, the monitoring device 12 is likely to notify the driver 40 of a warning regardless of the number of peripheral objects.

[0093] Figure 8 FIG. 27 is an example of an operation flowchart illustrating a modification 2 of the operation flowchart of FIG. 25. In the threshold setting processing of this modification, the changing section 236 changes the inattentive threshold set by the setting section 232 on the basis of the speed of the vehicle 10. Thus, the threshold setting processing of this modification differs from the above-described threshold setting processing in that steps S305 and S306 are added. The processing of steps S301 to S304 of this modification is the same as that of steps S101 to S104 described above.

[0094] After the inattentive threshold is set to the first value or the inattentive threshold is set to the second value, the setting section 232 determines whether the speed of the vehicle 10 is faster than a prescribed reference speed (step S305).

[0095] In the case where the speed of the vehicle 10 is faster than the prescribed reference speed (step S305 - "Yes"), the setting section 232 sets the inattentive threshold so as to be smaller than the inattentive threshold in the case where the speed of the vehicle 10 is the prescribed reference speed or less (step S306), and ends the series of processing. For example, the setting section 232 sets the product of the current inattentive threshold and a coefficient K (K is a positive real number of 1.0 or less) as the new inattentive threshold.

[0096] On the other hand, in the case where the speed of the vehicle 10 is not faster than the prescribed reference speed (step S305 - "No"), the series of processing is ended.

[0097] Thus, in the case where the speed of the vehicle 10 is relatively fast and the number of surrounding objects is relatively large, it is easy to notify the driver 40 of a warning regardless of the number of surrounding objects.

[0098] In the present disclosure, the monitoring device, the monitoring computer program, and the monitoring method of the above-described embodiments can be appropriately changed without departing from the gist of the present disclosure. In addition, the technical scope of the present disclosure is not limited to the above-described embodiments but encompasses the technical scope of the claims and equivalents thereof.

[0099] For example, the prescribed region of the periphery of the vehicle can also be a region other than the front of the vehicle. For example, it can be that, in the case where the vehicle 10 turns right, the prescribed region is set to a region on the right side of the vehicle, in the case where the vehicle 10 turns left, the prescribed region is set to a region on the left side of the vehicle, and in the case where the vehicle 10 backs up, the prescribed region is set to a region behind the vehicle.

[0100] In addition, the prescribed kind of surrounding object present in the prescribed area around the vehicle can also be an object other than a moving object. For example, it can be an object having a size smaller than a prescribed reference size. Since a smaller object is more likely to be overlooked than a larger object, it is useful to arouse the driver's further attention. Furthermore, the size of the object can be estimated based on the reflected wave information.

[0101] In addition, in the above-described embodiment, the information indicating the state of the driver is a monitoring image, but can also be information other than an image. For example, the posture of the driver can be estimated using a pressure-sensitive sensor or the like disposed at the driver's seat, and the degree to which the driver monitors the surroundings can be determined based on the posture of the driver.

[0102] In addition, the determination unit can determine the degree to which the driver of the vehicle monitors the surroundings so that the greater the degree to which the driver pays attention to the surroundings of the vehicle, the greater the value of the degree to which the driver of the vehicle monitors the surroundings. In this case, the warning unit notifies the driver of a warning when the determination unit determines that the degree to which the driver monitors the surroundings is less than the inattentive threshold.

[0103] In addition, the monitoring device can monitor a state of the driver other than the side view. For example, the monitoring device can monitor the degree of eye opening or the degree of mouth opening, which are indices of the degree to which the driver is involved in driving.

Claims

1. A monitoring device, characterized in that, The monitoring device has: The detection unit detects a specified type of surrounding objects existing within a specified area around the vehicle based on environmental information representing the environment around the vehicle. The decision unit determines the degree of monitoring of the surroundings by the driver of the vehicle based on information indicating the driver's state. The setting unit sets a first value as a standard for the driver's level of surveillance of the surroundings when the number of surrounding objects detected by the detection unit in the specified area is less than an object number threshold; and sets a second value, which is smaller than the first value, when the number of surrounding objects is greater than or equal to the object number threshold. The determination unit determines whether the degree of driver's monitoring of the surroundings, as determined by the decision unit, exceeds the negligence threshold. as well as The warning unit, when the determination unit determines that the driver's level of monitoring of the surroundings exceeds the negligence threshold, notifies the driver via the notification unit to issue a warning, so as to make the driver pay attention to the surroundings of the vehicle.

2. The monitoring device according to claim 1, characterized in that, The decision unit determines the degree to which the driver of the vehicle monitors the surroundings, such that the greater the degree to which the driver fails to pay attention to the vehicle's surroundings, the higher the value of the driver's degree of monitoring of the surroundings. The warning unit notifies the driver of the warning if the determination unit determines that the driver's level of monitoring of the surroundings is greater than the negligence threshold.

3. The monitoring device according to claim 2, characterized in that, The decision-making unit determines the degree of driver surveillance of the surroundings based on the duration of the driver's side-eye observation.

4. The monitoring device according to claim 2, characterized in that, The decision-making unit determines the degree of driver surveillance of the surroundings based on the number of times the driver glances around within a specified time.

5. The monitoring device according to claim 2, characterized in that, The decision-making unit determines the degree of driver surveillance of the surroundings based on the cumulative time the driver spends observing the surroundings within a specified period.

6. The monitoring device according to claim 1, characterized in that, The decision unit determines the degree to which the driver of the vehicle monitors the surroundings, such that the greater the driver's attention to the vehicle's surroundings, the higher the value of the driver's degree of monitoring of the surroundings. The warning unit notifies the driver of the warning if the determination unit determines that the driver's level of monitoring of the surroundings is less than the negligence threshold.

7. The monitoring device according to any one of claims 1 to 6, characterized in that, The monitoring device also has a first modification unit, which modifies the negligence threshold set by the setting unit based on the relative relationship between the direction of the surrounding object detected by the detection unit and the direction the driver is facing.

8. The monitoring device according to any one of claims 1 to 6, characterized in that, The monitoring device also has a second modification unit that modifies the negligence threshold set by the setting unit based on the speed of the vehicle.

9. The monitoring device according to claim 7, characterized in that, The monitoring device also has a second modification unit that modifies the negligence threshold set by the setting unit based on the speed of the vehicle.

10. A non-transitory storage medium, characterized in that, The non-transitory storage medium stores a surveillance computer program that can be read by a computer, the surveillance computer program causing the processor to perform the following processes, the processes including: Based on environmental information representing the environment surrounding the vehicle, detect a specified type of surrounding objects existing within a specified area around the vehicle; The degree to which the driver of the vehicle monitors the surroundings is determined based on information indicating the driver's state. When the number of surrounding objects detected in the specified area is less than the number of objects threshold, a first value is set as the standard for the driver's level of monitoring of the surroundings. When the number of surrounding objects is greater than the number of objects threshold, the negligence threshold is set as a second value that is smaller than the first value. Determine whether the driver's level of attention to the surrounding environment exceeds the negligence threshold; and If it is determined that the driver's level of attention to the surroundings exceeds the negligence threshold, a warning is issued to the driver via the notification unit to make the driver pay attention to the surroundings of the vehicle.

11. A monitoring method, wherein the monitoring method is performed by a monitoring device, characterized in that, include: Based on environmental information representing the environment surrounding the vehicle, detect a specified type of surrounding objects existing within a specified area around the vehicle; The degree to which the driver of the vehicle monitors the surroundings is determined based on information indicating the driver's state. When the number of surrounding objects detected in the specified area is less than the number of objects threshold, a first value is set as the standard for the driver's level of monitoring of the surroundings. When the number of surrounding objects is greater than the number of objects threshold, the negligence threshold is set as a second value that is smaller than the first value. Determine whether the driver's level of attention to the surrounding environment exceeds the negligence threshold; as well as If it is determined that the driver's level of attention to the surroundings exceeds the negligence threshold, a warning is issued to the driver via the notification unit to make the driver pay attention to the surroundings of the vehicle.

Images