Monitoring device, monitoring computer program, and monitoring method
The monitoring device adjusts criteria for monitoring drivers based on surrounding conditions, using sensors to detect objects and issue warnings, effectively reducing distracted driving by dynamically adapting to environmental changes.
Patent Information
- Application Number
- JP2022080857
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2022-05-17
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-05-17
AI Technical Summary
Existing monitoring systems for vehicle drivers do not adequately adjust criteria based on dynamic changes in the surrounding environment, such as the presence and movement of objects, leading to inadequate monitoring and potential safety risks.
A monitoring device that adjusts criteria for monitoring drivers based on the number, type, and relative position of surrounding objects, using sensors to detect objects and determine the driver's attention level, issuing warnings when necessary to maintain focus on the surroundings.
The system effectively monitors driver attention by dynamically adjusting thresholds based on environmental conditions, reducing the likelihood of distracted driving by providing timely warnings when the driver's attention is compromised.
Smart Images

Figure 0007586137000001 
Figure 0007586137000002 
Figure 0007586137000003
Abstract
Description
[Technical field]
[0001] The present disclosure relates to a monitoring device, a monitoring computer program, and a monitoring method. [Background technology]
[0002] Conventionally, a monitoring device mounted on a vehicle monitors the state of a driver and monitors whether the driver is paying attention to the surroundings of the vehicle. For example, the monitoring device mounted on a vehicle monitors whether the driver is looking ahead of the vehicle and issues a warning to the driver if the driver is looking aside.
[0003] For example, Patent Document 1 proposes an in-vehicle system that measures the degree of inappropriate driving, such as distracted driving (inappropriate driving level), and issues a warning when the inappropriate driving level reaches a countermeasure level. In this in-vehicle system, the countermeasure level is lowered in driving locations where inappropriate driving is likely to occur than in driving locations where it is not. In this way, the in-vehicle system is designed to prevent inappropriate driving from occurring. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2013-65246 A Summary of the Invention [Problem to be solved by the invention]
[0005] As described above, it is proposed that the criteria for monitoring the driver should be changed according to the driving location where inappropriate driving is likely to occur. In addition, moving objects such as vehicles or pedestrians may approach the vehicle. The number of moving objects may change dynamically. Therefore, it is considered preferable to change the criteria for monitoring the driver according to the surrounding situation of the vehicle.
[0006] An object of the present disclosure is to provide a monitoring device that can appropriately monitor a driver by changing the criteria for monitoring the driver depending on the surrounding conditions of the vehicle. [Means for solving the problem]
[0007] According to one embodiment, a monitoring device is provided, which includes a detection unit that detects a predetermined type of surrounding object present within a predetermined area around the vehicle based on environmental information representing the environment around the vehicle, a determination unit that determines the degree of surrounding monitoring by the driver of the vehicle based on information representing the state of the driver, a setting unit that sets an inattention threshold that is a standard of the degree of surrounding monitoring by the driver to a first value when the number of surrounding objects detected within the predetermined area by the detection unit is less than an object number threshold, and sets the inattention threshold to a second value smaller than the first value when the number of surrounding objects is equal to or greater than the object number threshold, a determination unit that determines whether the degree of surrounding monitoring by the driver determined by the determination unit exceeds the inattention threshold, and a warning unit that notifies the driver of a warning to pay attention to the surroundings of the vehicle via a notification unit when the determination unit determines that the degree of surrounding monitoring by the driver exceeds the inattention threshold.
[0008] In addition, in this monitoring device, it is preferable that the determination unit determines the degree of surrounding monitoring by the driver of the vehicle so that the greater the degree to which the driver is not paying attention to the surroundings of the vehicle, the larger the value of the determination unit, and the warning unit notifies the driver of a warning when the judgment unit determines that the degree of surrounding monitoring by the driver is greater than the inattention threshold.
[0009] In the monitoring device, it is preferable that the determination unit determines the degree of surroundings monitoring by the driver based on the duration for which the driver is looking aside.
[0010] In the monitoring device, it is preferable that the determination unit determines the degree of surroundings monitoring by the driver based on the number of times the driver looks aside within a predetermined time.
[0011] In the monitoring device, it is preferable that the determination unit determines the degree of surroundings monitoring by the driver based on an accumulated time during which the driver looks aside within a predetermined time.
[0012] In addition, in this monitoring device, it is preferable that the determination unit determines the degree of surrounding monitoring by the vehicle driver so that the greater the degree to which the driver pays attention to the surroundings of the vehicle, the larger the value of the determination unit, and the warning unit notifies the driver of a warning when the judgment unit determines that the degree of surrounding monitoring by the driver is less than the inattention threshold.
[0013] In addition, it is preferable that this monitoring device further includes a first change unit that changes the inattention threshold set by the setting unit based on the relative relationship between the direction in which a surrounding object is detected by the detection unit and the direction in which the driver is facing.
[0014] It is also preferable that the monitoring device further comprises a second change unit that changes the inattention threshold value set by the setting unit based on the speed of the vehicle.
[0015] According to another embodiment, a monitoring computer program is provided, which is characterized by causing a processor to execute a process including: detecting a predetermined type of surrounding object present within a predetermined area around the vehicle based on environmental information representing the environment around the vehicle; determining a degree of surrounding monitoring by a driver of the vehicle based on information representing a state of the driver; setting an inattention threshold serving as a criterion of the degree of surrounding monitoring by the driver to a first value when the number of surrounding objects detected within the predetermined area is less than an object number threshold; setting the inattention threshold to a second value smaller than the first value when the number of surrounding objects is equal to or greater than the object number threshold; determining whether the degree of surrounding monitoring by the driver exceeds the inattention threshold; and notifying the driver of a warning to pay attention to the surroundings of the vehicle via a notification unit when it is determined that the degree of surrounding monitoring by the driver exceeds the inattention threshold.
[0016] According to still another embodiment, there is provided a monitoring method, which is executed by a warning device, and includes the steps of: detecting a predetermined type of surrounding object present within a predetermined area around the vehicle based on environmental information representing the environment around the vehicle; determining a degree of surrounding monitoring by a driver of the vehicle based on information representing a state of the driver; setting an inattention threshold serving as a criterion of the degree of surrounding monitoring by the driver to a first value when the number of surrounding objects detected within the predetermined area is less than an object number threshold; setting the inattention threshold to a second value smaller than the first value when the number of surrounding objects is equal to or greater than the object number threshold; determining whether the degree of surrounding monitoring by the driver exceeds the inattention threshold; and notifying a warning to the driver via a notification unit to pay attention to the surroundings of the vehicle when it is determined that the degree of surrounding monitoring by the driver exceeds the inattention threshold. Effect of the Invention
[0017] The monitoring device according to the present disclosure can appropriately monitor the driver by changing the criteria for monitoring the driver depending on the surrounding conditions of the vehicle. [Brief description of the drawings]
[0018] [Figure 1] 1A and 1B are diagrams for explaining an overview of the operation of the monitoring device of this embodiment, in which (A) is a diagram showing one other vehicle traveling around the vehicle, (B) is a diagram showing multiple other vehicles traveling around the vehicle, and (C) is a diagram showing the inside of the vehicle cabin. [Diagram 2] 1 is a schematic configuration diagram of a vehicle in which a monitoring system according to an embodiment of the present invention is implemented. [Diagram 3] 5 is an example of an operational flowchart relating to a monitoring process of the monitoring device of the present embodiment. [Figure 4] FIG. 2 is a diagram illustrating a forward area and an aside looking area. [Diagram 5] 10A and 10B are diagrams illustrating an example of determining the degree of surrounding monitoring by a driver. [Figure 6] 10 is an example of an operational flowchart relating to a threshold setting process of the monitoring device of the present embodiment. [Figure 7] FIG. 13 is a diagram for explaining a first modified example. [Figure 8] 13 is an example of an operational flowchart illustrating a second modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0019] Fig. 1 is a diagram for explaining an outline of the operation of the monitoring device 12 of this embodiment. Fig. 1(A) is a diagram showing a state in which one other vehicle 60 is traveling around a vehicle 10, Fig. 1(B) is a diagram showing a state in which a plurality of other vehicles 61, 62, 63 are traveling around the vehicle, and Fig. 1(C) is a diagram showing the inside of a vehicle cabin. Below, an outline of the operation of the monitoring process of the monitoring device 12 disclosed in this specification for monitoring the driver will be explained with reference to Figs. 1(A) to 1(C).
[0020] The vehicle 10 has an object detection device 11 and a monitoring device 12. The object detection device 11 detects other objects present in a forward area around the vehicle 10 using the forward camera 2 and the LiDAR sensor 4. The area in front of the vehicle 10 where other objects are detected by the forward camera 2 and the LiDAR sensor 4 is an example of a predetermined area around the vehicle 10. The monitoring device 12 monitors the state of the driver.
[0021] 1(A), a vehicle 10 is traveling on a road 50. Another vehicle 60 is traveling in the area ahead of the vehicle 10. The object detection device 11 detects the other vehicle 60 in the area ahead of the vehicle 10 using the forward camera 2 and the LiDAR sensor 4.
[0022] The monitoring device 12 estimates the direction of the driver's 40's line of sight using the monitoring camera 3. The monitoring device 12 determines whether the direction of the driver's 40's line of sight is included in the forward area S (non-aside-looking area) or included in the aside-looking area N. The forward area S is represented by a predetermined range of angles in the horizontal direction to the left and right from the direction heading forward starting from the driver's seat 31 in the vehicle interior 30. The aside-looking area N is represented by an angle range excluding the range representing the forward area S in the horizontal direction to the left and right from the direction heading forward starting from the driver's seat 31 in the vehicle interior 30.
[0023] When the direction of the driver's line of sight is included in the forward region S, it is considered that the driver 40 is monitoring the region ahead of the vehicle 10. On the other hand, when the direction of the driver's line of sight is included in the inattentive region N, it is considered that the driver 40 is not monitoring the region ahead of the vehicle 10.
[0024] Therefore, when the direction of the driver's line of sight is included in the forward area S, the monitoring device 12 determines that the driver is not looking aside, and when the direction of the driver's line of sight is included in the inattentive area N, the monitoring device 12 determines that the driver is looking aside.
[0025] The monitoring device 12 also determines the degree of surroundings monitoring by the driver 40 based on the monitoring image. The monitoring image is an example of information representing the state of the driver. Specifically, the monitoring device 12 determines the duration during which the driver is looking aside as the degree of surroundings monitoring by the driver 40.
[0026] In the example shown in FIG. 1(A), the monitoring device 12 sets the inattention threshold, which is a criterion for a decrease in the degree of surrounding monitoring by the driver 40 of the vehicle 10, to a first value (e.g., 3 seconds) because the number (1) of other vehicles 60 detected in the area in front of the vehicle 10 by the object detection device 11 is less than the object number threshold (e.g., 2). The other vehicles 60 are an example of a predetermined type of surrounding object. The other vehicles 60 may approach the vehicle 10.
[0027] The monitoring device 12 does not provide a warning to the driver 40 because the degree of the driver's monitoring of the surroundings does not exceed the inattention threshold.
[0028] 1(B), other vehicles 61, 62, and 63 are traveling in the area ahead of the vehicle 10. Since the number (3) of other vehicles 61, 62, and 63 detected in the area ahead of the vehicle 10 by the object detection device 11 is equal to or greater than the object number threshold (e.g., 2), the monitoring device 12 sets the inattention threshold, which is a criterion for the degree of peripheral monitoring by the driver 40 of the vehicle 10, to a second value (e.g., 2 seconds) smaller than the first value.
[0029] When a large number of other vehicles are detected in the area ahead of the vehicle 10, the driver 40 is required to pay attention to each of the other vehicles, which in turn requires a higher degree of surrounding monitoring by the driver 40.
[0030] Since the degree of the driver's surrounding monitoring has exceeded the inattention threshold, the monitoring device 12 issues a warning to the driver 40 via the user interface (UI) 7 to direct his / her attention to the area ahead of the vehicle 10. The user interface 7 is an example of a notification unit.
[0031] As described above, the monitoring device 12 can appropriately monitor the driver 40 by changing the criteria for monitoring the driver 40 depending on the surrounding conditions of the vehicle 10.
[0032] 2 is a schematic diagram of a vehicle 10 in which the monitoring system 1 of this embodiment is implemented. The vehicle 10 includes a forward camera 2, a monitoring camera 3, a LiDAR sensor 4, a vehicle speed sensor 5, a seat belt drive device 6, a user interface (UI) 7, an object detection device 11, a monitoring device 12, and the like.
[0033] The forward camera 2, surveillance camera 3, LiDAR sensor 4, vehicle speed sensor 5, seat belt drive device 6, user interface (UI) 7, object detection device 11, and surveillance device 12 are communicatively connected via an in-vehicle network 13 that complies with a standard such as a controller area network.
[0034] The front camera 2 is an example of an imaging unit provided on the vehicle 10. The front camera 2 is attached to the vehicle 10 so as to face the front of the vehicle 10. The front camera 2 captures a camera image showing the environment of a predetermined area in front of the vehicle 10 at a camera image capture time set at a predetermined cycle, for example. The camera image may show a road included in the predetermined area in front of the vehicle 10 and road features such as lane markings on the road surface. The camera image is an example of environmental information showing the environment around the vehicle 10. The front camera 2 has a two-dimensional detector composed of an array of photoelectric conversion elements sensitive to visible light, such as a CCD or C-MOS, and an imaging optical system that forms an image of the area to be captured on the two-dimensional detector. The field of view of the front camera 2 preferably includes the forward area S.
[0035] Every time the front camera 2 captures a camera image, it outputs the camera image and the time the camera image was captured to the object detection device 11 or the like via the in-vehicle network 13. The camera image is used by the object detection device 11 in a process of detecting other objects around the vehicle 10.
[0036] The surveillance camera 3 is disposed in the vehicle interior 30 so as to be able to capture surveillance images including the vicinity of the driver's seat. The surveillance images may include the face of the driver 40 who drives the vehicle 10. The surveillance camera 3 is an example of an imaging device that captures surveillance images including the face of the driver 40. The surveillance camera 3 may be disposed, for example, on a steering column, a room mirror, a meter panel, a meter hood, etc. (not shown).
[0037] The surveillance camera 3 captures a surveillance image at a surveillance image capture time that is set, for example, at a predetermined cycle. The surveillance camera 3 has a two-dimensional detector composed of an array of photoelectric conversion elements sensitive to infrared rays, such as CCD or C-MOS, and an imaging optical system that forms an image of the area to be captured on the two-dimensional detector. The surveillance camera 3 preferably has a floodlight that can irradiate the driver together with the two-dimensional detector. The floodlight is an LED (light-emitting diode), for example, two near-infrared LEDs arranged on both sides of the imaging optical system. By irradiating the driver with near-infrared light, the driver's face can be captured without causing discomfort to the driver even in low illuminance such as at night. Every time the surveillance camera 3 captures a surveillance image, it outputs the surveillance image and the surveillance image capture time to the monitoring device 12 or the like via the in-vehicle network 13.
[0038] The LiDAR sensor 4 is attached to, for example, the outer surface of the vehicle 10 so as to face the front of the vehicle 10. The LiDAR sensor 4 emits a laser to scan the front of the vehicle 10 at a reflected wave information acquisition time set at a predetermined cycle, and receives a reflected wave reflected by a reflecting object. The time required for the reflected wave to return contains distance information between the vehicle 10 and another object located in the direction of the laser irradiation. The range in which the LiDAR sensor 4 scans the front of the vehicle 10 preferably includes the forward region S. The LiDAR sensor 4 outputs reflected wave information including the laser irradiation direction and the time required for the reflected wave to return, together with the reflected wave information acquisition time when the laser was emitted, to the object detection device 11 via the in-vehicle network 13. The reflected wave information is used in the object detection device 11 for processing to detect other objects around the vehicle 10. The reflected wave information is an example of environmental information that represents the environment around the vehicle 10.
[0039] The vehicle speed sensor 5 acquires speed information representing the speed of the vehicle 10 and outputs the speed information to the monitoring device 12 via the in-vehicle network 13. The speed information represents, for example, the number of rotations of tires (not shown). The speed information is used by the monitoring device 12 to calculate the speed of the vehicle 10.
[0040] The seat belt drive device 6 is controlled by, for example, the monitoring device 12 to adjust the tension of the seat belt 32 worn by the driver 40. The monitoring device 12 can issue a warning to the driver 40 to direct his / her attention to the front of the vehicle 10 by changing the tension of the seat belt 32.
[0041] The UI7 is an example of a notification unit. The UI7 is controlled by the monitoring device 12 or the like to notify the driver 40 of a warning or the like requesting the driver to pay attention to the front of the vehicle 10. The UI7 has a display device 7a such as a liquid crystal display or a touch panel to display the warning or the like. The UI7 may also have an audio output device (not shown) to notify the driver of the warning or the like. The UI7 also generates an operation signal according to an operation from the driver to the vehicle 10. The UI7 has, for example, a touch panel or an operation button as an input device for inputting operation information from the driver to the vehicle 10. The UI7 outputs the input operation information to the monitoring device 12 or the like via the in-vehicle network 13.
[0042] The object detection device 11 detects other objects and their types in front of the vehicle 10 based on the camera image. The other objects include moving objects such as pedestrians and other vehicles moving around the vehicle 10. The object detection device 11 has, for example, a classifier that identifies a moving object represented in an image by inputting the camera image to the classifier. As the classifier, for example, a deep neural network (DNN) that is trained in advance to detect an object represented in an input image from the image can be used. The object detection device 11 may use a classifier other than a DNN. For example, the object detection device 11 may use, as the classifier, a support vector machine (SVM) that is trained in advance to input a feature amount (for example, Histograms of Oriented Gradients, HOG) calculated from a window set on an image and output a confidence that the object to be detected is represented in the window. Alternatively, the object detection device 11 may detect an object region by performing template matching between a template representing the object to be detected and the image.
[0043] The object detection device 11 may also detect other objects in front of the vehicle 10 based on the reflected wave information. The object detection device 11 may also determine the direction of the other object relative to the vehicle 10 based on the position of the other object in the camera image, and determine the distance between the other object and the vehicle 10 based on the direction and the reflected wave information output by the LiDAR sensor 4. The object detection device 11 estimates the position of the other object, for example, expressed in a vehicle coordinate system having an origin on the vehicle, based on the current position of the vehicle 10 and the distance and direction to the other object relative to the vehicle 10. The object detection device 11 outputs object detection information including information indicating the type of the detected other object (for example, pedestrian, vehicle) and information indicating its position to the monitoring device 12, etc.
[0044] The monitoring device 12 executes detection processing, setting processing, decision processing, judgment processing, warning processing, and change processing. To this end, the monitoring device 12 has a communication interface (IF) 21, a memory 22, and a processor 23. The communication interface 21, the memory 22, and the processor 23 are connected via a signal line 24. The communication interface 21 has an interface circuit for connecting the monitoring device 12 to the in-vehicle network 13.
[0045] The memory 22 is an example of a storage unit and includes, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. The memory 22 stores computer programs of applications used in the information processing executed by the processor 23 and various data.
[0046] All or part of the functions of the monitoring device 12 are functional modules realized by a computer program running on the processor 23, for example. The processor 23 has a detection unit 231, a setting unit 232, a decision unit 233, a judgment unit 234, a warning unit 235, and a change unit 236. Alternatively, the functional modules of the processor 23 may be dedicated arithmetic circuits provided in the processor 23. The processor 23 has one or more CPUs (Central Processing Units) and their peripheral circuits. The processor 23 may further have other arithmetic circuits such as a logic arithmetic unit, a numerical arithmetic unit, or a graphic processing unit. The operation of the monitoring device 12 will be described later.
[0047] The object detection device 11 and the monitoring device 12 are, for example, electronic control units (ECUs). In Fig. 2, the object detection device 11 and the monitoring device 12 are described as separate devices, but all or part of these devices may be configured as a single device.
[0048] Fig. 3 is an example of an operation flowchart relating to the monitoring process of the monitoring device 12 of this embodiment. The monitoring process of the monitoring device 12 will be described below with reference to Fig. 3. The monitoring device 12 executes the monitoring process according to the operation flowchart shown in Fig. 3 at a monitoring time having a predetermined cycle.
[0049] First, the determination unit 233 determines the degree of surroundings monitoring by the driver 40 of the vehicle 10 based on the camera image (step S101). For example, the determination unit 233 determines the degree of surroundings monitoring by the driver of the vehicle 10 so that the degree of the driver 40 not paying attention to the surroundings of the vehicle 10 is larger. The process of the determination unit 233 determining the degree of surroundings monitoring by the driver will be described later in detail.
[0050] Next, the determination unit 234 determines whether or not the degree of surroundings monitoring by the driver 40 has exceeded an inattention threshold that is a criterion for a decrease in the degree of surroundings monitoring by the driver (step S102). A threshold setting process for setting the inattention threshold will be described later.
[0051] If the degree of the driver 40's monitoring of the surroundings exceeds the inattention threshold (step S102-Yes), the warning unit 235 notifies the driver 40 of a warning requesting the driver to pay attention to the area ahead of the vehicle 10 via the UI 7 (step S103), and ends the series of processes. The warning may be displayed using the display device 7a. The warning may also be issued by voice using the audio output device of the UI 7. Furthermore, the warning unit 235 may notify the driver 40 of a warning requesting the driver to pay attention to the area ahead of the vehicle 10 by periodically changing the tension of the seat belt 32 via the seat belt drive device 6. The method of notifying the driver of the warning is not limited to the above-mentioned method, and other methods may also be used.
[0052] Here, if the driver 40 does not direct his / her gaze toward the forward area S even after the warning has been given for a predetermined period of time, the warning unit 235 may decide to stop the vehicle 10.
[0053] On the other hand, if the degree of the driver's surrounding monitoring does not exceed the inattention threshold (step S102-No), the series of processes ends.
[0054] Next, the process in which the determination unit 233 determines the degree of surroundings monitoring by the driver will be described below with reference to FIG.
[0055] FIG. 4 is a diagram illustrating the forward area S and the inattentive area N. The forward area S is expressed as a predetermined range of angles in the horizontal direction to the left and right from the direction heading forward starting from the driver's seat 31 in the vehicle interior 30. For example, this predetermined range in the horizontal direction can be a range from the right end of the front window of the vehicle 10 to 2 / 3 of the front window to the left, in the direction heading forward starting from the driver's seat 31 in the vehicle interior 30. The inattentive area N is expressed as an angle range excluding the range representing the forward area S, in the horizontal direction to the left and right, in the direction heading forward starting from the driver's seat 31 in the vehicle interior 30. The area in front of the vehicle 10 in which other objects are detected by the object detection device 11 preferably includes the forward area S.
[0056] The determination unit 233 estimates the direction of the driver's 40's gaze using a monitoring image captured by the monitoring camera 3. For example, the determination unit 233 detects the direction of the gaze using a corneal reflex method or an ocular center method based on the monitoring image. The direction of the driver's 40's gaze is expressed as a horizontal angle and a vertical angle with respect to the direction toward the front of the vehicle 10. The determination unit 233 estimates the direction of the gaze of either the left or right eye of the driver 40. Alternatively, the determination unit 233 estimates the average direction of the gaze directions of both eyes of the driver 40 as the direction of the driver's 40's gaze.
[0057] The determination unit 233 judges whether the horizontal direction of the line of sight of the driver 40 is included in the forward area S (non-aside looking area) or included in the aside looking area N. When the horizontal direction of the line of sight of the driver 40 is included in the forward area S, the driver 40 is considered to be monitoring the area ahead of the vehicle 10. On the other hand, when the horizontal direction of the line of sight of the driver 40 is included in the aside looking area N, the driver 40 is considered not to be monitoring the area ahead of the vehicle 10.
[0058] Therefore, the decision unit 233 determines that the driver 40 is not looking aside when the horizontal direction of the driver's line of sight is included in the forward area S, and determines that the driver 40 is looking aside when the horizontal direction of the driver's line of sight is included in the inattentive area N.
[0059] The determination unit 233 determines the degree of surroundings monitoring by the driver 40 based on the duration for which the driver 40 is looking aside. Specifically, when the driver 40 is looking aside, the determination unit 233 determines the duration for which the driver 40 is looking aside as the degree of surroundings monitoring by the driver 40. The determination unit 233 finds the time for which the driver 40 is continuously directing his / her gaze to the aside looking area N without looking at the forward area S. The duration for which the driver 40 is looking aside becomes longer (larger value) the greater the degree to which the driver 40 is not paying attention to the area ahead of the vehicle 10.
[0060] Furthermore, the determination unit 233 may use other methods to determine the degree of surroundings monitoring by the driver 40. For example, the determination unit 233 may determine the degree of surroundings monitoring by the driver 40 based on the number of times the driver 40 has looked aside within a predetermined time. Specifically, the determination unit 233 may determine the number of times the driver 40 has looked aside within a predetermined determination time as the degree of surroundings monitoring by the driver 40. The determination time may be, for example, 30 to 60 seconds. The number of times the driver 40 has looked aside increases as the degree to which the driver 40 has not paid attention to the road ahead of the vehicle 10 increases.
[0061] Furthermore, the determination unit 233 may determine the degree of surroundings monitoring by the driver 40 based on the accumulated time during which the driver 40 has been looking aside within a predetermined time. Specifically, the determination unit 233 may determine the accumulated time during which the driver 40 has been looking aside within a predetermined time as the degree of surroundings monitoring by the driver 40. FIG. 5 is a diagram for explaining an example of determining the degree of surroundings monitoring by the driver. The judgment time may be, for example, 30 to 60 seconds. The accumulated time during which the driver 40 has been looking aside within a predetermined time becomes a larger value as the degree to which the driver 40 has not been paying attention to the area ahead of the vehicle 10 becomes larger.
[0062] In the example shown in FIG. 5, the driver 40 looks aside for two seconds, looks forward for one second, looks aside for two seconds, looks forward for one second, and repeats this cycle. In this way, the driver 40 often glances aside. On the other hand, if the inattention threshold is set to three seconds, the driver 40's inattention cannot be detected for the duration of the driver's inattention. Therefore, in this embodiment, the cumulative time that the driver 40 looks aside within a predetermined time is determined as the degree of surroundings monitoring by the driver 40. One or more of the processes for determining the degree of surroundings monitoring by the multiple drivers described above may be used.
[0063] Next, the threshold setting process in which the monitoring device 12 sets the carelessness threshold will be described below with reference to Fig. 6. Fig. 6 is an example of an operational flowchart relating to the threshold setting process of the monitoring device of this embodiment. The monitoring device 12 executes the threshold setting process at a threshold setting time having a predetermined cycle in accordance with the operational flowchart shown in Fig. 6. It is preferable that the cycle of the threshold setting time is equal to or shorter than the cycle of the monitoring time.
[0064] First, the detection unit 231 detects a predetermined type of surrounding object present in a predetermined area around the vehicle 10 based on the object detection information (step S201). In this embodiment, the predetermined area is a forward area around the vehicle 10. The predetermined type of surrounding object is a moving object. The moving object may approach the vehicle 10. Specifically, the moving object may be another vehicle or a pedestrian. The detection unit 231 detects a moving object located in the area in front of the vehicle 10 based on the object detection information input from the object detection device 11. The detection unit 231 may have the function of the object detection device 11.
[0065] Next, the setting unit 232 determines whether the number of surrounding objects detected within a predetermined area around the vehicle 10 is less than the object number threshold (step S202). Specifically, the setting unit 232 determines whether the number of other vehicles and pedestrians detected within the area in front of the vehicle 10 is less than the object number threshold.
[0066] The object number threshold may be set according to the number of objects to which the driver 40 can direct his / her gaze with normal attention. For example, when the surrounding object is a vehicle or a pedestrian, the object number threshold may be set to 2 or 3. When there is one surrounding object, the driver 40 only needs to direct his / her attention to this surrounding object, but when there are two or more surrounding objects, it may become difficult for the driver 40 to distribute his / her gaze. In particular, when the distance between two surrounding objects becomes long, it becomes even more difficult for the driver 40 to distribute his / her gaze.
[0067] For example, when multiple peripheral objects are detected, the object number threshold may be changed according to the minimum adjacent distance between two of the multiple peripheral objects. For example, when the minimum adjacent distance is longer than a predetermined reference distance, the object number threshold may be smaller than when the minimum adjacent distance is equal to or shorter than the predetermined reference distance.
[0068] If the number of surrounding objects is less than the object number threshold (step S202-Yes), the setting unit 232 sets the inattention threshold to a first value (step S203) and ends the series of processes. On the other hand, if the number of surrounding objects is equal to or greater than the object number threshold (step S202-No), the setting unit 232 sets the inattention threshold to a second value smaller than the first value (step S203) and ends the series of processes.
[0069] For example, when the degree of surrounding monitoring by the driver 40 is the duration during which the driver 40 is looking aside, the setting unit 232 sets the inattention threshold for the duration during which the driver 40 is looking aside. For example, the first value may be set to 3 seconds, and the second value may be set to 2 seconds.
[0070] In addition, when the degree of surrounding monitoring by the driver 40 is the number of times the driver 40 has looked aside within a predetermined time, the setting unit 232 sets the inattention threshold for the number of times the driver 40 has looked aside within a predetermined time. For example, the first value may be set to four times and the second value may be set to two times for the number of times the driver 40 has looked aside within 30 seconds.
[0071] Furthermore, when the degree of surrounding monitoring by the driver 40 is the accumulated time that the driver 40 looks aside within a predetermined time, the setting unit 232 sets the inattention threshold for the accumulated time that the driver 40 looks aside within the predetermined time. For example, the first value may be set to 10 seconds and the second value may be set to 5 seconds for the accumulated time that the driver 40 looks aside within 30 seconds.
[0072] When a large number of other vehicles are detected in the area ahead of the vehicle 10, the driver 40 is required to pay attention to each of the other vehicles. Therefore, a higher degree of peripheral monitoring is required of the driver 40. Therefore, the monitoring device 12 sets an inattention threshold in accordance with the number of surrounding objects detected in a predetermined area around the vehicle 10. As a result, the inattention threshold is set so that a warning is more likely to be notified to the driver 40 when the number of surrounding objects is large than when the number of surrounding objects is small.
[0073] As described above, the monitoring device 12 can appropriately monitor the driver 40 by changing the criteria for monitoring the driver 40 depending on the surrounding conditions of the vehicle 10.
[0074] Next, a modified example of the monitoring device 12 of the above-described embodiment will be described below with reference to FIGS.
[0075] 7 is a diagram illustrating a modified example 1. In this modified example, the change unit 236 changes the inattention threshold value set by the setting unit 232 based on the relative relationship between the direction in which the surrounding object is detected by the detection unit 231 and the direction in which the driver 40 is facing.
[0076] 7, vehicle 10 is about to turn right from road 51 to road 52 at intersection 53 where roads 51 and 52 intersect. The direction P1 of driver 40's line of sight is directed toward pedestrian 64 walking on crosswalk 66 of road 52 in the direction of the right turn. The direction of driver 40's line of sight can be used as the direction P1 in which driver 40 is facing.
[0077] Here, another vehicle 65 is traveling on the road 51 from the opposite side to the vehicle 10 toward the intersection 53. Since the direction P1 in which the driver 40 faces is separated from the direction P2 in which the detection unit 231 detects the other vehicle 65 by exceeding a predetermined reference angle in the horizontal direction, the change unit 236 changes the inattention threshold so that the inattention threshold is smaller than the case in which the direction P1 in which the driver 40 faces is equal to or smaller than the predetermined reference angle with respect to the direction P2 in which the other vehicle 65 is detected. Specifically, when the inattention threshold is a first value, the change unit 236 sets the inattention threshold to a third value smaller than the first value. Also, when the inattention threshold is a second value, the change unit 236 sets the inattention threshold to a fourth value smaller than the second value.
[0078] As a result, when the direction in which a surrounding object is detected by the detection unit 231 is relatively far from the direction in which the driver 40 is facing, the monitoring device 12 is more likely to notify the driver 40 of a warning regardless of the number of surrounding objects.
[0079] 8 is an example of an operation flowchart for explaining the second variation. In the threshold setting process of this variation, the change unit 236 changes the inattention threshold set by the setting unit 232 based on the speed of the vehicle 10. Therefore, the threshold setting process of this variation is different from the above-mentioned threshold setting process in that steps S305 and S306 are added. The process of steps S301 to S304 of this variation is similar to the process of steps S101 to S104 described above.
[0080] After the inattention threshold is set to the first value or after the inattention threshold is set to the second value, the setting unit 232 determines whether the speed of the vehicle 10 is faster than a predetermined reference speed (step S305).
[0081] When the speed of the vehicle 10 is faster than the predetermined reference speed (step S305-Yes), the setting unit 232 sets the inattention threshold so that the inattention threshold is smaller than when the speed of the vehicle 10 is equal to or lower than the predetermined reference speed (step S306), and ends the series of processes. For example, the setting unit 232 sets the product of the current inattention threshold and a coefficient K (K is a positive real number equal to or less than 1.0) as the new inattention threshold.
[0082] On the other hand, if the speed of the vehicle 10 is not faster than the predetermined reference speed (step S305-No), the series of processes ends.
[0083] As a result, when the speed of the vehicle 10 is relatively high and there are many surrounding objects, the driver 40 is more likely to be notified of the warning regardless of the number of surrounding objects.
[0084] In the present disclosure, the monitoring device, the monitoring computer program, and the monitoring method of the above-mentioned embodiments can be modified as appropriate without departing from the spirit of the present disclosure. Furthermore, the technical scope of the present disclosure is not limited to those embodiments, but extends to the inventions described in the claims and their equivalents.
[0085] For example, the predetermined area around the vehicle may be an area other than the front of the vehicle. For example, when the vehicle 10 turns right, the predetermined area may be the area on the right side of the vehicle, when the vehicle 10 turns left, the predetermined area may be the area on the left side of the vehicle, and when the vehicle 10 moves backward, the predetermined area may be the area behind the vehicle.
[0086] In addition, the predetermined type of surrounding object existing within the predetermined area around the vehicle may be an object other than a moving object. For example, it may be an object whose dimensions are smaller than a predetermined reference dimension. Small objects are more likely to be overlooked than large objects, so it is useful to call the driver's attention to them more. The dimensions of the object may be estimated based on the reflected wave information.
[0087] In the above embodiment, the information representing the driver's state is a monitoring image, but information other than an image may be used. For example, the driver's posture may be estimated using a pressure sensor or the like disposed in the driver's seat, and the degree of monitoring of the driver's surroundings may be determined based on the driver's posture.
[0088] The determination unit may determine the degree of surroundings monitoring by the driver of the vehicle so that the degree of the surroundings monitoring by the driver is greater, and the warning unit issues a warning to the driver when the determination unit determines that the degree of surroundings monitoring by the driver is less than the inattention threshold.
[0089] The monitoring device may also monitor a state of the driver other than inattentive driving. For example, the monitoring device may monitor the degree of eye opening or the degree of eye opening, which is an index of the degree of involvement of the driver in driving. [Explanation of symbols]
[0090] 1. Surveillance System 2. Front camera 3. Surveillance Cameras 4 LiDAR Sensor 5 Vehicle speed sensor 6 Seat belt drive mechanism 7 User Interface 7a Display device 10 Vehicles 11 Object detection device 12 Monitoring equipment 21 Communication Interface 22 Memory 23 Processors 231 Detection unit 232 Setting section 233 Decision Section 234 Judgment section 235 Warning section 236 Changes 13 In-vehicle network 30 Cabin 31 Driver's seat 32 Seatbelt 40 Drivers
Claims
1. a detection unit that detects a predetermined type of surrounding object present within a predetermined area around the vehicle based on environmental information that represents an environment around the vehicle; a determination unit for determining an extent to which a driver is not paying attention to the surroundings of the vehicle based on information representative of a state of the driver; a setting unit that sets an inattention threshold, which is a criterion of the degree to which the driver is not paying attention to the surroundings of the vehicle, to a first value when the number of the surrounding objects detected within the predetermined area by the detection unit is less than an object number threshold, and sets the inattention threshold to a second value smaller than the first value when the number of the surrounding objects is equal to or greater than the object number threshold; a determination unit that determines whether the degree to which the driver is not paying attention to the surroundings of the vehicle determined by the determination unit exceeds the inattention threshold; a warning unit that notifies the driver via a notification unit of a warning to pay attention to a surrounding of the vehicle when the determination unit determines that the degree of the driver not paying attention to a surrounding of the vehicle exceeds the inattention threshold; A monitoring device comprising:
2. The determination unit determines the degree to which the driver is not paying attention to the surroundings of the vehicle such that the degree has a larger value as the degree to which the driver is not paying attention to the surroundings of the vehicle increases, The monitoring device according to claim 1 , wherein the warning unit issues the warning to the driver when the determination unit determines that the degree to which the driver is not paying attention to the surroundings of the vehicle is greater than the inattention threshold.
3. The monitoring device of claim 2 , wherein the determining unit determines the extent to which the driver is not paying attention to the surroundings of the vehicle based on a duration of time the driver is looking aside.
4. The monitoring device according to claim 2 , wherein the determining unit determines the degree to which the driver is not paying attention to the surroundings of the vehicle based on the number of times the driver has looked aside within a predetermined time period.
5. The monitoring device according to claim 2 , wherein the determining unit determines the degree to which the driver is not paying attention to the surroundings of the vehicle based on an accumulated time during which the driver has been looking aside within a predetermined time.
6. The determination unit determines the degree to which the driver is not paying attention to the surroundings of the vehicle such that the degree has a larger value as the degree to which the driver is not paying attention to the surroundings of the vehicle increases, The monitoring device according to claim 1 , wherein the warning unit issues the warning to the driver when the determination unit determines that the degree to which the driver is not paying attention to the surroundings of the vehicle is less than the inattention threshold.
7. The monitoring device according to claim 1 , further comprising a first change unit that changes the inattention threshold set by the setting unit based on a relative relationship between a direction in which the surrounding object is detected by the detection unit and a direction in which the driver is facing.
8. 7. The monitoring device according to claim 1, further comprising a second change unit that changes the inattention threshold value set by the setting unit based on a speed of the vehicle.
9. Detecting a predetermined type of surrounding object present within a predetermined area around the vehicle based on environmental information representing an environment around the vehicle; determining an extent to which the driver is not paying attention to a surrounding of the vehicle based on information representative of the driver's state; When the number of the surrounding objects detected within the predetermined area is less than an object number threshold, an inattention threshold serving as a criterion of the degree to which the driver is not paying attention to the surroundings of the vehicle is set to a first value, and when the number of the surrounding objects is equal to or greater than the object number threshold, the inattention threshold is set to a second value smaller than the first value; determining whether the degree to which the driver is not paying attention to the surroundings of the vehicle exceeds the inattention threshold; A monitoring computer program that causes a processor to execute a process that includes, when it is determined that the degree to which the driver is not paying attention to the surroundings of the vehicle exceeds the inattention threshold, notifying the driver via a notification unit of a warning to pay attention to the surroundings of the vehicle.
10. 1. A monitoring method performed by an alert device, comprising: Detecting a predetermined type of surrounding object present within a predetermined area around the vehicle based on environmental information representing an environment around the vehicle; determining an extent to which the driver is not paying attention to a surrounding of the vehicle based on information representative of the driver's state; When the number of the surrounding objects detected within the predetermined area is less than an object number threshold, an inattention threshold serving as a criterion of the degree to which the driver is not paying attention to the surroundings of the vehicle is set to a first value, and when the number of the surrounding objects is equal to or greater than the object number threshold, the inattention threshold is set to a second value smaller than the first value; determining whether the degree to which the driver is not paying attention to the surroundings of the vehicle exceeds the inattention threshold; A monitoring method comprising: when it is determined that the degree to which the driver is not paying attention to the surroundings of the vehicle exceeds the inattention threshold, issuing a warning to the driver via a notification unit to pay attention to the surroundings of the vehicle.
Citation Information
Patent Citations
Device for operating degree of risk of driving behavior
JP2003099899A
Driving support system, on-vehicle device, server and driving support method
JP2013065246A
Travel support device
JP2018067198A
Inattentiveness detecting device, inattentiveness detecting method, and computer program
WO2010032491A1
