Driving monitoring method and driving monitoring device

Abstract

Provided is a driving monitoring method for monitoring the posture of a driver who drives a vehicle. A driving monitoring method comprises: a detection process for detecting the driving posture of a driver; an acquisition process for acquiring driving environment information that includes an environment around a vehicle and the driving status of the driver; an assessment process for assessing, on the basis of the driving posture of the driver and the driving environment information, whether the driving posture of the driver has collapsed; and a notification process for notifying the driver regarding the collapse of the driving posture of the driver in cases where it is assessed that the driving posture of the driver has collapsed.

Description

Driving Monitoring Method and Driving Monitoring Device

[0004]

[0001] The present invention relates to a driving monitoring method and a driving monitoring device for monitoring the posture of a driver who drives a vehicle.

[0002] Conventionally, technologies for monitoring the posture of a driver during vehicle driving have been proposed. For example, in JP2016 - 038793A, a technology is proposed in which a breakdown of the driver's posture is detected, and depending on whether the detected breakdown of the posture is due to the driver's habit or a breakdown of the posture other than the habit, a notification regarding the breakdown of the posture is given to the driver in different manners.

[0003] In the above - mentioned conventional technology, when the amount of breakdown of the posture increases with the passage of driving time and exceeds a threshold value, etc., it is determined as a breakdown of the posture other than the habit. On the other hand, when the amount of breakdown of the posture has exceeded the threshold value continuously for a longer time than a predetermined time immediately after the start of driving, it is determined as a breakdown of the posture due to the habit. In order to enhance the safety during vehicle driving, it is important to return the driver's posture during vehicle driving to an appropriate posture at an early stage. Therefore, it is conceivable to set the threshold value and the predetermined time used for the determination to small values to detect the breakdown of the posture at an early stage. However, when the threshold value and the predetermined time used for the determination are set to small values, there is a risk of false - detecting the breakdown of the posture. In this case, an inappropriate notification may be given to the driver, which may give the driver an unpleasant impression.

[0004] An object of the present invention is to provide a driving monitoring technology capable of more accurately detecting the driver's posture at an appropriate timing according to the driving environment of the vehicle.

[0005] One aspect of the present invention is a driving monitoring method for monitoring the posture of a driver who drives a vehicle. This driving monitoring method includes a detection process for detecting the driving posture of the driver, an acquisition process for acquiring driving environment information including the environment around the vehicle and the driving situation of the driver, a determination process for determining a breakdown of the driving posture of the driver based on the driving posture of the driver and the driving environment information, and a notification process for notifying the driver to that effect when a breakdown of the driving posture of the driver is determined.

[0006] Figure 1 is a block diagram showing an example of the system configuration of a driving monitoring system. Figure 2 is a diagram showing changes in the driver's driving posture. Figure 3 is a diagram showing a method for determining whether the driver's driving posture has deteriorated. Figure 4 is a diagram showing a method for determining whether the driver's driving posture has deteriorated. Figure 5 is a flowchart showing an example of the driving monitoring process.

[0007] Embodiments of the present invention will be described below with reference to the attached drawings.

[0008] [Example of Driving Monitoring System Configuration] Figure 1 is a block diagram showing an example of the system configuration of the driving monitoring system 100 installed in vehicle C1.

[0009] The driving monitoring system 100 includes an in-vehicle image acquisition unit 101, an out-of-vehicle image acquisition unit 102, a vehicle information acquisition unit 103, a driving monitoring device 110, a display unit 210, and a sound output unit 220. The driving monitoring device 110 is an example of a device that monitors the posture of the driver D1 (see Figure 2) operating the vehicle C1. The driving monitoring device 110 may also be referred to as a driver posture monitoring device, etc.

[0010] The in-vehicle image acquisition unit 101, the exterior image acquisition unit 102, the vehicle information acquisition unit 103, the driving monitoring device 110, the display unit 210, and the sound output unit 220 are each connected by a communication method using wired or wireless communication. The driving monitoring device 110 is also connected to the network 20 by a communication method using wired or wireless communication. The network 20 is a network such as a public telephone network or the Internet. The display unit 210, the sound output unit 220, etc. may also be connected to the network 20 by a communication method using wired or wireless communication.

[0011] The in-vehicle image acquisition unit 101 captures images (image data) of subjects inside the vehicle C1 and outputs image information (in-vehicle image) related to the generated image to the driving monitoring device 110. In this embodiment, the in-vehicle image acquisition unit 101 generates an in-vehicle image used to detect the driving posture of driver D1 sitting in the driver's seat S1 (see Figure 2) of vehicle C1. For this reason, the in-vehicle image acquisition unit 101 is installed in a position that captures the driving posture of driver D1. For example, as shown in Figure 2, when detecting the driving posture of driver D1 as viewed from the side (left and right direction of vehicle C1), the in-vehicle image acquisition unit 101 is installed in a position that captures the side of driver D1 (left and right direction of vehicle C1). One or more in-vehicle image acquisition units 101 may be installed in vehicle C1. For example, one in-vehicle image acquisition unit 101 may be installed in front of the vehicle C1 to capture images of the driver D1 from the front and generate an image, or another in-vehicle image acquisition unit 101 may be installed in rear of the vehicle C1 to capture images of the driver D from the rear and generate an image.

[0012] The external image acquisition unit 102 captures images of subjects outside the vehicle C1 and generates images (image data), and outputs image information related to the generated images to the driving monitoring device 110. Note that two or more external image acquisition units 102 may be provided, and images from all or some of these external image acquisition units 102 may be used. For example, one external image acquisition unit 102 may be installed in front of the vehicle C1 to capture images of subjects in front of the vehicle C1 and generate images, or another external image acquisition unit 102 may be installed behind the vehicle C1 to capture images of subjects behind the vehicle C1 and generate images. Alternatively, one or more devices capable of acquiring images of subjects present in all directions around the vehicle C1 and subjects inside the vehicle C1, such as a 360-degree camera, may be used. Note that the interior image acquisition unit 101 and the external image acquisition unit 102 may be the same device or configured as different devices.

[0013] The in-vehicle image acquisition unit 101 and the out-of-vehicle image acquisition unit 102 are each composed of, for example, an image sensor that receives light from a subject focused by a lens, and an image processing unit that performs predetermined image processing on the image data generated by the image sensor. For example, a CCD (Charge Coupled Device) type or a CMOS (Complementary Metal Oxide Semiconductor) type image sensor can be used as the image sensor.

[0014] The vehicle information acquisition unit 103 acquires various information regarding the vehicle status of vehicle C1 (vehicle information) and outputs the acquired vehicle information to the driving monitoring device 110. Vehicle information includes, for example, information obtainable from CAN (Controller Area Network) signals and information obtainable from various sensors installed on vehicle C1. For example, vehicle information includes vehicle speed, acceleration, shift lever position (e.g., P range, D range), accelerator pedal depression amount, brake pedal depression amount, whether or not the turn signal is operated, steering wheel SW1 (see Figure 2) operation amount, and vehicle C1 position information. For example, the vehicle speed, acceleration, shift lever position, etc., can be used to determine whether vehicle C1 is stopped, traveling, moving forward, moving backward, and its speed.

[0015] Furthermore, the sensors include, for example, LiDAR (Light Detection and Ranging), RADAR (Radio Detection and Ranging), Sonar, vehicle speed sensor, acceleration sensor, steering sensor (steering force angle meter), accelerator position sensor, position information acquisition sensor (position information acquisition unit), illuminance sensor, raindrop sensor, etc. It is possible to use known sensors for each of these. Note that LiDAR, RADAR, Sonar, illuminance sensor, raindrop sensor, etc. are examples of sensors that detect the environment around vehicle C1. Also, the vehicle speed sensor, acceleration sensor, steering sensor, accelerator position sensor, etc. are examples of sensors that detect the driving operation status (driving conditions) of driver D1. Note that these are just examples, and other sensors may be used. Furthermore, only some of these sensors may be used. In this way, the vehicle information acquisition unit 103 functions as a driver operation information acquisition unit that acquires operation information related to the driving operations of driver D1. Furthermore, the vehicle information acquisition unit 103 functions as an acquisition unit that acquires driving environment information, including the environment surrounding the vehicle C1 and the driving conditions of the driver D1.

[0016] The location information acquisition unit acquires location information regarding the location of vehicle C1. For example, this can be implemented using a GNSS (Global Navigation Satellite System) receiver that acquires location information using GNSS. This location information includes various positional data such as latitude, longitude, and altitude at the time of receiving the GNSS signal. Alternatively, location information may be acquired using other methods. For example, location information may be derived using information from nearby access points or base stations. Alternatively, location information may be acquired using beacons.

[0017] The driving monitoring device 110 comprises a control unit 120, a storage unit 130, and a communication unit 140. The communication unit 140 exchanges various types of information with other devices using wired or wireless communication based on the control of the control unit 120. For example, when the communication unit 140 receives weather information regarding the weather around vehicle C1, traffic congestion information regarding the roads around vehicle C1, etc., from an external device (e.g., a server), it outputs each of these pieces of information to the control unit 120. Also, for example, if a change in the driving posture of driver D1 is detected, the communication unit 140 transmits notification information to the electronic device 300 to notify it of this fact.

[0018] The control unit 120 controls each part based on various programs stored in the memory unit 130. The control unit 120 is implemented by a processing unit such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). The vehicle ECU (Electronic Control Unit) of the vehicle C1 may be used as the control unit 120, or a processing unit different from the vehicle ECU may be provided as the control unit 120.

[0019] The control unit 120 performs various controls based on the information output from the in-vehicle image acquisition unit 101, the out-of-vehicle image acquisition unit 102, the vehicle information acquisition unit 103, the communication unit 140, etc. For example, the control unit 120 performs a detection process to detect the driving posture of driver D1, an acquisition process to acquire driving environment information including the environment around the vehicle C1 and the driving status of driver D1, a determination process to determine if driver D1's driving posture is unstable based on driver D1's driving posture and the driving environment information, and a notification process to notify driver D1 if an instability in driver D1's driving posture is determined. The control unit 120 also performs control processes to control the operating state of the display unit 210 and the sound output unit 220. Specifically, the control unit 120 comprises a detection unit 121, a determination unit 122, and a notification control unit 123.

[0020] The detection unit 121 detects the driving posture of driver D1 while the vehicle C1 is in operation, based on the in-vehicle image acquired by the in-vehicle image acquisition unit 101. The detection unit 121 then outputs the detection result (driver D1's driving posture) to the determination unit 122. In this embodiment, an example is shown in which the amount of change in a specific part of driver D1 (for example, the angle made by the right foot) is detected as an indicator of driver D1's driving posture while the vehicle C1 is in operation. The detection method for detecting driver D1's driving posture will be explained in detail with reference to Figure 2.

[0021] The determination unit 122 determines whether driver D1's driving posture is compromised based on the detection result (driver D1's driving posture) from the detection unit 121 and the driving environment information. The determination unit 122 acquires driving environment information based on the information output from the external image acquisition unit 102, the vehicle information acquisition unit 103, the communication unit 140, etc. The determination unit 122 then outputs its determination result (compromise of driver D1's driving posture) to the notification control unit 123. The method for acquiring driving environment information will be explained in detail with reference to Figure 3.

[0022] For example, if the driving environment for vehicle C1 is a normal driving environment, the determination unit 122 can determine that the driving posture of driver D1 has deteriorated if the amount of change in a specific part of driver D1 detected by the detection unit 121 exceeds the threshold TH1 (see Figure 3) for a determination time JT1 (see Figure 3) or longer. Also, for example, if the driving environment for vehicle C1 is a predetermined driving environment, the determination unit 122 can change at least one of the threshold TH1 and the determination time JT1 to execute the above-described determination process. Also, for example, if the driving environment for vehicle C1 is a specific driving environment, the determination unit 122 can decide not to execute the above-described determination process. These determination methods will be explained in detail with reference to Figures 3 and 4.

[0023] Here, a predetermined driving environment refers to a driving environment in which the judgment criteria need to be changed compared to normal driving. A predetermined driving environment is, for example, a driving environment in which posture instability is less tolerable than during normal driving, or a driving environment in which posture instability is tolerable to a certain extent. In a driving environment in which posture instability is less tolerable than during normal driving, the judgment unit 122 executes a judgment process with stricter judgment criteria. On the other hand, in a driving environment in which posture instability is tolerable to a certain extent, the judgment unit 122 executes a judgment process with relaxed judgment criteria. This makes it possible to execute an appropriate judgment process according to the driving environment. A detailed explanation of the predetermined driving environment, the threshold when changing the judgment criteria, and the method of changing the judgment time will be explained in detail with reference to Figure 3.

[0024] Furthermore, a specific driving environment refers to a driving environment in which the process of determining vehicle posture instability is unnecessary. A specific driving environment is, for example, an environment in which driver D1 frequently performs actions that differ from normal driving, such as turning at an intersection or reversing. A detailed explanation of specific driving environments will be provided in detail in Figure 4.

[0025] The notification control unit 123, when the determination unit 122 determines that the driver D1's driving posture has deteriorated, executes a notification process to notify the driver D1 of this fact using the notification information stored in the notification information DB 132. Specifically, the notification control unit 123 controls the operating status of the display unit 210 and the sound output unit 220. The notification method will be explained in detail with reference to Figure 5. In this way, the notification control unit 123 functions as a driving posture improvement notification unit that executes a notification process to improve the driving posture.

[0026] The memory unit 130 is a storage medium for storing various types of information. For example, the memory unit 130 stores various types of information necessary for the control unit 120 to perform various processes (e.g., control programs, map information DB 131, notification information DB 132). The memory unit 130 also stores various types of information acquired via the communication unit 140. As the memory unit 130, for example, ROM (Read Only Memory), RAM (Random Access Memory), SRAM (Static Random Access Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), or a combination thereof can be used.

[0027] The map information DB131 stores map information such as road information related to roads that vehicle C1 can travel on. This map information includes information about roads such as expressways, general roads, and intersections.

[0028] The notification information DB 132 stores image information, audio information, etc., used to notify driver D1 when a deterioration in the driver's driving posture is detected. For example, the part of the body where the deterioration in driving posture is detected (e.g., the neck), related audio information (e.g., "Your neck is hunched over. Please straighten it."), and a corresponding image display are stored in association with each other in the notification information DB 132. Note that the map information DB 131 and notification information DB 132 may be stored in the storage unit 130 of the vehicle C1, or they may be acquired from external devices via the network 20.

[0029] The display unit 210 and the sound output unit 220 are devices capable of displaying various images and outputting various audio information based on instructions from the driving monitoring device 110, thereby conveying various information to the driver D1, etc. For example, at least one of the following can be used as the display unit 210 and the sound output unit 220: a navigation system, an audio system, an IVI (In-Vehicle Infotainment), etc. These are merely examples, and other devices installed in the vehicle C1 may also be used.

[0030] The display unit 210 displays various images based on instructions from the operation monitoring device 110. For example, the display unit 210 can be an OLED (Electro-Luminescence) panel, an LCD (Liquid Crystal Display) panel, or other display panel. The display unit 210 may be configured as a touch panel that allows the user to perform operation input by touching or bringing their finger close to the display surface, or it may be configured as a separate user interface.

[0031] The sound output unit 220 outputs various sounds based on instructions from the operation monitoring device 110. For example, one or more speakers can be used as the sound output unit 220. Furthermore, the display unit 210 and the sound output unit 220 are examples of a user interface, and some of them may be omitted, or other user interfaces may be used.

[0032] The electronic device 300 is a device that can transmit various information to the driver D1, etc., by displaying various images or outputting various audio information based on instructions from the driving monitoring device 110. The electronic device 300 can be implemented as, for example, an electronic device such as a smartphone, tablet terminal, or portable personal computer, or an information processing device. Alternatively, a device fixedly installed inside the vehicle C1 may be used as the electronic device 300.

[0033] For example, the electronic device 300 can be connected to the network 20 by a communication method using wired communication or wireless communication. Furthermore, the electronic device 300 and the operation monitoring device 110 are connected by a communication method using wired communication or wireless communication. For example, the electronic device 300 and the operation monitoring device 110 are connected via the network 20 using a communication method utilizing wireless communication. Alternatively, for example, the electronic device 300 and the operation monitoring device 110 are directly connected using a communication method utilizing wireless communication without going through the network 20. Various wireless communication methods can be used for this, such as wireless LAN (Local Area Network) (e.g., Wi-Fi (Wireless Fidelity)), Bluetooth®, etc.

[0034] [Example of Posture Change] Figure 2 shows a change in the driving posture of driver D1 sitting in the driver's seat S1 of vehicle C1. Figure 2 shows an example in which the detection unit 121 analyzes and detects the posture, movements, etc. of driver D1 using known skeletal detection technology. Here, skeletal detection technology refers to a technology that identifies the joint positions of a person based on an image that includes all or part of that person, and analyzes the posture, movements, etc. of that person. In other words, by using skeletal detection technology, it is possible to identify the head, neck, shoulders, elbows, hands, buttocks, knees, feet, etc. of the target person. Alternatively, for example, skeletal detection technology using artificial intelligence (AI) may be employed.

[0035] Furthermore, Figure 2 shows an example where the in-vehicle image acquisition unit 101 is installed in a position that captures the right side of the driver D1 who is sitting in the driver's seat S1 and gripping the steering wheel SW1 with both hands. In other words, it shows an example where an image taken from the right side of the driver D1 sitting in the driver's seat S1 is used.

[0036] Figure 2 also shows an example using the angles θ1 to θ3 formed by two line segments L1 and L2 that identify the right foot of driver D1. Note that the two line segments L1 and L2 correspond to the femur, tibia, and fibula connected to the knee joint.

[0037] Figure 2(b) shows an example of the basic posture of driver D1 sitting in the driver's seat S1. Figure 2(a) shows an example where driver D1's body is tilted forward, compared to the basic posture shown in Figure 2(b). Figure 2(c) shows an example where driver D1's body is tilted backward, compared to the basic posture shown in Figure 2(b).

[0038] As shown in Figure 2(a), when driver D1 adopts a forward-leaning posture relative to the basic posture, the angle θ2 formed by the two line segments L1 and L2 becomes smaller than the angle θ1 (see Figure 2(b)). Also, as shown in Figure 2(c), when driver D1 adopts a backward-leaning posture relative to the basic posture, the angle θ3 formed by the two line segments L1 and L2 becomes larger than the angle θ1. In this way, when driver D1's driving posture deviates from the basic posture and becomes either forward-leaning or backward-leaning, a specific part of driver D1 (right foot) changes. Therefore, in this embodiment, we show an example of using the amount of change (e.g., θ1 - θ2) of a specific part of driver D1 (e.g., right foot) as an indicator of driver D1's driving posture. Then, based on whether the amount of change (e.g., θ1 - θ2) of a specific part of driver D1 (e.g., right foot) exceeds a threshold for a judgment period of time, it is determined whether driver D1's driving posture has deteriorated. This judgment example will be explained in detail with reference to Figures 3 and 4.

[0039] Figure 2 shows an example where the change in driver D1's right foot (angle of change) is used as an indicator of driver D1's driving posture, but it is not limited to this. For example, the change in other parts of driver D1 may be used. For example, as shown in Figures 2(a) to (c), if driver D1's basic driving posture is disrupted and becomes a forward-leaning or backward-leaning posture, the position of driver D1's arms changes. Therefore, for example, the change in driver D1's arms (for example, the change in the angle between the two line segments connected to the elbow joint) can be used as an indicator of driver D1's driving posture. Also, for example, as shown in Figures 2(a) to (c), if driver D1's basic driving posture is disrupted and becomes a forward-leaning or backward-leaning posture, the position of driver D1's face (head) changes. Therefore, for example, the change in driver D1's face (head) can be used as an indicator of driver D1's driving posture. In this case, the forward, backward, left, and right movement of driver D1's face (head) can be used. Furthermore, as shown in Figures 2(a) to 2(c), for example, if the driver D1's basic driving posture is disrupted and they adopt a forward-leaning or backward-leaning posture, the position of the driver D1's body changes. Therefore, for example, the amount of change in the driver D1's body can be used as an indicator of the driver D1's driving posture. In this case, the amount of forward, backward, left-right movement of the driver D1's body can be used.

[0040] Note that these specific body parts are just examples, and changes in other body parts may also be used. For example, the angles between the body and feet, the angles between the body and arms, the position of the neck, the position of the waist, the angle of flexion of the ankles, and other body parts and their changes can be used. Furthermore, some or all of these may be used.

[0041] Furthermore, while the above example demonstrates how to determine the amount of change in a specific part of driver D1 using skeletal detection technology, the method is not limited to this. For example, since it is possible to detect the amount of movement of each part included in the in-vehicle image generated by the in-vehicle image acquisition unit 101, it is possible to use the amount of movement in that in-vehicle image as the amount of change in a specific part of driver D1.

[0042] [Example of a method for determining a driver's driving posture] Figure 3 shows an example of a method for determining a driver D1's driving posture. The vertical axis in Figure 3 shows the amount of change in driver D1's driving posture. The horizontal axis in Figure 3 shows the time axis.

[0043] Here, we show an example where a specific body part (for example, the angle θ1 to θ3 of the right foot) as described in Figure 2 is used as an indicator of the driver's posture. Furthermore, in the graph shown in Figure 3, we show an example where the value when driver D1 is in the basic posture, sitting in the driver's seat S1 and gripping the steering wheel SW1 with both hands, is taken as the reference value BP1 (for example, θ1). In this case, if driver D1's posture changes from the basic posture, the change amount PC1 is the angle of change from the reference value BP1 of the specific body part corresponding to that change (for example, θ1 - θ2). Note that in Figure 3, for the sake of simplicity, only the change amount in one direction (for example, the direction of angle θ2) (i.e., θ1 - θ2) when the angle θ1 of the right foot is taken as the basic posture is shown as an example. However, the same can be done when using the change amount in another direction (for example, the direction of angle θ3) (i.e., θ1 - θ3) when the angle θ1 of the right foot is taken as the basic posture. In this case, the change in driver D1's driving posture from the basic posture may be expressed as the difference value (positive or negative value) when the basic posture is set to 0, or as the absolute value of that difference value. When using the difference value as the amount of change, it is possible to set both a positive threshold and a negative threshold. When using the absolute value of the difference value as the amount of change, it is possible to set only a positive threshold.

[0044] Here, a setting example for setting the optimal position (or angle) of each part in the case where the driver D1 sitting in the driver's seat S1 and holding the steering wheel SW1 with both hands is in the basic posture will be described. For example, based on an in-vehicle image obtained by imaging the driver D1 driving the vehicle C1 for a predetermined time, it is possible to set the basic posture of the driver D1 and the optimal position (or angle) of each part corresponding to the basic posture. For example, among the driving postures of the driver D1 in a state that is not a predetermined driving environment (or a specific driving environment) described later, a posture that continues for a predetermined time can be set as the basic posture. Also, for example, based on experiments, simulations, etc. using a plurality of people, it is possible to set the basic postures of people of each height, people of each body type, etc., and the optimal position (or angle) of each part corresponding to the basic posture. In this case, based on the set values (numerical values such as height, body type, etc.) corresponding to the driver D1, the basic posture of the driver D1 can be set. Also, for example, the basic posture of the driver D1 may be set using artificial intelligence (AI). For example, the full-body images of a large number of people getting into the driver's seat S1 and the driving postures of each of those people are learned in advance, and using this learning data, the basic posture of the driver D1 can be estimated from the full-body image of the driver D1 getting into the driver's seat S1.

[0045] Also, in FIG. 3, an example of executing determination processing using threshold values TH1, TH2 and determination times JT1, JT2 is shown.

[0046] Thresholds TH1 and TH2 are used to determine if the driver D1's driving posture has deteriorated. Here, threshold TH1 is the threshold used under normal conditions. Threshold TH2 is a threshold set when the driving environment for vehicle C1 reaches a predetermined condition, and is an example of a threshold with stricter judgment criteria. In Figure 3, only two thresholds, TH1 and TH2, are shown as examples, but thresholds other than TH1 and TH2 may be set based on driving environment information for vehicle C1. For example, two or more threshold levels may be set based on driving environment information for vehicle C1. Also, in Figure 3, an example is shown where threshold TH2 is set to be smaller than threshold TH1, which is used under normal conditions, but a threshold larger than threshold TH1 may be set based on driving environment information for vehicle C1. In other words, it is possible to set thresholds with relaxed judgment criteria.

[0047] Judgment times JT1 and JT2 are used to determine if the driver D1's driving posture has deteriorated. Here, judgment time JT1 is the judgment time used under normal circumstances. Judgment time JT2 is set when the driving environment for vehicle C1 meets a predetermined driving environment, and is an example of a judgment time with stricter criteria. In Figure 3, only two judgment times JT1 and JT2 are shown as examples, but judgment times other than JT1 and JT2 may be set based on driving environment information for vehicle C1. For example, two or more judgment times may be set based on driving environment information for vehicle C1. Also, in Figure 3, an example is shown in which judgment time JT2 is set to be shorter than judgment time JT1, which is used under normal circumstances, but a judgment time longer than judgment time JT1 may be set based on driving environment information for vehicle C1. In other words, it is possible to set a judgment time with relaxed criteria. In Figure 3, an example is shown in which judgment is made using a threshold and judgment time, but judgment may also be made using only a threshold. In other words, the judgment time may be set to 0.

[0048] Here, the driving environment information includes environment information about the environment around the vehicle C1, driving situation information about the driving situation of the driver D1, and the like. The environment information includes, for example, road information about the road on which the vehicle C1 travels (for example, highway, general road, in traffic jam), weather information about the weather around the vehicle C1 (for example, good weather such as sunny, bad weather such as rain, snow), the time zone during which the vehicle C1 is traveling (for example, night, daytime), and the like. For example, the road information (for example, highway, general road) can be obtained based on the current location of the vehicle C1 acquired by the location information acquisition unit and the map information stored in the map information DB131. Also, for example, the road information (for example, in traffic jam) can be obtained from an external device (for example, traffic information providing server) via the communication unit 140. Also, for example, the weather information can be obtained based on various sensors (for example, illuminance sensor, raindrop sensor). Or, it can be obtained from an external device (for example, weather information providing server) via the communication unit 140.

[0049] For example, when the driving environment regarding the vehicle C1 is a normal driving environment, the determination unit 122 executes a determination process using the threshold value TH1 and the determination time JT1. Specifically, the determination unit 122 determines whether the change amount PC1 of the driving posture of the driver D1 detected by the detection unit 121 exceeds the threshold value TH1. When the change amount PC1 exceeds the threshold value TH1, the determination unit 122 determines whether the state in which the change amount PC1 exceeds the threshold value TH1 continues for a determination time JT1 or more. And when the state in which the change amount PC1 exceeds the threshold value TH1 continues for a determination time JT1 or more, the determination unit 122 determines that the driving posture of the driver D1 has collapsed. For example, it is determined that the driving posture of the driver D1 has collapsed at the timing of the time DT1. On the other hand, when the change amount PC1 of the driving posture of the driver D1 is less than the threshold value TH1, or when the state in which the change amount PC1 exceeds the threshold value TH1 does not continue for a determination time JT1 or more, the determination unit 122 determines that the driving posture of the driver D1 has not collapsed.

[0050] Here, let's assume that vehicle C1 is traveling on a highway. In this case, it is known that the faster vehicle C1 is traveling, the narrower the driver D1's field of vision becomes. Therefore, it is important to detect any deterioration in driver D1's driving posture early by changing the threshold TH1 and judgment time JT1 to smaller values ​​as the vehicle C1's speed increases. On the other hand, in the case of low-speed driving (for example, during traffic congestion), it is thought that driver D1's movements are often larger. Therefore, it is conceivable to change the threshold TH1 and judgment time JT1 to larger values ​​as the vehicle C1's speed decreases. In particular, if there is traffic congestion directly in front of vehicle C1, it is conceivable to either execute a change process that significantly alters the threshold TH1 and judgment time JT1, or to not execute the judgment process at all.

[0051] Furthermore, for example, if the automatic driving mode is set in vehicle C1, driver D1 does not need to perform any driving operations and can relax, so it is likely that driver D1's posture will change. For example, driver D1 may stretch their arms or legs. Thus, it is likely that driver D1's movements will be large. Therefore, if the automatic driving mode is set in vehicle C1, it is possible to either execute a modification process that significantly changes the threshold TH1 and the judgment time JT1, or not execute the judgment process at all. Also, for example, if driver D1 is operating the turn signal, steering wheel SW1, etc., it is likely that driver D1's movements will be large in the time period before and after such operation. Therefore, if driver D1 is operating the turn signal, steering wheel SW1, etc., it is possible to either execute a modification process that significantly changes the threshold TH1 and the judgment time JT1, or not execute the judgment process at all, for a certain period of time before and after such operation.

[0052] Furthermore, consider the case where vehicle C1 is traveling on an ordinary road. In this case, similar to a highway, if driver D1 is operating, for example, the turn signal, steering wheel SW1, etc., it is likely that driver D1's movements will be larger in the time period before and after such operation. Therefore, when driver D1 is operating the turn signal, steering wheel SW1, etc., it is conceivable to either execute a modification process that significantly changes the threshold TH1 and judgment time JT1, or not execute the judgment process at all, for a certain period before and after such operation. Also, consider the case where vehicle C1 enters an intersection and turns at that intersection. In this case, since driver D1 is performing a turning operation, for example, operating the turn signal or steering wheel SW1, it is likely that driver D1's movements will be larger in the time period before and after such operation. Therefore, when vehicle C1 enters an intersection and turns at that intersection, it is conceivable to either execute a modification process that significantly changes the threshold TH1 and judgment time JT1, or not execute the judgment process at all, for a certain period before and after such operation.

[0053] Thus, for example, if the driving environment for vehicle C1 is a predetermined driving environment, the determination unit 122 changes at least one of the threshold TH1 and the determination time JT1 and executes the determination process. Specifically, the determination unit 122 determines whether the driving environment for vehicle C1 is a predetermined driving environment based on the information output from the external image acquisition unit 102, the vehicle information acquisition unit 103, the communication unit 140, etc. If the driving environment for vehicle C1 is a predetermined driving environment, the determination unit 122 changes at least one of the threshold TH1 and the determination time JT1 according to the predetermined driving environment. On the other hand, if the driving environment for vehicle C1 is a normal driving environment, the determination unit 122 executes the determination process using the threshold TH1 and the determination time JT1.

[0054] Here, a predetermined driving environment means when the vehicle speed of vehicle C1 is above a predetermined value, when vehicle C1 is traveling on a highway, when the surroundings of vehicle C1 are in bad weather (e.g., rain, snow, strong wind), when vehicle C1 is traveling at night, when the turn signal of vehicle C1 is activated, when vehicle C1 is turning, when vehicle C1 is reversing, when there is traffic congestion ahead of vehicle C1, when automatic driving mode is set in vehicle C1, etc. Note that when vehicle C1 is turning, for example, when the steering wheel SW1 of vehicle C1 is activated, when vehicle C1 is turning at an intersection, etc. Note that these are just examples of predetermined driving environments, and other driving environments (for example, when parking in a parking lot) may also be considered predetermined driving environments.

[0055] Furthermore, in at least one of the following cases, it is possible to perform at least one of the following: a modification process to reduce the threshold TH1, and a modification process to shorten the judgment time JT1. For example, as shown in Figure 3, it is possible to change the threshold TH2 to be smaller than the threshold TH1, or to change the judgment time JT2 to be shorter than the judgment time JT1. The threshold TH2 and judgment time JT2 can be set appropriately based on experiments, simulations, etc. Also, whether the target of the change is the threshold, the judgment time, or both can be set appropriately based on experiments, simulations, etc.

[0056] Furthermore, if there are multiple predetermined driving environments, it is possible to set modification processes to reduce the threshold TH1 and shorten the judgment time JT1 according to the combination of those predetermined driving environments. For example, when vehicle C1 is traveling on a highway and the vehicle speed of vehicle C1 exceeds a predetermined value, it is possible to execute modification processes to further reduce the threshold TH2 or further shorten the judgment time JT2. Also, for example, when the surroundings of vehicle C1 are in bad weather, or when vehicle C1 is traveling at night and the vehicle speed of vehicle C1 exceeds a predetermined value, it is possible to execute modification processes to further reduce the threshold TH2 or further shorten the judgment time JT2.

[0057] On the other hand, in at least one of the following cases: when the turn signal of vehicle C1 is activated, when vehicle C1 is turning, when vehicle C1 is reversing, when there is traffic congestion in front of vehicle C1, or when automatic driving mode is set for vehicle C1, it is possible to execute at least one of the following: a modification process that significantly changes the threshold TH1, and a modification process that lengthens the judgment time JT1. These thresholds and judgment times can also be set appropriately based on experiments, simulations, etc. Furthermore, whether the target of the change is the threshold, the judgment time, or both can also be set appropriately based on experiments, simulations, etc.

[0058] Figure 3 shows an example using the change in one part of driver D1, but is not limited to this. For example, the deterioration of driver D1's driving posture may be determined using the change in multiple parts of driver D1. For example, it is possible to determine the deterioration of driver D1's driving posture using at least one of the change in each part, such as the change in the position of the arms, the change in the position of the face (head), the change in the position of the body, the change in the angle between the body and the feet, the change in the angle between the body and the arms, the change in the position of the neck, the change in the position of the waist, and the change in the angle at which the ankles bend. In this case, a threshold and judgment time are set for each part. Furthermore, the threshold, judgment time, etc. are changed for each part when a predetermined driving environment is present.

[0059] Furthermore, when determining whether the driving posture of driver D1 has deteriorated using the change amounts of multiple parts of driver D1, the determination unit 122 determines for each part whether the change amount of each part detected by the detection unit 121 exceeds the threshold for that part. If the change amount of at least one part exceeds the threshold for that part, the determination unit 122 determines whether the state in which the change amount exceeds the threshold has continued for longer than the determination time for that part. If the state in which the change amount exceeds the threshold has continued for longer than the determination time, the determination unit 122 determines that the driving posture of driver D1 has deteriorated. Note that this example shows an example in which the driving posture of driver D1 has deteriorated based on the change amount of at least one part, but it is not limited to this. For example, the driving posture of driver D1 may be deteriorated if the change amounts of at least multiple parts exceed the threshold for that part, and the state in which the change amount exceeds the threshold has continued for longer than the determination time for that part.

[0060] In Figure 3, an example is shown where the determination process is performed using the change in one direction from the basic posture as the change in the driver D1's parts, but this is not the only example. As mentioned above, the determination process may also be performed using the change in both directions from the basic posture, or the absolute value of that change, as the change in the driver D1's parts. Alternatively, for example, the determination process may be performed using the difference in the change from the basic posture (for example, the difference between the previous and current states), i.e., the change over time. In this case, for example, it is also possible to perform the determination process using the change per unit time.

[0061] [Example of determining a driver's driving posture] Figure 4 shows an example of a method for determining a driver D1's driving posture. In this example, the above-described determination process is not performed when the driving environment for vehicle C1 is a specific driving environment. Note that, as with Figure 3, the graph in Figure 4 shows the change in driver D1's driving posture on the vertical axis and the time axis on the horizontal axis.

[0062] Here, it is conceivable that it may be difficult to determine if the driving environment for vehicle C1 is a specific driving environment. For example, a specific driving environment means when the turn signal of vehicle C1 is activated, when vehicle C1 is turning, when vehicle C1 is reversing, or when there is traffic congestion in front of vehicle C1. Also, as mentioned above, when the automatic driving mode is set for vehicle C1, driver D1 does not need to perform any driving operations and is relaxed, so it is thought that driver D1's posture is likely to change. For this reason, it is also possible to set a specific driving environment when the automatic driving mode is set. Note that these are just examples of specific driving environments, and other driving environments (for example, when parking in a parking lot) may also be designated as specific driving environments. Furthermore, as shown in Figure 3, at least a part of the predetermined driving environment and the specific driving environment may be common.

[0063] Figure 4 shows an example where the driving environment for vehicle C1 becomes a specific driving environment during the time period from time T1 to time T2. In this case, as shown in Figure 4, it is possible that the change in driver D1's driving posture PC2 exceeds the threshold TH1. However, when the driving environment for vehicle C1 is a specific driving environment, the determination unit 122 does not perform the determination process using the threshold TH1 and determination time JT1 described above. That is, when the driving environment for vehicle C1 is a specific driving environment, it is difficult to determine the deterioration of driver D1's driving posture, and there is a risk of false detection of the deterioration of driver D1's driving posture, so that period is excluded from the determination process. In this way, by not performing the determination process when the driving environment for vehicle C1 is a specific driving environment, it is possible to reduce the computational load related to the determination process during that period. Note that the time period from time T1 to time T2 means a predetermined time period based on the time period when the driving environment for vehicle C1 became a specific driving environment. In this case, the predetermined time may be the period during which the specific operating conditions are met, or the predetermined time may be the period obtained by adding an additional time (for example, a few seconds to tens of seconds) to at least one of the periods before or after the period during which the specific operating conditions are met.

[0064] Note that Figure 4 shows an example in which the determination process is not executed when the driving environment for vehicle C1 becomes a specific driving environment, but it is not limited to this. For example, as shown in Figure 3, when the driving environment for vehicle C1 becomes a specific driving environment, at least one of the following may be executed: a change process that significantly changes the threshold TH1, and a change process that lengthens the determination time JT1. In this case, by making the change value larger than in the example shown in Figure 3, it is possible to make it more difficult to determine that the driver D1's driving posture has deteriorated. Alternatively, when the driving environment for vehicle C1 becomes a specific driving environment, the determination process for deterioration of the driver D1's driving posture may be executed, but the result of that determination process may not be adopted. That is, when the driving environment for vehicle C1 becomes a specific driving environment, the determination process is executed, but by not adopting the result of that determination, it is possible to avoid notifying the driver D1 about deterioration of their driving posture.

[0065] Furthermore, while the above example shows how to determine whether or not the driver D1's driving posture has deteriorated using the in-vehicle image acquired by the in-vehicle image acquisition unit 101, the system is not limited to this. For example, the system may determine whether or not the driver D1's driving posture has deteriorated using sensor values ​​detectable by various other sensors. For instance, a sensor capable of measuring pressure applied to a sensor surface (e.g., a measurement sensor, a pressure sensor) may be installed in the driver's seat of vehicle C1, and the system may determine whether or not the driver D1's driving posture has deteriorated using the sensor values ​​(measured values) from this sensor. In this case, the determination process can be performed based on a comparison between the amount of change in each sensor value in the driver's seat and the threshold and the determination time.

[0066] [Example of Operation of the Operation Monitoring Device] Figure 5 is a flowchart showing an example of the operation monitoring process in the operation monitoring device 110. This operation monitoring process is executed by the control unit 120 (see Figure 1) based on a program stored in the memory unit 130 (see Figure 1). This operation monitoring process is executed continuously at each control cycle. This operation monitoring process will be explained with reference to Figures 1 to 4 as appropriate.

[0067] In step S501, the determination unit 122 acquires driving environment information based on the information output from the external image acquisition unit 102, the vehicle information acquisition unit 103, the communication unit 140, etc. This driving environment information includes, as described above, environmental information regarding the environment around the vehicle C1, driving status information regarding the driving status of the driver D1, etc.

[0068] In step S502, the determination unit 122 determines whether the driving environment for vehicle C1 is a specific driving environment based on the driving environment information acquired in step S501. If the driving environment for vehicle C1 is a specific driving environment, the process proceeds to step S503. On the other hand, if the driving environment for vehicle C1 is not a specific driving environment, the process proceeds to step S504.

[0069] In step S503, the determination unit 122 decides not to perform the determination process for whether the driver D1's driving posture has deteriorated.

[0070] In step S504, the determination unit 122 sets a threshold and a determination time used in the determination process to determine whether the driver D1's driving posture is compromised, based on the driving environment information acquired in step S501. For example, as shown in Figure 3, if the driving environment for vehicle C1 is a normal driving environment, the determination unit 122 sets a threshold TH1 and a determination time JT1. On the other hand, if the driving environment for vehicle C1 is a predetermined driving environment, the determination unit 122 changes at least one of the threshold TH1 and the determination time JT1.

[0071] In step S505, the detection unit 121 acquires the driver D1's posture information based on the in-vehicle image output from the in-vehicle image acquisition unit 101. This posture information includes, as described above, information about each part of the driver D1 (for example, information detected by the skeletal detection technology). The detection unit 121 also stores the driver D1's posture information in memory and calculates the amount of change from the driver D1's basic posture for each part. The detection unit 121 then outputs the amount of change for each part to the determination unit 122. A known calculation method can be used for this calculation.

[0072] In step S506, the determination unit 122 determines whether the amount of change in the driver D1's driving posture exceeds the threshold set in step S504. If the amount of change in the driver D1's driving posture exceeds the threshold, the process proceeds to step S507. On the other hand, if the amount of change in the driver D1's driving posture does not exceed the threshold, the operation of the driving monitoring process is terminated.

[0073] In step S507, the determination unit 122 determines whether the time during which the change in driver D1's driving posture exceeds the threshold has continued for longer than the determination time set in step S504. If the time during which the change in driver D1's driving posture exceeds the threshold has continued for longer than the determination time, the process proceeds to step S508. On the other hand, if the time during which the change in driver D1's driving posture exceeds the threshold has not continued for longer than the determination time, the operation of the driving monitoring process is terminated.

[0074] In step S508, the notification control unit 123 performs notification control to notify the driver D1 of a change in driving posture. For example, the notification control unit 123 obtains notification information from the notification information DB 132 (see Figure 1) that is associated with the part of the body that was determined to have a change in driving posture in step S507. Based on the obtained notification information, the notification control unit 123 performs at least one of the following: displaying an image on the display unit 210 or outputting audio information from the sound output unit 220. The notification control unit 123 also performs at least one of the following based on the obtained notification information: displaying an image on the electronic device 300 or outputting audio information from the electronic device 300. The timing of the output of the notification information from the electronic device 300 may be simultaneous (or nearly simultaneous) with the output timing of the display unit 210, sound output unit 220, etc., or it may be at the timing when the vehicle C1 stops, or at the timing when the driver D1 finishes driving (i.e., when the vehicle C1 is turned off).

[0075] For example, if the part of the body determined to be in a poor driving posture in step S507 is the neck, it is possible to output audio information such as "Your neck is hunched over. Please straighten it," from the sound output unit 220, or to display a corresponding image on the display unit 210. Also, for example, if the part of the body determined to be in a poor driving posture in step S507 is the arm, it is possible to output audio information such as "Your elbow is extended at the steering wheel. Please straighten it," from the sound output unit 220, or to display a corresponding image on the display unit 210. Also, for example, if the part of the body determined to be in a poor driving posture in step S507 is the body, it is possible to output audio information such as "You are sliding down from the seat. Please straighten it," from the sound output unit 220, or to display a corresponding image on the display unit 210. Furthermore, if the timing of outputting notification information from the electronic device 300 is set to coincide with the timing when vehicle C1 stops or when driver D1 finishes driving, notification information including the part of the vehicle whose driving posture was determined to be unstable in step S507 and an appropriate method for restoring the unstable driving posture may be output.

[0076] Furthermore, the processing steps shown in Figure 5 are merely examples for realizing this embodiment, and the order of some of the processing steps may be rearranged, some of the processing steps may be omitted, or other processing steps may be added, as long as this embodiment is feasible. For example, steps S502 and S503 may be omitted, and the judgment process may be executed even in the case of a specific operating environment. However, in the case of a specific operating environment, it is possible to set a value that significantly relaxes the judgment criteria as the threshold in step S504, or to set a judgment time that significantly relaxes the judgment criteria. Alternatively, for example, only the threshold may be set in step S504, and 0 may be used as the judgment time. Note that if 0 is used as the judgment time, step S507 can be omitted. Also, for example, the judgment processes in steps S506 and S507 may be executed as a single process.

[0077] [Example of Effects in This Embodiment] If driver D1's driving posture remains poor while vehicle C1 is in operation, driver D1 is more likely to accumulate fatigue. Therefore, if driver D1's driving posture deteriorates, it is preferable to appropriately notify driver D1 to correct their driving posture and to have driver D1 correct their driving posture appropriately. In particular, if the deterioration of driving posture is due to fatigue, decreased attention, etc., it is important to immediately alert driver D1 and encourage them to rest in order to prevent accidents while vehicle C1 is in operation.

[0078] Therefore, it is conceivable to set small values ​​for the threshold and judgment time used in the judgment to detect a change in driver D1's driving posture early. However, if small values ​​are simply set for the threshold and judgment time in this way, there is a risk of falsely detecting a change in driver D1's driving posture. In this case, driver D1 may receive an inappropriate notification, which may give driver D1 an unpleasant impression.

[0079] Therefore, in this embodiment, the threshold and judgment time used for determination are set to values ​​that have a certain margin according to the driving environment information. By appropriately adjusting (optimizing) the threshold and judgment time used for determination in comparison with the driving environment information in this way, false detections can be reduced and a deterioration in the driver D1's driving posture can be detected early. As mentioned above, the driving environment information includes, for example, the environment around the vehicle C1 and information regarding the driver D1's driving condition.

[0080] In other words, by comparing driver D1's driving posture with driving environment information, it is possible to more accurately detect any deterioration in driver D1's driving posture. This makes it possible to reduce driver D1's fatigue by promptly returning to an appropriate driving posture. As a result, it is possible to improve operability and reaction speed in emergencies, thereby enhancing safety. In other words, by appropriately analyzing driver D1's driving posture, it is possible to prevent driver D1 from becoming fatigued while driving, thereby enhancing the safety of vehicle C1.

[0081] Furthermore, it is known that under certain driving conditions (e.g., high vehicle speed, bad weather, nighttime), the driver D1's field of vision is narrowed. In such cases, the driver D1's perception time while driving is often extended. In such situations, if the driver D1's driving posture deviates significantly from the basic posture (i.e., if the driving posture is poor), it is thought that the driver D1's reaction time is often extended. It is known that the sum of perception time and reaction time is required for emergency avoidance while driving. Therefore, shortening perception time and reaction time is important for emergency avoidance while driving. Accordingly, in this embodiment, by performing an appropriate determination process for the driving posture according to the predetermined driving environment, it is possible to return the driver D1's driving posture to the basic posture at the appropriate timing. This makes it possible to appropriately perform emergency avoidance while driving.

[0082] [Examples of executing processing on other devices or systems] In the above examples, acquisition processing, detection processing, judgment processing, notification processing, etc., are shown to be executed on the driving monitoring device 110 (or driving monitoring system 100). However, all or part of each of these processes may be executed on other devices. In this case, the driving monitoring system is composed of devices that execute part of each of these processes. For example, at least part of each process can be executed using various information processing devices such as in-vehicle devices, devices usable by the user (e.g., smartphones, tablet terminals, personal computers, car navigation systems, IVIs), servers that can be connected via a predetermined network such as the Internet, and various electronic devices.

[0083] Furthermore, some (or all) of the operation monitoring system capable of performing the functions of the operation monitoring device 110 (or operation monitoring system 100) may be provided by an application that can be provided via a predetermined network such as the Internet. This application may be, for example, SaaS (Software as a Service).

[0084] [Configuration Example and Effects of This Embodiment] The driving monitoring method according to this embodiment is a driving monitoring method that monitors the posture of a driver D1 operating a vehicle C1. This driving monitoring method includes a detection process (step S505) for detecting the driving posture of driver D1, an acquisition process (step S501) for acquiring driving environment information including the environment around the vehicle C1 and the driving status of driver D1, a determination process (steps S506, S507) for determining whether the driving posture of driver D1 has deteriorated based on the driving posture of driver D1 and the driving environment information, and a notification process (step S508) for notifying driver D1 if a deterioration in the driving posture of driver D1 has been determined. Furthermore, the program according to this embodiment is a program that causes a computer to execute each of these processes. In other words, the program according to this embodiment is a program that causes a computer to realize each of the functions that the driving monitoring device 110 can execute.

[0085] This configuration makes it possible to perform appropriate judgment processing according to driving environment information. For example, by appropriately adjusting (optimizing) the threshold, judgment time, etc. used in the judgment processing in comparison with the driving environment information, false detections can be reduced, and a deterioration in the driver D1's driving posture can be detected early. This makes it possible to provide driving monitoring technology that can more accurately detect the driver D1's posture at the appropriate timing according to the driving environment of the vehicle C1.

[0086] In the driving monitoring method according to this embodiment, the detection process (step S505) detects the amount of change in a specific part of driver D1, which is an indicator of driver D1's driving posture. In the determination process (steps S504, S506, S507), if the amount of change in a specific part of driver D1 exceeds the threshold TH1 for a determination time JT1, it is determined that driver D1's driving posture has deteriorated. In the determination process (step S504), at least one of the threshold TH1 and the determination time JT1 is changed based on driving environment information. For example, as shown in Figure 3, it is possible to change them to threshold TH2 and determination time JT2.

[0087] With this configuration, thresholds, judgment times, etc., are adjusted (optimized) as appropriate in comparison with driving environment information, and the judgment process is executed using the amount of change in a specific part of driver D1 and the adjusted thresholds, judgment times, etc., thereby reducing false detections and enabling early detection of deterioration in driver D1's driving posture. As a result, it is possible to provide a driving monitoring technology that can more accurately detect driver D1's posture at an appropriate timing according to the driving environment of vehicle C1.

[0088] In the driving monitoring method according to this embodiment, the determination process (step S504) performs at least one of the following: a first modification in which the threshold value TH1 is set to a value that makes the determination criteria stricter in accordance with the increase in the vehicle speed of vehicle C1 (for example, if the absolute value of the amount of change is used, the threshold value TH1 is changed to a smaller value); and a second modification in which the determination time JT1 is shortened.

[0089] It is known that as the vehicle speed of vehicle C1 increases, the driver D1's field of vision narrows, and the driver D1's perception time increases. Therefore, by performing the first modification, the second modification, etc., in accordance with the vehicle speed of vehicle C1, it is possible to shorten the perception time for emergency avoidance during driving, and to appropriately perform emergency avoidance during driving.

[0090] In the driving monitoring method according to this embodiment, in the determination process (step S504), if vehicle C1 is driving on a highway, if the surroundings of vehicle C1 are in bad weather, or if vehicle C1 is driving at night, at least one of the following is performed: a third modification to set a threshold value TH1 with a stricter determination criterion (for example, if the absolute value of the change is used, the threshold value TH1 is changed to a smaller value), and a fourth modification to shorten the determination time JT1.

[0091] It is known that when driving at high speeds on highways, driving in bad weather (e.g., heavy rain), or driving at night, the driver D1's field of vision narrows, and the driver D1's perception time increases. Therefore, under these conditions, by implementing the third and fourth modifications, it is possible to shorten the perception time required for emergency avoidance while driving, and to appropriately perform emergency avoidance while driving.

[0092] In the driving monitoring method according to this embodiment, the determination process (steps S506, S507) does not perform the determination process for a breakdown in the driving posture of driver D1 if any of the following occur: within a predetermined time period based on the time period when the turn signal of vehicle C1 is operating; within a predetermined time period based on the time period when vehicle C1 is turning; within a predetermined time period based on the time period when vehicle C1 is reversing; within a predetermined time period based on the time period when there is congestion in front of vehicle C1; or when automatic driving mode is set for vehicle C1 (steps S502, S503). In any of these cases, at least one of the following may be performed: a modification process that sets the threshold TH1 to a value obtained by relaxing the determination criteria; or a modification process that changes the determination time JT1 to a longer value.

[0093] Here, it is considered that the driver D1's movements during driving operations are often larger when vehicle C1 is turning, reversing, or when there is congestion in front of vehicle C1. Therefore, by not executing the judgment process for the driver D1's driving posture change during these driving operations, it is possible to prevent false detections. This prevents inappropriate notifications to driver D1 and prevents giving driver D1 an unpleasant impression. In addition, by not executing the judgment process, it is possible to reduce the computational load related to the judgment process during that period.

[0094] The driving monitoring device 110 is a driving monitoring device that monitors the posture of driver D1 operating vehicle C1. The driving monitoring device 110 includes a detection unit 121 that detects the driving posture of driver D1, an acquisition unit (in-vehicle image acquisition unit 101, exterior image acquisition unit 102, vehicle information acquisition unit 103, communication unit 140) that acquires driving environment information including the environment around vehicle C1 and the driving status of driver D1, a determination unit 122 that determines whether driver D1's driving posture has deteriorated based on driver D1's driving posture and the driving environment information, and a notification control unit 123 (an example of a notification unit) that notifies driver D1 if it has been determined that driver D1's driving posture has deteriorated. Alternatively, instead of the driving monitoring device 110, a driving monitoring system consisting of multiple devices capable of executing each of the processes realized by the driving monitoring device 110 may be used.

[0095] This configuration makes it possible to perform appropriate judgment processing according to driving environment information. For example, by appropriately adjusting (optimizing) the threshold and judgment time used in the judgment processing in comparison with the driving environment information, false detections can be reduced, and a deterioration in the driver D1's driving posture can be detected early. This makes it possible to provide a driving monitoring technology that can more accurately detect the driver D1's posture at the appropriate timing according to the driving environment of the vehicle C1.

[0096] Furthermore, each process in this embodiment is executed based on a program that causes a computer to perform various processing procedures. This embodiment can also be understood as an embodiment of a program that realizes the function of executing each of these processes, and a recording medium that stores that program. For example, by performing an update process to add a new function to the operation monitoring device, the program can be stored in the storage device of the operation monitoring device. This makes it possible to have the updated operation monitoring device perform each of the processes shown in this embodiment.

[0097] Although embodiments of the present invention have been described above, these embodiments merely illustrate examples of how the present invention can be applied, and are not intended to limit the technical scope of the present invention to the specific configurations of the embodiments described above.

Claims

1. A driving monitoring method for monitoring the posture of a driver operating a vehicle, comprising: a detection process for detecting the driver's driving posture; an acquisition process for acquiring driving environment information including the environment surrounding the vehicle and the driver's driving status; a determination process for determining whether the driver's driving posture has deteriorated based on the driving posture and the driving environment information; and a notification process for notifying the driver if a deterioration in the driving posture has been determined.

2. A driving monitoring method according to claim 1, wherein the detection process detects a change in a specific part of the driver that serves as an indicator of the driving posture, the determination process determines that the driving posture has deteriorated if the change in the specific part exceeds a threshold for a determination period of time, and the determination process changes at least one of the threshold and the determination period based on the driving environment information.

3. A driving monitoring method according to claim 2, wherein the determination process performs at least one of the following: a first modification, in which the determination criteria are made stricter as the vehicle speed increases, and a second modification, in which the determination time is shortened.

4. A driving monitoring method according to claim 2, wherein in the determination process, in any of the cases where the vehicle is driving on a highway, the surroundings of the vehicle are in bad weather, or the vehicle is driving at night, at least one of a third modification, in which the threshold value is set to a value that makes the determination criteria stricter, and a fourth modification, in which the determination time is shortened.

5. A driving monitoring method according to any one of claims 1 to 4, wherein the determination process does not perform the determination process for deterioration of driving posture in any of the following cases: within a predetermined time period based on the time period when the vehicle's turn signal is operating; within a predetermined time period based on the time period when the vehicle is turning; within a predetermined time period based on the time period when the vehicle is reversing; within a predetermined time period based on the time period when there is congestion in front of the vehicle; or when an automatic driving mode is set in the vehicle.

6. A driving monitoring device for monitoring the posture of a driver operating a vehicle, comprising: a detection unit for detecting the driver's driving posture; an acquisition unit for acquiring driving environment information including the environment around the vehicle and the driver's driving status; a determination unit for determining whether the driver's driving posture has deteriorated based on the driving posture and the driving environment information; and a notification unit for notifying the driver if a deterioration in the driving posture has been determined.

Images