Driving assistance method and driving assistance device

Abstract

The present invention includes: a detection process for detecting an action of a driver and an object within a vehicle; and an output process for outputting driving assistance information for causing the driver to stop a specific operation when it is estimated that a stopped vehicle will set off and it is determined to be highly likely that the driver will execute the specific operation after the start of the travel of the vehicle, or when it is estimated that a traveling vehicle will stop and it is determined to be highly likely that the driver will execute the specific operation before the vehicle stops.

Description

Driving assistance method and driving assistance device

[0001] The present invention relates to a driving assistance method and a driving assistance device for assisting a driver in driving a vehicle.

[0002] Conventionally, there are technologies for assisting a driver in driving a vehicle. For example, Japanese Patent Application Laid-Open Publication No. 2006-92129 discloses a technology that, when a traffic light ahead of a vehicle changes from not permitting departure to permitting departure while the vehicle is stopped, notifies the driver by voice that the traffic light has changed to permitting departure when there is another vehicle behind the vehicle or when there is no passenger in the passenger seat.

[0003] Here, it is assumed that the driver of a stopped vehicle may perform a specific action other than driving, such as operating a navigation device. It is also assumed that the driver of a vehicle stopped because the traffic light ahead indicates that starting is prohibited may also perform such a specific action. In this case, even if the driver is notified of the change by voice using the above-mentioned conventional technology when the traffic light ahead changes from prohibiting starting to permitting starting, the driver may continue the specific action that he or she was performing while waiting at the traffic light when starting the vehicle after receiving the notification. It is also assumed that the driver of a moving vehicle may start such a specific action just before stopping. Therefore, it is important to appropriately prevent the driver from performing such a specific action while driving, after starting driving or just before stopping.

[0004] The present invention aims to appropriately prevent specific actions while a vehicle is in motion, after the vehicle has started to move or immediately before the vehicle is stopped.

[0005] One aspect of the present invention is a driving assistance method for assisting a driver of a vehicle in driving, the driving assistance method including: a detection process for detecting at least one of a driver's behavior and an object in the vehicle; an estimation process for estimating that a stopped vehicle will start or a moving vehicle will stop; a determination process for determining, based on the detection result of the detection process when it is estimated that the stopped vehicle will start, whether or not the driver is likely to perform a specific driving operation, in which the driver performs a specific act other than an act related to driving, after the vehicle starts moving, or, based on the detection result of the detection process when it is estimated that the moving vehicle will stop, whether or not the driver is likely to perform the specific driving operation before the vehicle stops; and, when it is determined in the determination process that the driver is likely to perform the specific driving operation, an output process for causing an output unit to output driving assistance information for causing the driver to stop the specific driving.

[0006] FIG. 1 is a diagram illustrating an example of a system configuration of an information processing system. FIG. 2 is a diagram illustrating an example of a transition when driving assistance information is output. FIG. 3 is a diagram illustrating an example of a transition when driving assistance information is output. FIG. 4 is a diagram illustrating a simplified example of a configuration of a notification level list. FIG. 5 is a diagram illustrating a simplified example of a configuration of a warning level list. FIG. 6 is a diagram illustrating a simplified example of a configuration of a driving assistance setting information DB. FIG. 7 is a diagram illustrating a simplified example of a configuration of a driver response history information DB. FIG. 8 is a flowchart illustrating an example of an occupant behavior detection process. FIG. 9 is a flowchart illustrating an example of an in-vehicle object detection process. FIG. 10 is a flowchart illustrating an example of a driving assistance information output process. FIG. 11 is a flowchart illustrating an example of a driving assistance information output process. FIG. 12 is a simplified diagram illustrating the external configuration of an arrow-type traffic light. FIG. 13 is a diagram illustrating a simplified example of a configuration of a notification level list. FIG. 14 is a flowchart illustrating an example of a driving assistance information output process. FIG. 15 is a flowchart illustrating an example of a driving assistance information output process.

[0007] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.

[0008] 1 is a diagram showing an example of the system configuration of an information processing system 100 installed in a vehicle C1. The information processing system 100 is an example of a driving assistance device that notifies a driver D1 (see FIGS. 2 and 3 ) of a change in the indication of a traffic light ahead of the vehicle C1.

[0009] The information processing system 100 includes an exterior image acquisition unit 111, an image analysis unit 112, a traffic light detection unit 113, an interior image acquisition unit 121, an image analysis unit 122, an object detection unit 123, a behavior detection unit 124, a vehicle information acquisition unit 130, a control unit 140, a memory unit 150, a communication unit 160, and an output unit 170.

[0010] The information processing system 100 is connected to a network 20 by a communication method using wireless communication. The network 20 is a network such as a public line network or the Internet.

[0011] The outside-vehicle image acquisition unit 111 captures an image of a subject outside the vehicle C1 to generate an image (image data), and outputs image information related to the generated image to the image analysis unit 112. The outside-vehicle image acquisition unit 111 is configured, for example, with an imaging element (image sensor) that receives light from the subject collected by a lens and an image processing unit that performs predetermined image processing on the image data generated by the imaging element. For example, a charge-coupled device (CCD) or complementary metal oxide semiconductor (CMOS) imaging element can be used as the imaging element. Note that two or more outside-vehicle image acquisition units 111 may be provided, and all or some of the images captured by these outside-vehicle image acquisition units 111 may be used. For example, one outside-vehicle image acquisition unit 111 may be provided in front of the vehicle C1 to capture an image of a subject in front of the vehicle C1 to generate an image (image data), and another outside-vehicle image acquisition unit 111 may be provided behind the vehicle C1 to capture an image of a subject behind the vehicle C1 to generate an image (image data). Alternatively, one or more devices capable of capturing images of subjects present in all directions around the vehicle C1 and subjects inside the vehicle C1, such as a 360-degree camera, may be used.

[0012] The image analysis unit 112 performs a predetermined image analysis process on the image information output from the outside-of-vehicle image acquisition unit 111, and outputs the analysis results to the traffic light detection unit 113. This image analysis process can use a known image analysis process, and for example, an analysis that can be used when the traffic light detection unit 113 performs traffic light detection processing is performed.

[0013] The traffic light detection unit 113 detects traffic lights and their instructions included in the image (image of the front of the vehicle C1) acquired by the outside image acquisition unit 111 based on the analysis results of the image information output from the image analysis unit 112. The traffic light detection unit 113 then outputs the detection results to the control unit 140. Note that the traffic lights to be detected by the traffic light detection unit 113 are basically those that are located in front of the vehicle C1 and are the closest. Furthermore, traffic lights and their instructions can be detected using known image recognition technology. Furthermore, traffic light instructions include, for example, instructions indicating permission to start, denial of start, etc. Note that, if it is possible to acquire the presence of a traffic light and its instructions based on wave information (e.g., beacon or wireless communication) generated from a traffic light or its vicinity, the traffic light and its instructions may be detected based on the wave information.

[0014] The interior image acquisition unit 121 captures an image of a subject inside the vehicle C1 and generates an image (image data), and outputs image information related to the generated image to the image analysis unit 122. Similar to the exterior image acquisition unit 111, the interior image acquisition unit 121 is configured with an imaging element that receives light from the subject collected by a lens and an image processing unit that performs predetermined image processing on the image data generated by the imaging element. Two or more interior image acquisition units 121 may be provided, and all or some of the images from these interior image acquisition units 121 may be used. For example, one interior image acquisition unit 121 may be provided in the front of the interior of the vehicle C1 and capture an image of a subject inside the vehicle C1 from the front of the vehicle C1 to generate an image (image data). Another interior image acquisition unit 121 may be provided in the rear of the interior of the vehicle C1 and capture an image of a subject inside the vehicle C1 from the rear of the vehicle C1 to generate an image (image data). Alternatively, one or more devices capable of capturing an image of a subject inside the vehicle C1, such as a 360-degree camera, may be used. Furthermore, the vehicle exterior image acquisition unit 111 and the vehicle interior image acquisition unit 121 may be shared by one or more devices.

[0015] The image analysis unit 122 performs a predetermined image analysis process on the image information output from the in-vehicle image acquisition unit 121, and outputs the analysis results to the object detection unit 123 and the behavior detection unit 124. This image analysis process can use a known image analysis process, and for example, an analysis that can be used when the detection process is performed by the object detection unit 123 and the behavior detection unit 124 is performed. Note that the image analysis unit 112 and the image analysis unit 122 may be shared by one or more devices.

[0016] The object detection unit 123 detects objects and their positions within the vehicle cabin contained in the image (image of the interior of the vehicle C1) acquired by the interior image acquisition unit 121 based on the analysis result of the image information output from the image analysis unit 122. The object detection unit 123 then outputs the detection result to the control unit 140. Note that the object and its position can be detected using known image recognition technology. For example, it is possible to detect a smartphone, book, bag, etc. present within the vehicle cabin. Note that the vehicle cabin object detection process performed by the object detection unit 123 will be described in detail with reference to FIG. 9 .

[0017] The behavior detection unit 124 detects occupants seated in each seat of the vehicle C1 and their behaviors contained in the image (image of the interior of the vehicle C1) acquired by the interior image acquisition unit 121 based on the analysis results of the image information output from the image analysis unit 122. The behavior detection unit 124 then outputs the detection results to the control unit 140. Note that the occupants and their behaviors can be detected using known image recognition technology. For example, it is possible to detect the presence or absence of an occupant in each seat, the facial orientation, facial expression, movement, hand movement, etc. of each occupant. Note that the occupant behavior detection process performed by the behavior detection unit 124 will be described in detail with reference to FIG. 8 . The presence and behavior of an occupant may also be detected based on sounds acquired by a sound acquisition unit (e.g., a microphone).

[0018] The signal detection process by the traffic light detection unit 113, the vehicle interior object detection process by the object detection unit 123 (see FIG. 9), and the occupant behavior detection process by the behavior detection unit 124 (see FIG. 8) can also be performed by utilizing artificial intelligence (AI) to detect each object. In addition, some or all of these detection processes may be performed by the control unit 140.

[0019] The vehicle information acquisition unit 130 acquires various vehicle information related to the vehicle C1 and outputs the acquired vehicle information to the control unit 140. The vehicle information includes, for example, vehicle speed, acceleration, the position of the shift lever (e.g., P range, D range), the amount of depression of the accelerator pedal, the amount of depression of the brake pedal, and whether an error has occurred in each system constituting the information processing system 100. If the vehicle C1 is equipped with a sensor (e.g., a capacitance-type touch sensor or a contact sensor) capable of detecting the contact state of the driver D1 with the steering wheel 3, it is possible to acquire the contact state of the driver D1 with the steering wheel 3 detected by the sensor. For example, it is possible to determine whether the vehicle C1 is stopped or moving based on the vehicle speed, acceleration, etc.

[0020] The vehicle information acquisition unit 130 may also acquire sensor detection information output from various sensors installed in the vehicle C1. Examples of the sensors include a light detection and ranging (LiDAR), a radio detection and ranging (RADAR), a sonar, a vehicle speed sensor, an acceleration sensor, a steering sensor (steering force angle meter), an accelerator position sensor, and a position information acquisition sensor (position information acquisition unit). Known sensors can be used for each of these sensors. The LiDAR, RADAR, and sonar are examples of sensors that detect the situation around the vehicle C1. The vehicle speed sensor, acceleration sensor, steering sensor, and accelerator position sensor are examples of sensors that detect the driving operation status of the driver D1. These are merely examples, and other sensors may be used. Alternatively, only some of these sensors may be used.

[0021] The location information acquisition unit acquires location information regarding the location of the vehicle C1. For example, the location information acquisition unit can be implemented by a GNSS receiver that acquires location information using a Global Navigation Satellite System (GNSS). The location information includes data regarding the location, such as latitude, longitude, and altitude, at the time of receiving the GNSS signal. The location information may also be acquired by other methods. For example, the location information may be derived using information from nearby access points or base stations. The location information may also be acquired using a beacon.

[0022] The control unit 140 controls each unit based on various programs stored in the storage unit 150. The control unit 140 is realized by a processing device such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). The vehicle ECU (Electronic Control Unit) of the vehicle C1 may also be used as the control unit 140, or a processing device different from the vehicle ECU may be provided as the control unit 140.

[0023] The control unit 140 executes various controls based on the information output from the traffic light detection unit 113, the object detection unit 123, the behavior detection unit 124, the vehicle information acquisition unit 130, the storage unit 150, the communication unit 160, etc. For example, the control unit 140 executes a control process to control the operating state of the output unit 170. Specifically, the control unit 140 includes an estimation unit 141, a determination unit 142, a decision unit 143, an output control unit 144, and a setting unit 145.

[0024] The estimation unit 141 executes estimation processing to estimate that the stopped vehicle C1 will start moving or that the traveling vehicle C1 will stop, based on the information output from the traffic light detection unit 113, the behavior detection unit 124, the vehicle information acquisition unit 130, the storage unit 150, the communication unit 160, etc. Then, the estimation unit 141 outputs the estimation result to the determination unit 142, the output control unit 144, etc. These estimation processing will be described in detail with reference to FIGS. 10 , 11 , 14 , 15 , etc.

[0025] The determination unit 142 determines whether or not the driver D1 is likely to perform specific driving, in which the driver D1 performs a specific action other than driving-related actions, based on information output from the object detection unit 123, the behavior detection unit 124, the vehicle information acquisition unit 130, the storage unit 150, the communication unit 160, etc. Then, the determination unit 142 outputs the determination result to the decision unit 143, the output control unit 144, etc. Note that specific actions include, for example, operating various electronic devices, staring at various electronic devices, eating and drinking, reading a book, etc. Eating and drinking means, for example, holding a drink in one's hand or food in one's hand. Driving while holding a drink in one's hand or food in one's hand is referred to as specific driving. Specific driving while performing a specific action is also sometimes referred to as, for example, distracted driving.

[0026] Specifically, when the estimation unit 141 estimates that the stopped vehicle C1 will start moving, the determination unit 142 determines whether or not there is a high possibility that the driver D1 will perform the specific driving after the vehicle C1 starts moving, based on the detection results of the detection processes by the object detection unit 123 and the behavior detection unit 124. Furthermore, when the estimation unit 141 estimates that the moving vehicle C1 will stop, the determination unit 142 determines whether or not there is a high possibility that the driver D1 will perform the specific driving immediately before the vehicle C1 stops, based on the detection results of the detection processes by the object detection unit 123 and the behavior detection unit 124. These determination processes will be described in detail with reference to FIGS. 10 , 11 , 14 , 15 , etc.

[0027] The decision unit 143 decides the content of the driving assistance information based on the information output from the traffic light detection unit 113, the object detection unit 123, the behavior detection unit 124, the vehicle information acquisition unit 130, the storage unit 150, the communication unit 160, etc. For example, when the determination unit 142 determines that the driver D1 is likely to perform a specific driving operation, the decision unit 143 decides the content of the driving assistance information for stopping the specific driving operation based on a risk level indicating the likelihood of performing the specific driving operation. The decision unit 143 then outputs the decision result to the output control unit 144. Note that a method for deciding the content of the driving assistance information will be described in detail with reference to FIGS. 4 to 11, 13 to 15, etc. The driving assistance information shown in this embodiment can be understood as information for stopping the driver D1 from performing a specific driving operation in which the driver D1 performs a specific act other than driving, information for prohibiting the driver D1 from performing the specific driving operation, or information for urging the driver D1 to take some action to stop the specific driving operation.

[0028] The output control unit 144 controls the output state of the output unit 170 based on each piece of information output from the traffic light detection unit 113, the object detection unit 123, the behavior detection unit 124, the vehicle information acquisition unit 130, the storage unit 150, the communication unit 160, etc. Specifically, when the determination unit 142 determines that there is a high possibility that the driver D1 will perform a specific driving behavior, the output control unit 144 executes an output process to cause the output unit 170 to output the driving assistance information determined by the determination unit 143. Note that output examples of the driving assistance information are shown in FIGS. 2 and 3 .

[0029] When driving assistance information is output from the output unit 170, if the driver D1 has a negative reaction to the output, the setting unit 145 stores negative reaction information related to the negative reaction in the driver response history information DB 230 (see FIG. 7 ). For example, the setting unit 145 can determine whether the driver D1 has a negative reaction to the driving assistance information output from the output unit 170 based on information (e.g., the facial expression and gaze of the driver D1) output from the behavior detection unit 124. Furthermore, the setting unit 145 changes the content of the driving assistance setting information DB 220 (see FIG. 6 ) based on the negative reaction information stored in the driver response history information DB 230. Note that a method of storing negative reaction information in the driver response history information DB 230 and a method of changing the driving assistance setting information DB 220 will be described in detail with reference to FIGS. 6 and 7 .

[0030] The storage unit 150 is a storage medium that stores various types of information. For example, the storage unit 150 stores various types of information (e.g., a control program, a notification level list 200 (see FIG. 4 ), a warning level list 210 (see FIG. 5 ), a driving assistance setting information DB 220 (see FIG. 6 ), a driver response history information DB 230 (see FIG. 7 ), and a map information DB) that are required for the control unit 140 to perform various processes. The storage unit 150 also stores various types of information acquired via the communication unit 160. For example, it is possible to update various programs based on information acquired from the server 10 via the communication unit 160. The storage unit 150 may be, for example, a read-only memory (ROM), a random access memory (RAM), a static random access memory (SRAM), a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0031] The communication unit 160 exchanges various types of information with other devices using wired or wireless communication under the control of the control unit 140. For example, the communication unit 160 can exchange information with the server 10 via the network 20. The communication unit 160 can also exchange information indirectly with the electronic device 30 via the network 20, or directly with the electronic device 30 using wireless communication.

[0032] The output unit 170 outputs various information based on the control of the output control unit 144. Specifically, the output unit 170 includes an audio output unit 171 and a display unit 172. It is possible to use various devices installed in the vehicle C1 as the output unit 170. For example, a navigation device, an IVI (In-Vehicle Infotainment), etc. can be used as the audio output unit 171 and the display unit 172. It is also possible to use a speaker as the audio output unit 171. It is also possible to use a HUD (Head Up Display) implemented on the windshield 4 as the display unit 172. FIGS. 2 and 3 show an example in which an IVI is used as the audio output unit 171 and the display unit 172.

[0033] The server 10 is an information processing device capable of providing various types of information to the vehicle C1. The server 10 may also manage the various types of information transmitted from the vehicle C1 in a vehicle management DB and transmit various types of control information to the vehicle C1 based on the vehicle management DB.

[0034] The electronic device 30 is a portable electronic device carried by the driver D1. The electronic device 30 includes, for example, a control unit, an operation unit, a memory unit, a wireless communication unit (not shown), a display unit 31 (see FIGS. 2 and 3 ), and the like. The control unit of the electronic device 30 controls each unit based on various programs stored in the memory unit of the electronic device 30. The control unit of the electronic device 30 also communicates with other devices via the wireless communication unit of the electronic device 30, and is capable of outputting information to the other devices and outputting information from the display unit 31 based on control from the other devices.

[0035] For example, installing a predetermined application (vehicle app) in the electronic device 30 enables the exchange of various pieces of information between the electronic device 30 and the information processing system 100. For example, it is possible to perform a predetermined operation (e.g., an audio operation) in the vehicle C1 based on a user operation using the electronic device 30. Furthermore, for example, it is possible to output various pieces of information from the output unit (display unit 31, audio output unit) of the electronic device 300 based on control information transmitted from the information processing system 100.

[0036] 2 and 3 are diagrams showing an example of a transition when driving assistance information is output in the vehicle C1. Note that Fig. 2 and Fig. 3 show a simplified view of the interior of the vehicle C1, and omit illustrations of components other than the dashboard 2, steering wheel 3, front window 4, rearview mirror 5, and display unit 172. Fig. 2 and Fig. 3 also show an example in which a red traffic light indicates a state in which starting is not permitted, and a green traffic light indicates a state in which starting is permitted.

[0037] 2 and 3 show simplified views of the actual scene seen through the front window 4 of vehicle C1. Fig. 2 shows an example in which vehicle C1 is stopped because traffic light SG1 is red just before intersection IS1 where roads R1 and R2 intersect. Fig. 3 shows an example in which vehicle C1 is moving just before intersection IS11 where roads R11 and R12 intersect because traffic light SG1 is green just before intersection IS11.

[0038] As shown in Figure 2 (A), when traffic light SG1 is red and vehicle C1 is stopped at intersection IS1, it is expected that driver D1 will hold electronic device 30 in his / her hand, operate electronic device 30, and check various information displayed on electronic device 30.

[0039] FIG. 2B shows an example in which the traffic light SG1 changes from red to green.

[0040] As described above, it is expected that the driver D1 of the stopped vehicle C1 may engage in specific behaviors other than driving, such as operating the electronic device 30 or watching videos on the electronic device 30. Furthermore, as shown in FIG. 2A , it is also expected that the driver D1 may take his hands off the steering wheel 3 and engage in such specific behaviors even when the vehicle is stopped because the traffic light SG1 ahead is red. In such a case, if the specific behavior continues after the traffic light SG1 ahead changes to green, there is a high possibility that the driver D1 will engage in specific driving while engaging in the specific behavior (so-called distracted driving) after the vehicle C1 starts moving. If such specific behavior continues and the driver D1 engages in specific driving after the vehicle C1 starts moving, the risk of an accident may increase. Therefore, it is important to prevent the specific behavior from continuing and to prevent the driver D1 from engaging in specific driving after the vehicle C1 starts moving. In other words, it is important to appropriately prevent the specific behavior after the traffic light SG1 ahead changes to green and the vehicle C1 starts moving, thereby reducing the risk of an accident caused by the specific behavior.

[0041] Therefore, in this embodiment, as shown in Fig. 2(B), if the driver D1 of the stopped vehicle C1 is performing a specific behavior immediately after the traffic light SG1 changes from red to green, driving assistance information for stopping the specific behavior is output from the output unit 170. Specifically, the content and output method of the driving assistance information are changed and output depending on the specific behavior performed by the driver D1 of the stopped vehicle C1. Note that setting examples of the content and output method of the driving assistance information will be described in detail with reference to Figs. 4 to 7.

[0042] For example, as shown in FIG. 2B , when the driver D1 is operating or viewing the electronic device 30, it is determined that the driver D1 is engaging in a specific behavior that could lead to danger. In this manner, when the driver D1 is not gripping the steering wheel 3 and is determined to be engaging in a specific behavior that could lead to danger, driving assistance information with a predetermined notification level NL2 (see FIG. 4 ) is output. In this embodiment, among the specific behaviors, behaviors or actions that could lead to danger are particularly referred to as specific behaviors. For example, as shown in FIG. 2B , audio information WS1 such as "The traffic light has turned green. Please stop using your smartphone and hold the steering wheel" is output from the audio output unit 171, and text information such as "It is dangerous. Please stop using your smartphone immediately" is displayed on the display unit 172. Depending on the behavioral state of the driver D1, either audio output or display may be selected as the output target. For example, when the driver D1 cannot see the display unit 172, it is possible to output only from the audio output unit 171. Furthermore, when predetermined communication using wireless communication is possible between the electronic device 30 held by the driver D1 and the information processing system 100, it is also possible to display text information such as "Danger" on the display unit 31, as shown in FIG. 2(B). This makes it possible to output driving assistance information from an optimal output unit depending on the devices present in the vehicle C1, the behavioral state of the driver D1, etc. This makes it possible to appropriately prevent specific actions (e.g., operating the electronic device 30, watching the electronic device 30) after the vehicle C1 starts traveling, thereby reducing the risk of an accident caused by the specific actions.

[0043] As shown in FIG. 3A , when the traffic light SG11 is green and the vehicle C1 is traveling toward the intersection IS11, the driver D1 is expected to grip the steering wheel 3 and perform a driving-related action. In this case, it is expected that the driver D1 is unlikely to perform a specific driving operation in which the driver D1 performs a specific action other than driving while driving. Assume that the traffic light SG11 changes from red to green under such a situation. In this case, the driver D1 performs a driving operation to stop the vehicle C1 before passing the intersection IS11. Here, after the vehicle C1 stops before passing the intersection IS11, the driver D1 is also expected to perform a specific action other than driving, such as operating the electronic device 30 or watching the electronic device 30. It is also expected that the driver D1 will start such a specific action after the vehicle C1 stops and before performing the specific action (i.e., immediately before the vehicle C1 stops). Such a situation is shown in FIG. 3B .

[0044] FIG. 3B shows an example in which the driver D1 starts a specific action immediately before the vehicle C1 stops after the traffic light SG11 changes from green to red.

[0045] As shown in Figure 3 (B), when the traffic light SG1 ahead changes to red, it is assumed that the driver D1 performs a driving operation to stop the traveling vehicle C1, and then starts a specific behavior just before the vehicle C1 stops. If such a specific behavior continues until the vehicle stops and the driver D1 engages in distracted driving, the risk of an accident may increase. Therefore, it is important to prevent the specific behavior while traveling just before stopping and to prevent the driver D1 from engaging in distracted driving while traveling. In other words, it is important to appropriately prevent the specific behavior just before the traffic light SG1 ahead changes from green to red and the vehicle C1 stops, thereby reducing the risk of an accident caused by the specific behavior.

[0046] Therefore, in this embodiment, as shown in Fig. 3(B), when the driver D1 of the vehicle C1 before stopping starts a specific behavior after the traffic light SG1 changes from green to red, driving assistance information for stopping the specific behavior is output from the output unit 170. Specifically, the content and output method of the driving assistance information are changed and output depending on the specific behavior being performed by the driver D1 of the vehicle C1 while it is moving. Note that setting examples of the content and output method of the driving assistance information will be described in detail with reference to Figs. 4 to 7.

[0047] For example, as shown in FIG. 3B , if the driver D1 starts operating the electronic device 30, viewing the electronic device 30, or the like, it is determined that the driver D1 is engaging in a specific behavior that could lead to danger. In this way, if the driver D1 is not gripping the steering wheel 3 and is determined to be engaging in a specific behavior that could lead to danger, driving assistance information with the highest warning level WL4 (see FIG. 5 ) is output. For example, as shown in FIG. 3B , audio information WS11 stating "Please stop using your smartphone and hold the steering wheel until the vehicle has completely stopped" is output from the audio output unit 171, and text information stating "It is dangerous. Please stop using your smartphone immediately" is displayed on the display unit 172. Furthermore, if a predetermined exchange using wireless communication is possible between the electronic device 30 held by the driver D1 and the information processing system 100, text information stating "It is dangerous" can also be displayed on the display unit 31, as shown in FIG. 3B . These features make it possible to appropriately prevent specific behaviors (e.g., operating the electronic device 30, viewing the electronic device 30) while the vehicle C1 is traveling, thereby reducing the risk of an accident caused by the specific behavior.

[0048] 2 and 3 show an example in which driving assistance information is output to the driver D1 by both voice and text display, but other output methods may be used. For example, icons corresponding to the levels of driving assistance information may be set in advance, and the icon of the level determined by the determination unit 143 may be displayed on the display unit 172. This makes it possible to provide driving assistance to the driver D1 using visually easy-to-understand driving assistance information.

[0049] [Configuration Example of Notification Level List] FIG. 4 is a diagram showing a simplified configuration example of the notification level list 200 stored in the storage unit 150. As shown in FIG.

[0050] The notification level list 200 is a list indicating the level of driving assistance information to be output when it is estimated that the stopped vehicle C1 will start moving and it is determined that the driver D1 is likely to perform specific driving (distracted driving) after the vehicle C1 starts moving. Specifically, the notification level list 200 is a list indicating the notification level of driving assistance information to be output when it is detected that the indication of a traffic light has changed from "start prohibition" to "start permission" while the vehicle C1 is stopped in front of the traffic light. Note that a specific method for determining the notification level will be described in detail with reference to FIG. 11 .

[0051] Here, when it is estimated that the stopped vehicle C1 will start moving, at the timing when it is determined that the driver D1 is likely to perform the specific driving after the vehicle C1 starts moving, the vehicle C1 is stopped and therefore does not fall under the specific driving. Therefore, the driving assistance information output at such a timing is output as notification information to alert the driver D1. In addition, the level of these notification information will be referred to as the notification level.

[0052] FIG. 4 shows an example in which the notification level 201 is set to four levels depending on the behavior of the driver D1. Specifically, a level at which it is estimated that there is a risk of an accident due to the behavior of the driver D1 but the degree of the risk is low is set to "NL1." Furthermore, "NL2" to "NL4" are set depending on the level at which it is estimated that the risk of an accident due to the behavior of the driver D1 is high. In other words, a level at which it is estimated that there is an extremely high risk of an accident due to the behavior of the driver D1 is "NL4." Note that, although an example in which four levels are set is shown in this embodiment, two, three, five or more levels may be set.

[0053] Furthermore, the notification level 201 is associated with and set to notification content 202, notification frequency 203, and notification volume 204.

[0054] The notification content 202 indicates text information to be output as driving assistance information. The text information set in the notification content 202 can be output from the audio output unit 171 and displayed on the display unit 172. The text information set in the notification content 202 may be output as is, or may be abbreviated or have other information added to it. For example, if a specific object that may cause a specific behavior is detected, the name of the specific object may be added to the output. The specific object refers to an object that may induce the driver D1 to look away. Examples of the specific object include a mobile phone, a smartphone, a book, etc. For example, as shown in FIG. 2B , if it is detected that the driver D1 is holding a smartphone, the word "smartphone" may be added to the output. This makes it possible to clearly notify the driver D1 to stop using his smartphone in order to stop the specific driving behavior.

[0055] The notification frequency 203 indicates the frequency with which the driving assistance information is output. When the notification frequency 203 is "once," this means that the driving assistance information is output only once when it is determined that the driver D1 is likely to perform the specific driving after the vehicle C1 starts traveling. On the other hand, when the notification frequency 203 is "repeated until stopped," this means that, after it is determined that the driver D1 is likely to perform the specific driving after the vehicle C1 starts traveling, the driving assistance information is continuously output until the driver D1 stops the specific driving. Whether the driver D1 has stopped the specific driving (or the specific behavior) can be determined based on the detection result by the behavior detection unit 124.

[0056] The notification volume 204 indicates the volume of the driving assistance information when it is output as audio. As the notification volume 204 changes from "low" to "high," the volume of the driving assistance information that is output as audio is set to increase.

[0057] 4 shows an example in which change information for notifying a change in a traffic light ahead of the vehicle C1 and notification information for discontinuing specific driving while performing specific actions other than driving are simultaneously output. However, these pieces of information may be output at different times rather than simultaneously, or only the notification information may be output as needed.

[0058] [Configuration Example of Warning Level List] FIG. 5 is a diagram showing a simplified configuration example of the warning level list 210 stored in the storage unit 150. As shown in FIG.

[0059] The warning level list 210 is a list indicating the level of driving support information to be output when it is estimated that the traveling vehicle C1 will stop and it is determined that the driver D1 is likely to perform a specific driving operation immediately before the vehicle C1 stops. Specifically, the warning level list 210 is a list indicating the warning level of driving support information to be output when the vehicle C1 is traveling in front of a traffic light and it is detected that the traffic light indicates that starting is not permitted. A specific method for determining the warning level will be described in detail with reference to FIG. 10.

[0060] Here, when it is estimated that the vehicle C1 is going to stop while traveling, the vehicle C1 is still traveling at the time when it is determined that the driver D1 is likely to perform a specific driving behavior immediately before the vehicle C1 stops. Therefore, performing a specific behavior would be considered specific driving. Therefore, the driving assistance information output at such a time is output as warning information to strongly inform the driver D1 that he or she should stop. In addition, the level of this warning information will be referred to as a warning level.

[0061] 5 shows an example in which the warning level 211 is set to four levels depending on the behavior of the driver D1. Specifically, a level at which it is estimated that there is a risk of an accident due to the behavior of the driver D1 but the degree of the risk is low is set to "WL1." Furthermore, "WL2" to "WL4" are set depending on the level at which it is estimated that the behavior of the driver D1 will increase the risk of an accident. In other words, a level at which it is estimated that there is an extremely high risk of an accident due to the behavior of the driver D1 is set to "WL4." Note that, although an example in which four levels are set is shown in this embodiment, two, three, five or more levels may also be set.

[0062] Furthermore, the warning level 211 is associated with a warning content 212, a warning frequency 213, and a warning volume 214.

[0063] The warning content 212 indicates text information to be output as driving assistance information. The text information set in the warning content 212 can be output from the audio output unit 171 and displayed on the display unit 172. The text information set in the warning content 212 may be output as is, or may be abbreviated or have other information added to it. For example, as shown in FIG. 3B , when it is detected that the driver D1 is holding a smartphone, the text "smartphone" can be added and output. This makes it possible to clearly notify the driver D1 to stop using their smartphone in order to stop the specific driving behavior. Furthermore, while FIG. 5 shows an example in which the warning content 212 contains only information to stop the specific driving behavior, the warning content 212 may also include information to notify the driver D1 of a change in the traffic light's indication.

[0064] The warning frequency 213 indicates the frequency at which the driving assistance information is output. When the warning frequency 213 is "once," this means that the driving assistance information is output only once when it is determined that the driver D1 is likely to perform the specific driving immediately before the vehicle C1 stops. On the other hand, when the warning frequency 213 is "repeated until the driver grips the steering wheel," this means that after it is determined that the driver D1 is likely to perform the specific driving immediately before the vehicle C1 stops, the driving assistance information is continuously output until the driver D1 grips the steering wheel. Whether the driver D1 has stopped the specific driving (or specific behavior) can be determined based on the detection result by the behavior detection unit 124.

[0065] The warning volume 214 indicates the volume of the driving assistance information to be output as a voice. As the warning volume 214 changes from "low" to "high," the volume of the driving assistance information to be output as a voice is set to increase.

[0066] 2 and 3 show an example in which driving assistance information is output from both the audio output unit 171 and the display unit 172. However, it may be output from either the audio output unit 171 or the display unit 172. For example, it is possible to set it so that when the level is low, only the audio output unit 171 outputs the information, and as the level increases, both the audio output unit 171 and the display unit 172 output the information. Furthermore, the driving assistance information or information for conveying the information may be output by other transmission means, such as vibration or output from another device.

[0067] [Configuration Example of Driving Assistance Setting Information DB] FIG. 6 is a diagram showing a simplified configuration example of the driving assistance setting information DB 220 stored in the storage unit 150. As shown in FIG.

[0068] The driving assistance setting information DB 220 is a database for managing various information related to the output of driving assistance information. Specifically, personal identification information 222, device identification information 223, output frequency 224, output time 225, output function on / off 226, object detection parameters 227, excluded locations 228, and excluded time periods 229 are stored in the driving assistance setting information DB 220 in association with the driver name 221.

[0069] The driver name 221 is identification information for identifying the driver among the occupants riding in the vehicle C1. For example, "Mr. A" in the driver name 221 corresponds to the driver D1 (see FIGS. 2 and 3). Furthermore, "Mr. B" and "Mr. C" in the driver name 221 correspond to other drivers who have previously rode in the vehicle C1. Note that, for ease of explanation, FIG. 6 shows an example in which names such as "Mr. A" to "Mr. C" are stored in the driver name 221, but other information that can identify each driver (for example, a serial number, a name input by a user operation) may also be stored. For example, each time a new driver rides in the vehicle C1, the new driver's name is added to the driver name 221.

[0070] The personal identification information 222 is information for identifying each person whose driver name is stored in the driver name 221. For example, it is possible to identify a person sitting in the driver's seat of the vehicle C1 based on an image of the interior of the vehicle C1 acquired by the interior image acquisition unit 121. In this case, image identification information is stored in the personal identification information 222. This image identification information is identification information for identifying the person sitting in the driver's seat based on a facial image of the person included in the image of the interior of the vehicle C1 acquired by the interior image acquisition unit 121. Note that a known person recognition technology or personal identification technology can be used to identify the person included in the image (e.g., a facial identification method). For example, every time a driver gets into the vehicle C1, an image of the driver's face (facial image) is acquired, and the driver can be identified based on this facial image and the personal identification information 222. Furthermore, for example, it is assumed that every time a new driver gets into the vehicle C1, a facial image of the new driver is acquired, and this facial image is added to the personal identification information 222. For ease of explanation, FIG. 6 shows an example in which image information showing the face of each person is stored in the personal identification information 222.

[0071] The device identification information 223 is information for identifying devices owned by each person whose driver name is stored in the driver name 221. For example, for an electronic device (e.g., a smartphone) that can be connected via the communication unit 160 using wireless communication, the electronic device can be identified based on device identification information included in the radio waves output from the electronic device. This device identification information is, for example, information that can uniquely identify each device (e.g., terminal identification information, serial information). For example, each time a new driver gets into the vehicle C1, device identification information related to the devices owned by the new driver is acquired, and this device identification information is added to the device identification information 223. Note that, for ease of explanation, FIG. 6 illustrates an example in which MC1 to MC3, etc. are stored in the device identification information 223.

[0072] The output frequency 224 is information indicating the frequency at which driving assistance information is output when it is determined that there is a high possibility of performing a specific driving operation. If the output frequency 224 is "5 times," this means that the driving assistance information is output a maximum of five times from the time when it is determined that there is a high possibility of performing a specific driving operation. The output frequency 224 can also be referred to as an upper limit of output.

[0073] The output time 225 is information indicating the time from when it is determined that there is a high possibility of performing a specific driving to when the driving assistance information is output. If the output time 225 is "2.4 seconds", this means that the driving assistance information is output 2.4 seconds after it is determined that there is a high possibility of performing a specific driving.

[0074] The output function on / off 226 is information indicating whether or not to output driving assistance information. When the output function on / off 226 is "on", it means that the setting is set to output driving assistance information. On the other hand, when the output function on / off 226 is "off", it means that the setting is set not to output driving assistance information.

[0075] The object detection parameters 227 are information indicating parameters for setting the detection accuracy of the object detection process performed by the object detection unit 123 (see FIG. 1 ). In other words, the information indicates parameters for adjusting weighting coefficients in the object detection algorithm. For example, when detecting an object using image recognition (e.g., machine learning), the parameter can be adjusted to adjust the detection probability of the object to be detected. For example, a threshold value for setting the detection accuracy of an object to be detected (e.g., a smartphone) by the object detection unit 123 is stored as a parameter in the object detection parameters 227. Note that the object detection parameters 227 may be set as parameters for each object to be detected by the object detection unit 123, or parameters for all objects to be detected by the object detection unit 123. Note that, for ease of explanation, FIG. 6 illustrates an example in which parameters such as TH1 to TH3 are stored in the object detection parameters 227.

[0076] For example, assume that the object to be detected is a smartphone and the threshold for the smartphone detection probability is 40%. In this case, even if the value calculated by the object detection process indicating that the object is a smartphone is 50%, the object is detected as a smartphone. In this case, for example, an ID card holder may be detected as a smartphone. On the other hand, assume that the object to be detected is a smartphone and the threshold for the smartphone detection probability is 60%. In this case, if the value calculated by the object detection process indicating that the object is a smartphone is 50%, the object is not detected as a smartphone. In this way, increasing the value of the object detection parameter 227 can reduce false object detection and improve object detection accuracy. On the other hand, decreasing the value of the object detection parameter 227 increases the likelihood of detecting the object, but can also increase false object detection.

[0077] The excluded location 228 is location information indicating a location where driving assistance information is not output when it is determined that there is a high possibility of specific driving. The excluded time period 229 is time information indicating a time period where driving assistance information is not output when it is determined that there is a high possibility of specific driving. For example, assume that driver D1 uses vehicle C1 to commute to work. Also assume that vehicle C1 requires an XYZ company ID card to enter the parking lot of driver D1's XYZ company. In this case, it is assumed that driver D1 takes out his XYZ company ID card when he stops at a traffic light near XYZ company. Here, ID cards often have a shape and size similar to a smartphone. Therefore, driver D1's act of taking out his XYZ company ID card may be recognized as driver D1 taking out his smartphone. In this case, it is conceivable that driving assistance information for stopping the smartphone may be output. However, if driving assistance information for stopping the smartphone is output every time driver D1 takes out his / her XYZ Company ID card during his / her daily commute, the output of the driving assistance information may cause driver D1 to feel annoyed and give driver D1 an unpleasant impression. To prevent this, location information (e.g., latitude and longitude) related to "XYZ Company" and its vicinity is stored in excluded location 228, and time information related to the time period (e.g., 8:00 to 8:30) during which driver D1 enters the parking lot of "XYZ Company" is stored in excluded time period 229. This makes it possible to prevent driving assistance information for stopping the smartphone from being output every time driver D1 takes out his / her XYZ Company ID card during his / her daily commute.

[0078] The excluded locations 228 and excluded time periods 229 may be stored based on a user operation, or may be automatically set by the setting unit 145 based on the driver response history information DB 230 (FIG. 7). The setting method for automatic setting by the setting unit 145 will be described in detail with reference to FIG. 7.

[0079] [Configuration Example of Driver Response History Information DB] FIG. 7 is a diagram showing a simplified configuration example of the driver response history information DB 230 stored in the storage unit 150. As shown in FIG.

[0080] The driver response history information DB 230 is a database for managing various information (e.g., negative response information) related to the driver's response to the output of driving assistance information. Specifically, a driver name 231 and a negative response history 232 are associated and stored in the driver response history information DB 230. The driver name 231 corresponds to the driver name 221 shown in FIG. 6 .

[0081] The negative reaction history 232 is history information showing a record of a negative reaction of the driver to the driving support information output at a timing when it was determined that there was a high possibility of the driver performing a specific driving. For example, negative reaction information including date and time information 233, location information 234, and negative information 235 is stored as the negative reaction history 232 for the number of times a negative reaction occurred.

[0082] The date and time information 233 is date and time information indicating the date and time when the driver had a negative reaction. The location information 234 is location information indicating the location of the vehicle C1 when the driver had a negative reaction. The location information 234 stores location information acquired by a location information acquisition unit installed in the vehicle C1.

[0083] The negative information 235 is information indicating the degree of negativity when the driver has a negative reaction. For example, if the degree of negativity can be expressed as a number between 0 and 100 (100 being the most negative), a numerical value between 0 and 100 is stored. The negative information 235 stores a value calculated based on the facial expression and line of sight of the driver detected by the behavior detection unit 124. Note that this negative information can be calculated using known facial expression recognition technology.

[0084] It is also possible to detect the degree of negativity of the driver D1 based on the content of the driver D1's speech. The content of the driver D1's speech can be acquired by a voice acquisition unit (for example, a microphone inside the vehicle (not shown)). For example, it is possible to convert the content of the driver D1's speech into text information (character information) using a known voice recognition technology, and extract the degree of negativity of the driver D1 based on the text information. For example, it is possible to estimate the degree of negativity of the driver D1 by applying natural language processing (for example, a positive / negative analysis method) to the text information. The positive / negative analysis method is a method of analyzing whether the content contained in the text information is positive, negative, or neutral.

[0085] Here, a storage method for storing negative reaction information in the driver reaction history information DB 230 when the driver has a negative reaction will be described. For example, the setting unit 145 monitors the reaction of the driver D1 within a certain time period after the driving assistance information is output from the output unit 170, based on the driver reaction information (e.g., the facial expression and gaze of the driver D1) output from the behavior detection unit 124. For example, if the driver D1 has some kind of reaction to the output of the driving assistance information and the reaction is negative, the setting unit 145 stores negative reaction information related to the output in the driver reaction history information DB 230. In this case, the setting unit 145 may store the negative reaction information related to the output in the driver reaction history information DB 230 on the condition that the negative information indicating the degree of negativity is equal to or greater than a predetermined value. Note that a negative reaction by the driver D1 can be determined based on whether the driver D1's gaze is directed toward the output unit 170 and his facial expression is negative. Furthermore, as described above, the negative information can be calculated based on the facial expression and gaze of the driver D1, the content of the speech of the driver D1, etc. For example, the setting unit 145 acquires the facial expression and gaze of the driver D1 within the certain period of time from the behavior detection unit 124, and if the direction of the gaze is directed toward the output unit 170 (the audio output unit 171 or the display unit 172) and the facial expression is negative (for example, a negative or violent expression), the setting unit 145 can store negative reaction information related to the output in the driver reaction history information DB 230.

[0086] The setting unit 145 also changes the contents of the driving assistance setting information DB 220 (see FIG. 6 ) based on the negative reaction information stored in the driver reaction history information DB 230. For example, when negative reaction information is stored in the driver reaction history information DB 230, the setting unit 145 can adjust the object detection parameter 227 by decreasing the value of the output frequency 224, increasing the value of the output time 225, setting the output function on / off 226 to off, and adjusting the object detection parameter 227 in accordance with the increase in the number of stored negative reaction information. For example, by increasing the value of the object detection parameter 227, it is possible to improve object detection accuracy and reduce false detections. In this way, by reducing the possibility of false detection of an object, it is possible to reduce the possibility of outputting unnatural driving assistance information, thereby preventing discomfort to the driver D1 and improving safety. It is also possible to prevent excessive issuance of driving assistance information due to false detections.

[0087] For example, each time the negative reaction information stored in the driver reaction history information DB 230 is increased by one, the value of the output frequency 224 can be decreased by one, the value of the output time 225 can be increased by 0.2 seconds, and the value of the object detection parameter 227 can be adjusted to increase (i.e., adjustments to reduce false detections). Also, for example, when the number of negative reaction information stored in the driver reaction history information DB 230 reaches a predetermined number (e.g., about 2 to 5), the output function on / off 226 can be set to off.

[0088] Furthermore, for example, the exclusion location 228 and the exclusion time period 229 may be automatically updated based on the date and time information 233 and the location information 234. For example, when a predetermined number (e.g., several) of pieces of negative reaction information are stored in which the time period stored in the date and time information 233 is approximately the same (e.g., a difference of about several minutes to tens of minutes) and the location information stored in the location information 234 is approximately the same (e.g., a difference of about several meters to tens of meters), the setting unit 145 stores new information in the exclusion location 228 and the exclusion time period 229 based on each of the negative reaction information. For example, the setting unit 145 can calculate the average value of the time stored in the date and time information 233 and the average value of the location information stored in the location information 234, and store these respective average values ​​in the exclusion location 228 and the exclusion time period 229.

[0089] For example, assume that a driver uses vehicle C1 for daily commuting. In this case, if vehicle C1 is traveling at the same location at the same time every day, it is expected that driver D1 will perform a similar behavior similar to a specific act (e.g., taking out an ID card while the vehicle is stopped). In such a case, it is determined that there is a high possibility that the driver will perform a specific driving behavior, and driving assistance information is output to driver D1. This may cause driver D1 to feel uncomfortable or distrustful. Therefore, if negative response information is stored in the driver response history information DB 230 for such a behavior, it is possible to prevent driving assistance information from being output for that behavior from the next time onwards by changing the contents of the driving assistance setting information DB 220.

[0090] [Operation Example of Occupant Behavior Detection Processing] Figure 8 is a flowchart showing an example of occupant behavior detection processing in the information processing system 100. This occupant behavior detection processing is executed by the behavior detection unit 124 (see Figure 1) based on the analysis results obtained by the image analysis unit 122 analyzing images of the interior of the vehicle C1 acquired by the interior image acquisition unit 121. This occupant behavior detection processing is constantly executed for each control cycle. This occupant behavior detection processing will be described with appropriate reference to Figures 1 to 7.

[0091] In step S401, the behavior detection unit 124 detects the seating positions of one or more occupants in the vehicle C1. For example, it is possible to detect the presence or absence of occupants sitting in each seat of the vehicle C1 using known person detection technology or face detection technology. Note that other sensors, such as seating sensors or seat belt sensors, may also be used to detect the presence or absence of occupants sitting in each seat of the vehicle C1.

[0092] In step S402, the behavior detection unit 124 detects the face and head of each occupant of the vehicle C1 detected in step S401, as well as their orientation. For example, it is possible to detect the face and head of each occupant of the vehicle C1, as well as their orientation, using a known face recognition technology.

[0093] In step S403, the behavior detection unit 124 detects the skeleton of each occupant of the vehicle C1 detected in step S401. Here, it is assumed that only the hands of the occupants are to be detected. In this case, depending on the position of the detected hand, it may be difficult to determine whether the hand belongs to the occupant in the passenger seat or the driver in the driver's seat. Therefore, skeleton detection is necessary to determine whose skeleton the detected hand belongs to. For example, it is possible to detect the skeleton of each occupant of the vehicle C1 using known skeleton detection technology.

[0094] In step S404, the behavior detection unit 124 detects the hands of each occupant of the vehicle C1 detected in step S401. For example, it is possible to detect the hands of each occupant of the vehicle C1 using a known hand detection technique.

[0095] For example, by sequentially recording the coordinates of the hand positions detected in step S404, it is possible to determine the hand trajectory. Furthermore, future hand movements can also be estimated from past hand movements. Furthermore, when seated in a seat in the vehicle C1, the body is secured by a seat belt, so the range in which each occupant can move their hands is limited. Furthermore, since the locations of the switches and other components installed in the vehicle C1 are fixed, the hand movements of each occupant are often determined to follow multiple patterns within a certain range. Therefore, when the driver D1 moves his / her hand to operate a switch for a certain component, the hand movements of the driver D1 can be estimated. Incidentally, the occupant's behavior may be estimated using artificial intelligence (AI).

[0096] In step S405, the behavior detection unit 124 detects the gaze of each occupant of the vehicle C1 detected in step S401. For example, it is possible to detect the gaze of each occupant of the vehicle C1 using a known gaze detection technique.

[0097] In step S406, the behavior detection unit 124 detects an object held in the hand of each occupant of the vehicle C1 detected in step S401. For example, it is possible to detect the object held in the hand of each occupant using object information (the object and its position) related to the object detected by the object detection unit 123 in step S411 (see FIG. 9 ). For example, if the position of the object detected by the object detection unit 123 overlaps with or is close to the position of the hand of a certain occupant, it is possible to determine that the hand of that occupant is holding the object. Note that it is also possible to detect the object held in the hand of each occupant using other detection information (skeleton, line of sight).

[0098] In step S407, the behavior detection unit 124 determines the behavior of each occupant of the vehicle C1 detected in step S401. For example, the behavior detection unit 124 determines whether the occupant is drinking a drink, reaching into a cup holder, holding a portable electronic device (e.g., a smartphone, a mobile phone, or a tablet), reading something (e.g., a book or a newspaper), reaching out to another seat to look for something, searching for something (including bending down to look for something on the floor), or looking away. These behaviors can be determined based on the detection results of the occupant's behavior and the detection results of objects in the vehicle cabin.

[0099] For example, the behavior of reaching for a cup holder can be determined based on the shape of the cup holder and the path of the occupant's hand. The behavior of reading can be determined based on the shape of a book and the position of the occupant's head. For example, if the occupant's head is close to the book, it can be determined that the occupant is reading. Similarly, it can be determined that the occupant is reading a newspaper. The behavior of bending over to search for something on the floor can be determined when the detected occupant's skeleton appears folded, the occupant's head is facing downward, and the occupant's face disappears from the image. The behavior of looking toward another seat can be determined when the detected occupant's skeleton is twisted toward the other seat. The behavior of looking away can be determined based on the direction of the driver D1's face and the driver D1's line of sight. For example, if the driver D1 is looking to the side rather than forward, it can be determined that the driver D1 is looking away.

[0100] Furthermore, for example, the behavior detection unit 124 can detect at least one of the following: the driver D1 is performing a specific behavior that may induce the driver D1 to look away; the presence of a specific object that may induce the driver D1 to look away around the hands of the driver D1; and the presence of a specific object around the steering wheel 3. In this case, the determination unit 142 can determine that there is a high possibility that the driver D1 will perform a specific driving operation when it is detected that the driver D1 is performing a specific behavior, when it is detected that a specific object is present around the hands of the driver D1, or when it is detected that a specific object is present around the steering wheel 3. In particular, when it is estimated that the stopped vehicle C1 will start moving, and a specific object is detected at a specific location, the determination unit 142 determines, based on transitions of the driver D1's hands and line of sight, whether the driver D1 is likely to hold the specific object in his / her hands and whether the driver D1 is likely to gaze at or operate the specific object. Then, if the judgment unit 142 determines that there is a high possibility that the driver D1 will hold a specific object in his / her hand, or if it determines that there is a high possibility that the driver D1 will look at or operate a specific object, it can determine that there is a high possibility that a specific driving operation will be performed.

[0101] [Operation Example of Vehicle Interior Object Detection Processing] Figure 9 is a flowchart showing an example of vehicle interior object detection processing in the information processing system 100. This vehicle interior object detection processing is executed by the object detection unit 123 (see Figure 1) based on the analysis results obtained by the image analysis unit 122 of an image of the vehicle interior of the vehicle C1 acquired by the vehicle interior image acquisition unit 121. This vehicle interior object detection processing is constantly executed at every control cycle. This vehicle interior object detection processing will be described with appropriate reference to Figures 1 to 8.

[0102] In step S411, the object detection unit 123 detects objects present in the cabin of the vehicle C1 and their positions. For example, known object detection technology can be used to detect objects present in the cabin of the vehicle C1 and their positions. Here, the objects to be detected can be set in advance. For example, objects that an occupant may bring into the cabin of the vehicle C1 and that the occupant may hold can be set in advance. For example, portable electronic devices (e.g., smartphones, mobile phones, tablet devices), books, newspapers, bags, and drinks (e.g., bottled juice, canned juice, and cartoned juice) can be set as detection targets.

[0103] In step S412, the object detection unit 123 detects the position of the steering wheel 3. Here, the position of the steering wheel 3 in the image acquired by the interior image acquisition unit 121 is detected. For example, assume that the vehicle C1 is a vehicle in which the position of the steering wheel 3 can be adjusted up, down, left, and right. In this case, there is a possibility that the position of the steering wheel 3 will be changed by the driver D1. Therefore, the position of the steering wheel 3 is detected in order to determine the relationship between the steering wheel 3 and the driver's hands or other objects. This makes it possible to appropriately detect objects present around the steering wheel 3.

[0104] The above example illustrates the detection of objects and the like present in the cabin of the vehicle C1 based on images acquired by the interior image acquisition unit 121. Here, for an electronic device capable of wireless communication (e.g., a smartphone), detection can be performed using radio waves emitted by the electronic device. For example, assume that direct or indirect wireless communication is possible between the electronic device 30 and the communication unit 160. In this case, the electronic device 30 can be detected based on information exchanged between the electronic device 30 and the communication unit 160. Furthermore, if the information processing system 100 can acquire operation information regarding the operation status of the electronic device 30, it can detect that the driver D1 is operating the electronic device 30 based on the operation information. It is also possible that multiple occupants are riding in the vehicle C1, and each occupant carries an electronic device. In this case, the electronic device 30 of the driver D1 can be identified based on the device identification information 223 (see FIG. 6 ) in the driving assistance setting information DB 220. Furthermore, if the position of each electronic device can be estimated based on radio wave intensity, the electronic device 30 of the driver D1 sitting in the driver's seat can be identified based on the radio wave intensity.

[0105] [Example of Operation of Information Processing System] Figures 10 and 11 are flowcharts showing an example of driving assistance information output processing in the information processing system 100. This driving assistance information output processing is mainly executed by the control unit 140 (see Figure 1) based on a program stored in the storage unit 150 (see Figure 1). This driving assistance information output processing is also constantly executed for each control cycle. This driving assistance information output processing will be described with appropriate reference to Figures 1 to 9.

[0106] 10 and 11 show an example in which the start or stop of the vehicle C1 is estimated based on the indication of a traffic light ahead of the vehicle C1, that is, start permission or start denial.

[0107] In step S501, the control unit 140 acquires various types of vehicle information via the vehicle information acquisition unit 130.

[0108] In step S502, the traffic light detection unit 113 starts a traffic light detection process to detect a traffic light and its indications ahead of the vehicle C1 based on the image ahead of the vehicle C1 acquired by the outside image acquisition unit 111. The detection results of this traffic light detection process are used as appropriate in each determination process (e.g., steps S503 and S505) described below.

[0109] In step S400, the behavior detection unit 124 starts an occupant behavior detection process (see FIG. 8) for detecting the behavior of an occupant of the vehicle C1. The detection results of this occupant behavior detection process are used as appropriate in the various determination processes described below.

[0110] In step S410, the object detection unit 123 starts a vehicle interior object detection process (see FIG. 9 ) to detect an object present in the vehicle interior of the vehicle C1. In this vehicle interior object detection process, the detection accuracy of the object detection unit 123 is changed based on the object detection parameters 227 in the driving assistance setting information DB 220 (see FIG. 6 ). The detection results of this vehicle interior object detection process are used as appropriate in the various determination processes described below.

[0111] In step S503, the estimation unit 141 determines whether a traffic light was detected ahead of the vehicle C1 in step S502. If a traffic light was detected, the process proceeds to step S504. On the other hand, if a traffic light was not detected, the operation of the driving assistance information output process ends.

[0112] In step S504, the control unit 140 determines whether the autonomous driving is off in the vehicle C1. If the autonomous driving is off, the process proceeds to step S505. On the other hand, if the autonomous driving is on, the occupant of the vehicle C1 does not need to drive the vehicle C1, and the need for driving assistance is low, so the operation of the driving assistance information output process is terminated.

[0113] In step S505, the estimation unit 141 determines whether the traffic light ahead detected in step S503 prohibits starting. If the traffic light ahead prohibits starting, the process proceeds to step S506. On the other hand, if the traffic light ahead permits starting, the process proceeds to step S519 shown in FIG. 11.

[0114] In step S506, the estimation unit 141 determines whether the vehicle C1 is traveling based on the vehicle information (e.g., vehicle speed, acceleration) acquired in step S501. If the vehicle C1 is traveling, the process proceeds to step S507. For example, if the traffic light in front of the traveling vehicle C1 changes from green to red, the process proceeds to step S507. On the other hand, if the vehicle C1 is not traveling, i.e., if the vehicle C1 is stopped, the operation of the driving assistance information output process is terminated. Note that, if the traffic light in front of the vehicle C1 changes from start prohibition to start permission while the vehicle C1 is stopped, the process proceeds to step S520 after the determination process of step S519.

[0115] In step S507, the determination unit 142 determines whether or not the driver D1 is not gripping the steering wheel 3 based on the result of the occupant behavior detection process in step S400 and the result of the vehicle interior object detection process in step S410. For example, it is determined whether or not the hands of the driver D1 detected in step S404 (see FIG. 8 ) are gripping the steering wheel 3 whose position was detected in step S412 (see FIG. 9 ). If the driver D1 is not gripping the steering wheel 3, the process proceeds to step S508. On the other hand, if the driver D1 is gripping the steering wheel 3, the process proceeds to step S511.

[0116] In step S508, the determination unit 142 determines whether the driver D1 is engaging in a specific behavior that could lead to danger. Here, the specific behavior refers to a specific behavior (or actions) other than driving-related behavior that could lead to danger. Examples of the specific behavior include operating various electronic devices, staring at various electronic devices, eating and drinking, reading a book, etc. If the driver D1 is not gripping the steering wheel 3 and is engaging in a specific behavior, the determination unit 142 determines that there is a high possibility that the driver D1 will engage in specific driving, and determines the risk level to be level 4 (the highest level). Then, the process proceeds to step S509.

[0117] On the other hand, even if the driver D1 is not engaging in a specific behavior that could lead to danger, there is a possibility that the driver D1 will engage in some action other than driving when the driver D1 is not gripping the steering wheel 3. Therefore, when the driver D1 is not engaging in a specific behavior, the determination unit 142 determines that there is a high possibility that the driver D1 will engage in specific driving, and determines the risk level as Level 3. Then, the process proceeds to step S510. In other words, when the driver D1 is not gripping the steering wheel 3, even when it is determined that the driver D1 is not engaging in a specific behavior that could lead to danger, it is considered difficult for the driver D1 to quickly take some kind of risk avoidance action, and therefore the risk level is determined to be Level 3.

[0118] In step S509, the determination unit 143 determines the warning level for the driver D1 to be WL4 (see FIG. 5 ). For example, if the driver D1 is not gripping the steering wheel 3 and is performing a specific behavior that could lead to danger (steps S507 and S508), it is considered highly likely that the driver D1 will continue the specific behavior (specific action) and perform specific driving until the vehicle C1 stops. Therefore, if this determination is made, the determination unit 143 determines driving assistance information (warning level WL4) including content to prompt the driver D1 to immediately stop the specific behavior. In this case, the determination unit 143 may determine to include information regarding the specific behavior determined in step S508 in the driving assistance information. For example, if the smartphone operation behavior is determined to be a specific behavior in step S508, it is determined to output information instructing the driver D1 to stop using the smartphone.

[0119] In step S510, the determination unit 143 determines the warning level for the driver D1 to be WL3 (see FIG. 5).

[0120] In step S511, the determination unit 142 determines whether the driver D1 is performing a specific behavior. The criteria for determining this specific behavior are the same as those in step S508. If the driver D1 is performing a specific behavior, the process proceeds to step S513. That is, if the driver D1 is holding the steering wheel 3 but is performing a specific behavior that could lead to danger, there is a possibility that the driver D1 will continue the specific behavior. Therefore, in this case, the determination unit 142 determines that there is a high possibility that the driver D1 will perform a specific driving operation and determines the risk level as level 2. That is, if the driver D1 is holding the steering wheel 3, even if it is determined that the driver D1 is performing a specific behavior that could lead to danger, it is considered that the driver D1 can quickly take some kind of risk avoidance action, and therefore the risk level is determined as level 2. On the other hand, if the driver D1 is not performing a specific behavior, the process proceeds to step S512.

[0121] In step S512, the determination unit 142 determines whether or not there is a specific object around the steering wheel 3 (including when it is in contact with the steering wheel 3). Here, the specific object refers to an object that may be used to perform the specific behavior described above, such as various electronic devices, food and drink, books, etc. For example, if a smartphone is placed on the steering wheel 3 in a state where it can be operated, it is determined that there is a specific object around the steering wheel 3. If there is a specific object around the steering wheel 3, the process proceeds to step S514. In other words, even if the driver D1 is holding the steering wheel 3 and is not performing a specific behavior that could lead to danger, if there is a specific object around the steering wheel 3, there is a possibility that the driver D1 will operate or look at the specific object. Therefore, in this case, the determination unit 142 determines that there is a high possibility that the driver D1 will perform the specific driving and determines the risk level to be level 1 (the lowest level).

[0122] On the other hand, if there is no specific object around the steering wheel 3, the determination unit 142 determines that the driver D1 is unlikely to perform the specific driving, and ends the operation of the driving assistance information output process.

[0123] In step S513, the determination unit 143 determines the warning level for the driver D1 to be WL2 (see FIG. 5 ). In this case, similar to step S509, the determination unit 143 may determine that the information on the specific behavior determined in step S511 is to be included in the driving assistance information.

[0124] In step S514, the determination unit 143 determines the warning level for driver D1 to be WL1 (see FIG. 5 ). In this case, the determination unit 143 may determine to include information about the specific object determined in step S512 in the driving assistance information. For example, if it is determined in step S512 that a smartphone is placed on the steering wheel 3, the determination unit 143 determines to output information indicating that the smartphone should not be operated.

[0125] In this way, when the driver D1 is not gripping the steering wheel 3, it is possible to increase the risk level indicating the degree of likelihood of the specific driving being performed, compared to when the driver D1 is gripping the steering wheel 3. Also, when the driver D1 is performing a specific behavior that could lead to danger, it is possible to increase the risk level indicating the degree of likelihood of the specific driving being performed, compared to when the driver D1 is not performing a specific behavior that could lead to danger. Also, when a specific object is present around the steering wheel 3, it is possible to increase the risk level indicating the degree of likelihood of the specific driving being performed, compared to when the specific object is not present around the steering wheel 3. Also, in steps S509, S510, S513, and S514, the determination unit 143 determines warning levels WL1 to WL4 (see FIG. 5) for the driver D1 based on the risk levels 1 to 4 determined by the determination unit 142.

[0126] In step S515, the output control unit 144 causes the output unit 170 to output driving assistance information based on the warning level determined in steps S509, S510, S513, and S514 and the driver's response history of the driver D1. Specifically, the output control unit 144 outputs driving assistance information corresponding to the warning level determined in steps S509, S510, S513, and S514 based on the contents of the output frequency 224 and the output time 225 in the driving assistance setting information DB 220 (see FIG. 6 ).

[0127] However, if "off" is stored in the output function on / off 226 of the driving assistance setting information DB 220, the output control unit 144 does not output the driving assistance information and ends the operation of the driving assistance information output process. Furthermore, the output control unit 144 compares the vehicle information (location information of the vehicle C1) acquired in step S501 with the location information stored in the excluded location 228 of the driving assistance setting information DB 220, and determines whether the vehicle C1 is traveling near the excluded location. Then, if the vehicle C1 is traveling near the excluded location and the current time is included in the time zone stored in the excluded time zone 229, the output control unit 144 ends the operation of the driving assistance information output process without outputting the driving assistance information.

[0128] Furthermore, when the device identification information of the electronic device 30 possessed by the driver D1 is stored in the device identification information 223 of the driving assistance setting information DB 220, the output control unit 144 may transmit control information for outputting the driving assistance information to the electronic device 30, and cause the electronic device 30 to output the driving assistance information. In this case, the driving assistance information may be output from the electronic device 30 only when the electronic device 30 possessed by the driver D1 is related to the specific behavior determined in step S508 or step S511, or when the electronic device 30 is determined as a specific object in step S512. For example, as shown in FIG. 3B , it is possible to cause the electronic device 30 to output the driving assistance information.

[0129] In step S516, the setting unit 145 acquires driver response information relating to the facial expression and line of sight of the driver D1 detected by the behavior detection unit 124 until a certain time has elapsed since the driving assistance information was output in step S515.

[0130] In step S517, the setting unit 145 determines whether the driver D1 has reacted negatively to the output of the driving assistance information based on the driver response information acquired in step S516. This negative response can be determined in the same manner as in the determination process described in FIG. 7. If the driver D1 has reacted negatively to the output of the driving assistance information, the process proceeds to step S518. On the other hand, if the driver D1 has not reacted negatively to the output of the driving assistance information, the operation of the driving assistance information output process is terminated.

[0131] In step S518, the setting unit 145 stores negative reaction information indicating that the driver D1 reacted negatively to the output of the driving assistance information in the driver reaction history information DB 230 (see FIG. 7). This negative reaction information is stored in the same manner as the storage process described in FIG. 7. Furthermore, the setting unit 145 changes the contents of the driving assistance setting information DB 220 (see FIG. 6) based on the negative reaction information stored in the driver reaction history information DB 230, as necessary.

[0132] In this way, when the traffic light in front of the vehicle C1 indicates that departure is not permitted while the vehicle C1 is traveling, the driver D1 decelerates the vehicle C1 and stops it in front of the traffic light. The system detects the driver D1's behavior before the vehicle C1 stops, and if the driver D1 is performing a specific action other than driving (in the case of specific driving) even though the vehicle C1 has not completely stopped, or if there is a possibility that the driver D1 will perform a specific action, it issues a warning to the driver D1 to stop the specific action until the vehicle C1 has completely stopped. This allows the driver D1 to safely stop and wait for the traffic light to change direction.

[0133] 11 , the estimation unit 141 determines whether the vehicle C1 is stopped based on the vehicle information (e.g., vehicle speed, acceleration) acquired in step S501. If the vehicle C1 is stopped, the process proceeds to step S520. For example, if a traffic light ahead of the stopped vehicle C1 changes from red to green, the process proceeds to step S520. On the other hand, if the vehicle C1 is not stopped, i.e., if the vehicle C1 is traveling, the operation of the driving assistance information output process ends.

[0134] The determination process of step S520 corresponds to the determination process of step S507. The determination processes of steps S521 and S525 correspond to the determination processes of steps S508 and S511. The determination process of step S526 corresponds to the determination process of step S512. Therefore, some of the descriptions of these determination processes will be omitted.

[0135] In step S522, the determination unit 142 determines whether the driver D1 has released the brake pedal or whether the driver D1 has depressed the accelerator pedal. If the driver D1 has released the brake pedal or depressed the accelerator pedal, the process proceeds to step S524. That is, when it is estimated that the vehicle C1 is about to start, if the driver D1 is not gripping the steering wheel 3, is performing a specific behavior that could lead to danger, and then releases the brake pedal or depresses the accelerator pedal, there is a possibility that the vehicle C1 will start with the driver D1 not gripping the steering wheel 3 but continuing the specific behavior. In this case, the determination unit 142 determines that there is a very high possibility that the driver D1 will perform the specific driving and sets the risk level at level 4 (the highest level).

[0136] On the other hand, if the driver D1 is depressing the brake pedal, i.e., if the accelerator pedal is not being depressed, the process proceeds to step S523. Here, when it is estimated that the vehicle C1 is about to start, if the driver D1 is not gripping the steering wheel 3 and is performing a specific behavior that could lead to danger, but the driver D1 is depressing the brake pedal but not the accelerator pedal, the possibility of the vehicle C1 starting immediately is low. In this case, the determination unit 142 determines that there is a high possibility that the specific driving will be performed, but because the possibility of the vehicle C1 starting immediately is low, the determination unit 142 determines that the risk level is level 2.

[0137] Furthermore, if the driver D1 is gripping the steering wheel 3 (step S520) but is performing a specific behavior that could lead to danger (step S525), there is a possibility that the driver D1 will continue the specific behavior while gripping the steering wheel 3 and preparing to start the vehicle C1. In this case, the determination unit 142 determines that there is a very high possibility that the driver D1 will perform the specific driving, and determines the risk level to be level 4 (the highest level).

[0138] In step S523, the determination unit 143 determines the notification level for the driver D1 to be NL2 (see FIG. 4 ). In this case, similar to step S509, the determination unit 143 may determine that the information on the specific behavior determined in step S521 is to be included in the driving assistance information.

[0139] In step S524, the determination unit 143 determines the notification level for the driver D1 to be NL4 (see FIG. 4 ). In this case, similar to step S509, the determination unit 143 may determine that the information on the specific behavior determined in step S521 or S525 is to be included in the driving assistance information.

[0140] Furthermore, if the driver D1 is not engaging in a specific behavior that could lead to danger (steps S521 and S525), the risk level is determined based on whether a specific object is present around the steering wheel 3, regardless of whether the driver D1 is gripping the steering wheel 3 (step S520). That is, even if the driver D1 is not engaging in a specific behavior that could lead to danger, if a specific object is present around the steering wheel 3, the driver D1 may operate or gaze at the specific object while the vehicle is stopped. Therefore, in step S526, the determination unit 142 determines that there is a high possibility that the driver D1 will perform the specific driving and determines the risk level to be Level 3. On the other hand, if there is no specific object around the steering wheel 3, there is a low possibility that the driver D1 will operate or gaze at some object while the vehicle is stopped. Therefore, in step S526, the determination unit 142 determines that there is a high possibility that the driver D1 will perform the specific driving, but determines that the risk is very low and determines the risk level to be Level 1.

[0141] In step S527, the determination unit 143 determines the notification level for the driver D1 to be NL3 (see FIG. 4 ). In this case, similar to step S514, the determination unit 143 may determine that the information about the specific object determined in step S526 is to be included in the driving assistance information.

[0142] In step S528, the determination unit 143 determines the notification level for driver D1 to be NL1 (see FIG. 4).

[0143] In this way, when it is estimated that the vehicle C1 is about to start, if the driver D1 releases the brake pedal and depresses the accelerator pedal, it is possible to determine that the driver D1 is more likely to perform the specific driving mode and to increase the risk level, compared to when the driver D1 does not release the brake pedal or when the driver D1 does not depress the accelerator pedal. Also, in steps S523, S524, S527, and S528, the determination unit 143 determines the notification level NL1 to NL4 (see FIG. 4) for the driver D1 based on the risk levels 1 to 4 determined by the determination unit 142.

[0144] In this way, in this embodiment, when it is estimated that the vehicle C1 is stopping or starting, it is possible to determine whether or not the driver D1 is likely to perform a specific driving operation based on at least one of whether or not the driver D1 is holding the steering wheel 3, whether or not the driver D1 is performing a specific behavior that could lead to danger, whether or not a specific object is present around the steering wheel 3, and whether or not the driver D1 has released the brake pedal or whether or not the driver D1 has depressed the accelerator pedal.

[0145] Furthermore, when it is estimated that the vehicle C1 will stop or start, it is possible to change the output content of the driving assistance information over multiple stages depending on the behavior of the driver D1 within a certain period of time and the state of the vehicle C1 (depressing the brake pedal, depressing the accelerator pedal, etc.). In other words, it is possible to appropriately determine the level of the driving assistance information based on the behavior of the driver D1 within a certain period of time, etc. This makes it possible to prompt the driver D1 to start driving more safely.

[0146] In the processes of steps S512 and S526, an example of determining whether a specific object exists around the steering wheel 3 has been described. However, it may also be determined whether a specific object exists around the driver D1's hands. Furthermore, it may be determined whether a specific object exists regardless of the location around the steering wheel 3. For example, the object detection unit 123 constantly detects in advance whether a specific object (e.g., a smartphone) that may induce the driver D1 to look away is present in the cabin of the vehicle C1 and the location of the specific object. Then, if the specific object is present in a specific location (e.g., within the driver D1's reach), the determination unit 142 determines the relationship between the driver D1 and the specific object. For example, when the traffic light ahead of the vehicle C1 changes to a start permission signal, it can determine whether the driver D1 will reach out and hold the specific object based on the driver D1's hand movements, line of sight, and the like. Alternatively, it can determine whether the driver D1 will operate or gaze at the specific object. In this case, if it is determined that the driver D1 reaches out and picks up the specific object, or if it is determined that the driver D1 operates or looks at the specific object, there is a possibility that the driver D1 will pick up the specific object or look away. Therefore, in such a case, the determination unit 142 determines that the driver D1 is likely to engage in specific driving. In this case, the determination unit 143 can determine driving assistance information to encourage the driver D1 not to pick up the specific object or look away, based on the determination result by the determination unit 142. It is also assumed that the driver D1 uses a fixed smartphone instead of a navigation device. In this case, since the driver D1 often looks at the smartphone, it is possible to determine driving assistance information to encourage the driver D1 to concentrate on operating the smartphone and not look away. This makes it possible to encourage the driver D1 to start and drive safely.

[0147] In step S529, the output control unit 144 causes the output unit 170 to output driving assistance information based on the notification level determined in steps S523, S524, S527, and S528 and the driver's response history of driver D1. Specifically, the output control unit 144 outputs driving assistance information corresponding to the notification level determined in steps S523, S524, S527, and S528 based on the contents of the output frequency 224 and the output time 225 in the driving assistance setting information DB 220 (see FIG. 6 ). Note that the method of outputting driving assistance information based on the driver's response history of driver D1 is the same as in step S515, and therefore will not be described here.

[0148] [Examples of Traffic Lights Other Than Green, Yellow, and Red Traffic Lights] The above describes an example of estimating the start and stop of the vehicle C1 based on the indications of a traffic light with three colors: green, yellow, and red. Note that there are traffic lights other than those with three colors: green, yellow, and red. For this reason, Figure 12 shows examples of traffic lights other than those with three colors: green, yellow, and red.

[0149] Fig. 12 is a simplified diagram showing the external configuration of an arrow-type traffic light AS1. The arrow-type traffic light AS1 is a traffic light equipped with blue arrows AS15 to AS17 in addition to a green light AS11, a yellow light AS12, and a red light AS13. For ease of explanation, in Fig. 12, the circles representing the blue light AS11, the yellow light AS12, and the red light AS13 are indicated with letters of the corresponding color.

[0150] For example, assume that there is an arrow-type traffic light AS1 ahead of vehicle C1, and that traffic light AS1 is displaying a red light AS13. Even in this case, when any of the green arrows AS15 to AS17 is displayed, vehicle C1 can proceed in the direction of the displayed arrow, regardless of the signal of red light AS13. In other words, when vehicle C1 proceeds in the direction of any of the green arrows AS15 to AS17, it can be assumed that departure is permitted when the green arrow AS15 to AS17 in that direction is displayed.

[0151] When an arrow-type traffic light AS1 is ahead of the vehicle C1, the direction of travel of the vehicle C1 can be estimated based on the route to the destination set in the navigation device, or based on the operation of a turn signal switch to flash the turn signal.

[0152] Although not shown in the figures, the present invention can be applied to traffic lights that flash one color (for example, a traffic light that flashes red or yellow) and traffic lights installed in areas where construction work is underway (for example, a traffic light with two colors, red and green). For example, in the case of a traffic light that flashes red, it is possible to presume that departure is not permitted while the vehicle C1 is traveling, and that departure is permitted after the vehicle C1 has stopped temporarily.

[0153] [Example of Estimating Vehicle Start and Stop Based on Criteria Other Than Traffic Lights] The above describes an example of estimating the start and stop of vehicle C1 based on the indication of a traffic light. However, it is possible to estimate the start and stop of vehicle C1 regardless of whether a traffic light is present. Therefore, below, an example of estimating the start and stop of vehicle C1 based on criteria other than traffic lights is described. Furthermore, in the example described below, instead of the traffic light detection unit 113 shown in FIG. 1 , an object detection unit capable of detecting various objects (e.g., predetermined signs, other vehicles, people, crosswalks) present in front of vehicle C1 is provided. In this case, objects can be detected using known image recognition technology.

[0154] [Example of Estimating Vehicle Start] First, an estimation method for estimating the start of the vehicle C1 based on criteria other than traffic lights will be described.

[0155] For example, in the case of a vehicle C1 that is stopped with the shift lever in the P range position, it is possible to estimate that the vehicle C1 will start moving at the time when the driver D1 puts the shift lever in the D range position.

[0156] Furthermore, for example, when the vehicle C1 is stopped with the shift lever in the D range and the driver D1 depressing the brake pedal, it is possible to estimate the start of the vehicle C1 at the timing when the driver D1 releases the brake pedal. Alternatively, it is also possible to estimate the start of the vehicle C1 at the timing when the driver D1 releases the brake pedal and depresses the accelerator pedal. For example, this estimation is possible when the vehicle C1 is waiting in a traffic jam or when the vehicle C1 is temporarily stopped on the shoulder of the road.

[0157] In this way, it is possible to estimate the start of the vehicle C1 based on the operation of each of the shift lever, brake pedal, accelerator pedal, etc. In this case, the estimation unit 141 estimates the start of the vehicle C1 based on vehicle information (e.g., information related to the shift lever, brake pedal, accelerator pedal) acquired via the vehicle information acquisition unit 130.

[0158] Note that artificial intelligence (AI) may be used to estimate the start of the vehicle C1. For example, it is possible to learn the behavior of the driver D1 when the vehicle C1 starts, and use the learning results to estimate the start of the vehicle C1. For example, assume that the driver D1 has the habit of checking his face and hair while looking at the rearview mirror 5 immediately before setting off with the vehicle C1. In this case, it is possible to estimate the start of the vehicle C1 immediately after the driver D1 checks his face and hair while looking at the rearview mirror 5.

[0159] [Example of Estimating Whether a Vehicle is Stopped] Next, an estimation method for estimating whether the vehicle C1 is stopped based on criteria other than traffic lights will be described.

[0160] For example, assume that a destination is set in a navigation device and the vehicle C1 is traveling along a route to the destination. In this case, it is possible to estimate that the vehicle C1 will stop when the vehicle C1 approaches the destination. For example, when the vehicle C1 is approaching the destination and the distance between the vehicle C1 and the destination is less than a predetermined value, for example, several meters to several tens of meters, it is possible to estimate that the vehicle C1 will stop. In this case, the estimation unit 141 estimates that the vehicle C1 will stop based on the relationship between the destination set in the navigation device and the current location of the vehicle C1. Note that the current location of the vehicle C1 can be obtained based on location information obtained by a location information acquisition unit installed in the vehicle C1.

[0161] Also, for example, assume that the vehicle C1 is traveling on a route to a registered location (e.g., home, office). In this case, it is possible to estimate that the vehicle C1 has stopped when the vehicle C1 approaches the registered location. For example, it is possible to estimate that the vehicle C1 has stopped when the vehicle C1 reaches a position several meters to several tens of meters from the registered location.

[0162] For example, location information regarding the registered location is stored in association with the map information DB of the storage unit 150. Then, it is possible to determine whether the vehicle C1 is approaching the registered location based on the location of the registered location stored in the map information DB and the location information acquired by a location information acquisition unit installed in the vehicle C1. For example, when the vehicle C1 is approaching the registered location and the distance between the vehicle C1 and the registered location is less than a predetermined value, for example, several meters to several tens of meters, it is possible to estimate that the vehicle C1 will stop.

[0163] It is also assumed that the vehicle C1 enters a parking lot to stop. In this case, it is possible to estimate that the vehicle C1 will be stopped at the time when the vehicle C1 enters the parking lot. The location of the parking lot can be determined based on the detection process by the object detection unit described above or the map information DB in the storage unit 150.

[0164] Furthermore, for example, if a predetermined sign (e.g., a stop sign) is present ahead of the vehicle C1, the vehicle C1 needs to stop in front of the sign. Therefore, if a predetermined sign is present ahead of the vehicle C1, it is possible to estimate that the vehicle C1 will be stopped when the vehicle C1 approaches the sign and reaches a predetermined position (e.g., a stop line, in front of an intersection). The presence or absence of a predetermined sign ahead of the vehicle C1 and the predetermined position, such as a stop line, can be detected by the object detection unit described above. Alternatively, the presence or absence of a predetermined sign ahead of the vehicle C1 and the predetermined position, such as a stop line, can be determined based on predetermined information (e.g., road information) in the map information DB stored in the memory unit 150 and location information acquired by a location information acquisition unit installed in the vehicle C1.

[0165] In addition, if there is a predetermined sign ahead of the vehicle C1 and it is estimated that the vehicle C1 will stop at the above-mentioned timing, it is estimated that the vehicle C1 will start moving after the driver D1 has checked for safety after stopping. Therefore, it is possible to estimate the start of the vehicle C1 in this case as well.

[0166] Furthermore, for example, if there is a crosswalk ahead of the vehicle C1 and a person waiting to cross the crosswalk, the vehicle C1 needs to stop in front of the crosswalk. Therefore, if there is a crosswalk and a person crossing ahead of the vehicle C1, it is possible to estimate that the vehicle C1 will stop when the vehicle C1 approaches the crosswalk and reaches a predetermined position (e.g., in front of the crosswalk). The presence or absence of a crosswalk ahead of the vehicle C1 and a person crossing the crosswalk can be detected by the object detection unit described above. Alternatively, the presence or absence of a crosswalk ahead of the vehicle C1 can be determined based on predetermined information (e.g., road information) in the map information DB stored in the memory unit 150 and location information acquired by a location information acquisition unit installed in the vehicle C1.

[0167] If there is a crosswalk and a person crossing the crosswalk ahead of the vehicle C1 and it is estimated that the vehicle C1 will stop at the timing described above, it is estimated that the vehicle C1 will start moving after the driver D1 has checked for safety after stopping. Therefore, it is possible to estimate the start of the vehicle C1 in this case as well.

[0168] Furthermore, for example, if there is a construction site ahead of the vehicle C1, the vehicle C1 may stop before the construction site. For example, if there is a construction site ahead of the vehicle C1 and the vehicle C1 is instructed to stop by some means (e.g., a traffic guide or traffic guidance equipment), the vehicle C1 must stop based on the instruction from that means. Therefore, if the vehicle C1 approaches the construction site and is instructed to stop by some means, it is possible to estimate the vehicle C1's stopping at the timing of the instruction. Note that the presence or absence of a construction site ahead of the vehicle C1 and the means for instructing the vehicle C1 to stop can be detected by the object detection unit described above. Alternatively, the presence or absence of a construction site ahead of the vehicle C1 can be determined based on traffic information, road information, etc., obtainable from an external device (e.g., a traffic information server) via the network 20.

[0169] Furthermore, for example, if a large number of vehicles are stopped on the road ahead of the vehicle C1, it is estimated that a traffic jam has occurred ahead of the vehicle C1. In this case, there is a possibility that the vehicle C1 will stop just before the location of the traffic jam. Therefore, if a traffic jam has occurred ahead of the vehicle C1, it is possible to estimate that the vehicle C1 will be stopped when the vehicle C1 approaches the location of the traffic jam and reaches a predetermined position (for example, just before the last vehicle in the traffic jam). Note that the presence or absence of a traffic jam ahead of the vehicle C1 can be detected by the object detection unit described above. Alternatively, it can be determined based on traffic information, road information, etc. that can be obtained from an external device (for example, a traffic information server) via the network 20.

[0170] Furthermore, if schedule information related to the driver D1 is set in the vehicle C1, the vehicle C1's stop may be estimated based on the schedule information. For example, if the schedule information includes a scheduled time for a location, it is possible that the vehicle C1 will move close to the location at a time close to the scheduled time. In such a case, it is possible to estimate that the vehicle C1 will stop when the vehicle C1 is approaching the location and the distance between the vehicle C1 and the location is less than a predetermined value, for example, a few meters to several tens of meters.

[0171] [Configuration example of notification level list] Fig. 13 is a diagram showing a simplified configuration example of the notification level list 240 stored in the storage unit 150. Note that the notification level list 240 is obtained by changing the notification content 202 of the notification level list 200 shown in Fig. 4, and is substantially the same as the notification level list 200 except for the notification content 242. Therefore, the following description will focus on the differences from the notification level list 200.

[0172] The notification level list 240 is a list showing the level of driving assistance information to be output when it is determined that the driver D1 is likely to perform a specific driving operation after the vehicle C1 starts traveling, in a case where the vehicle C1 is estimated to start from a stopped state based on criteria other than traffic lights. A specific method for determining the notification level will be described in detail with reference to FIG. 15 .

[0173] FIG. 13 shows an example in which the notification level 241 is set to four levels depending on the behavior of the driver D1. Specifically, a level at which it is estimated that there is a risk of an accident due to the behavior of the driver D1 but the degree of the risk is low is set to "NL11." Furthermore, "NL12" to "NL14" are set depending on the level at which it is estimated that the behavior of the driver D1 is likely to increase the risk of an accident. In other words, the level at which it is estimated that there is a very high risk of an accident due to the behavior of the driver D1 is "NL14." Note that, although an example in which four levels are set is shown in this embodiment, two, three, five or more levels may be set.

[0174] Furthermore, the notification level 241 is associated with and set to notification content 242, notification frequency 243, and notification volume 244.

[0175] [Configuration example of warning level list] For a list of warning levels of driving assistance information that is output when it is estimated that the traveling vehicle C1 will stop based on criteria other than traffic lights, it is possible to use the warning level list 210 shown in Fig. 5. Note that these are examples of lists of notification levels or warning levels of driving assistance information that are output when it is estimated that the vehicle C1 will start or stop, and other notification level lists or other warning level lists may also be used.

[0176] [Example of operation of information processing system] Figures 14 and 15 are flowcharts showing an example of driving assistance information output processing in the information processing system 100. This driving assistance information output processing is a partial modification of the driving assistance information output processing shown in Figures 10 and 11, and shows an example of a case where the start or stop of the vehicle C1 is estimated based on criteria other than traffic lights. Therefore, in the following, parts that are common to the driving assistance information output processing shown in Figures 10 and 11 are assigned the same reference numerals, and some of the description thereof will be omitted. Furthermore, this driving assistance information output processing will be described with appropriate reference to Figures 1 to 13.

[0177] In step S600, the object detection unit starts an object detection process to detect an object present ahead of the vehicle C1. The detection results of this object detection process are used as appropriate in the various determination processes described below.

[0178] In step S601, the estimation unit 141 determines the state of the vehicle C1 based on the vehicle information (e.g., vehicle speed, acceleration) acquired in step S501. Specifically, it is determined whether the vehicle C1 is moving or stopped. If the vehicle C1 is moving, the process proceeds to step S602. On the other hand, if the vehicle C1 is stopped, the process proceeds to step S603 shown in FIG. 15.

[0179] In step S602, the estimation unit 141 determines whether or not it is estimated that the vehicle C1 is stopped. The estimation method described above can be used to estimate that the vehicle C1 is stopped. If it is estimated that the vehicle C1 is stopped, the process proceeds to step S507. On the other hand, if it is not estimated that the vehicle C1 is stopped, the operation of the driving assistance information output process is terminated.

[0180] In step S603 shown in Fig. 15, the estimation unit 141 determines whether or not the start of the vehicle C1 has been estimated. The above-mentioned estimation methods can be used as a method for estimating the start of the vehicle C1. If the start of the vehicle C1 has been estimated, the process proceeds to step S520. On the other hand, if the start of the vehicle C1 has not been estimated, the operation of the driving assistance information output process is terminated.

[0181] The determination processes in steps S604 to S607 correspond to the determination processes in steps S523, S524, S527, and S528 shown in Fig. 11. That is, one of the notification levels NL1 to NL4 is determined from the notification level list 240 shown in Fig. 13.

[0182] 14 and 15 show an example in which it is estimated in step S602 that the vehicle C1 is stopped and in step S603 that the vehicle C1 is starting based on criteria other than the traffic light. However, it is also possible to estimate in step S602 that the vehicle C1 is stopped and in step S603 that the vehicle C1 is starting based on an instruction from the traffic light.

[0183] [Example of Determining Output Content of Driving Assist Information Based on Behavior of Occupant Other Than the Driver] The above describes an example of determining the output content of driving assist information based on the behavior of the driver D1, etc. Here, if an occupant other than the driver D1 is riding in the passenger seat or the like of the vehicle C1, it is assumed that the occupant may provide some kind of driving assistance to the driver D1. In addition, there is a possibility that an exchange between the driver D1 and another occupant may be determined to be a specific action. Therefore, an example of determining the output content of driving assist information based on the behavior of an occupant other than the driver D1 will be described.

[0184] As shown in FIG. 8 , the behavior detection unit 124 can detect the presence or absence of an occupant sitting in each seat of the vehicle C1. The behavior detection unit 124 can also detect the behavior of each occupant (e.g., face, head, orientation thereof, and gaze). When the traffic light ahead of the vehicle C1 changes to permit departure and the vehicle C1 is stopped, the determination unit 143 acquires the facial orientation and gaze of each occupant (occupant other than the driver D1) detected by the behavior detection unit 124. In steps S523, S524, S527, and S528, if the determination unit 143 detects that at least one of the occupants was looking toward the traffic light ahead of the vehicle C1 before the output of the driving assistance information, the determination unit 143 determines to delay the output of the driving assistance information or to stop the output of the driving assistance information. That is, when a passenger other than the driver D1 certainly notices the change in the traffic light, there is a high possibility that the passenger will provide driving assistance to the driver D1, so the timing of outputting the driving assistance information is delayed or the output of the driving assistance information is stopped. Note that if the passenger utters a predetermined phrase such as "It's green" to the driver D1 before the output of the driving assistance information, it is possible to further lengthen the delay time or stop the output of the driving assistance information. This makes it possible to reduce the discomfort that the output of the driving assistance information causes to the driver D1.

[0185] Here, even when an interaction determined to be a specific behavior occurs between the driver D1 and another occupant, it is assumed that the interaction only continues temporarily. For example, it is assumed that the other occupant may perform an action such as handing an object (an Electronic Toll Collection System (ETC) card, a bag, etc.) to the driver D1. For example, it is assumed that the driver D1 realizes that he or she forgot to insert the ETC card when the vehicle C1 is stopped in front of a traffic light. In this case, the driver D1 requests the ETC card from the other occupant, and the ETC card is handed to him or her by the other occupant. Similarly, it is assumed that the driver D1 requests a wallet, a bag, etc. from the other occupant when the vehicle C1 is stopped in front of a toll entrance. In this case, the driver D1 receives the requested item (wallet, bag, etc.) from the other occupant.

[0186] Although these actions of the driver D1 may be determined to be specific actions, it is considered unlikely that the actions will be continued once they are completed. In other words, for actions that are only performed temporarily and do not continue, even if they are determined to be specific actions, it is possible to stop the output of driving assistance information or lower the level of the output content of the driving assistance information. Specifically, in steps S509, S510, S513, S514, S523, S524, S527, S528, and S604 to S607, the decision unit 143 acquires the actions of each occupant (the driver D1 and other occupants) detected by the behavior detection unit 124. Then, if the decision unit 143 determines, before outputting the driving assistance information, that the driver D1's action is only performed temporarily and does not continue, it decides to stop the output of the driving assistance information or lower the output content of the driving assistance information. This makes it possible to reduce the discomfort caused to the driver D1 and other occupants by the output of the driving assistance information.

[0187] In this way, it is possible to change the output content of the driving assistance information based on the behavior of other occupants of the driver D1 of the vehicle C1.

[0188] Although the above describes an example in which the operation assistance information is output from at least one of the audio output unit 171, the display unit 172, and the electronic device 30, the operation assistance information may be output from another output unit. For example, the operation assistance information may be displayed on a HUD realized in the windshield 4.

[0189] [Example of Effect of the Present Embodiment] As described above, in the present embodiment, when a traffic light ahead of a stopped vehicle C1 changes and the vehicle C1 is about to start, if there is a possibility that the driver D1 will continue the specific behavior that he or she was performing while waiting at the traffic light, driving assistance information is output to stop the specific behavior. For example, when the vehicle C1 transitions from a waiting state to a running state, it is possible to prevent the vehicle C1 from starting to run while the driver D1 is still performing a specific behavior that could lead to danger or while a risk factor remains. Also, for example, assume that the driver D1 is performing an action such as looking at the electronic device 30 while stopped at a traffic light. In this case, the behavior of the driver D1 during the stopped state from the waiting state to the time the traffic light changes to green is detected, and if the driver D1 is still performing the action when the traffic light changes to green and the driver D1 grips the steering wheel 3 before starting the vehicle C1, driving assistance information is output to stop the driver D1 from looking at the electronic device 30. Furthermore, when the driver D1 grips the steering wheel 3 before starting the vehicle C1, the system detects whether or not there is a specific object on (or around) the driver D1's hands or on (or around) the steering wheel 3 that may induce the specific behavior. If a specific object is present, the system outputs driving assistance information to notify the driver that the specific object is dangerous to driving. In this way, by outputting appropriate notifications regarding the specific behavior of the driver D1 and specific objects present around the driver D1, the specific behavior can be suppressed and the vehicle can start driving more safely. This makes it possible to reduce the risk of an accident when starting the vehicle C1 after waiting at a traffic light. In other words, it is possible to appropriately prevent the specific behavior after starting driving and reduce the risk of an accident caused by the specific behavior, allowing the vehicle C1 to start driving more safely.

[0190] Furthermore, when a traffic light ahead of a traveling vehicle C1 changes and the vehicle C1 is stopped, if there is a possibility that the driver D1 will begin a specific behavior before the vehicle C1 comes to a complete stop, driving assistance information is output to stop the specific behavior. For example, if the driver D1 takes his / her hands off the steering wheel 3 and looks at the electronic device 30 before the vehicle C1 comes to a complete stop, driving assistance information is output to warn the driver to immediately stop the behavior and grip the steering wheel 3. For example, if the driver D1 is gripping the steering wheel 3 but starts looking at the electronic device 30 before the vehicle C1 comes to a complete stop, driving assistance information is output to warn the driver to refrain from the behavior. For example, if the driver D1 is gripping the steering wheel 3 before the vehicle C1 comes to a complete stop, the system detects whether a specific object that induces the specific behavior is present on (or around) the driver D1's hands or on (or around) the steering wheel 3, and if a specific object is present, driving assistance information is output to notify the driver that the specific object is a driving hazard. This makes it possible to reduce the risk of an accident when stopping the vehicle C1 in front of a traffic light. In other words, it is possible to appropriately prevent specific actions while driving just before stopping, reduce the risk of an accident caused by the specific actions, and allow the vehicle C1 to stop more safely.

[0191] In addition, it is possible to change the output content of the driving assistance information based on the state of the driver D1, the reaction of the driver D1 to past outputs of driving assistance information, the state of other occupants, etc. This makes it possible to realize driving assistance that is less likely to cause discomfort to the driver D1.

[0192] [Example of Executing Processing in Other Devices or Systems] Note that, although the above describes an example in which the detection processing, estimation processing, determination processing, decision processing, output control processing, setting processing, etc. are executed in the information processing system 100, all or part of each of these processes may be executed in other devices. In this case, the information processing system is configured by the devices that execute part of each of these processes. For example, at least part of each process may be executed using in-vehicle devices, devices available to the user (e.g., smartphones, tablet terminals, personal computers, car navigation devices, IVIs), various driving assistance devices such as servers connectable via a predetermined network such as the Internet, and various electronic devices.

[0193] Furthermore, a part (or all) of the information processing system capable of executing the functions of the information processing system 100 may be provided by an application that can be provided via a predetermined network such as the Internet. This application is, for example, SaaS (Software as a Service).

[0194] [Configuration Example and Effects of the Present Embodiment] The driving assistance method according to the present embodiment is a driving assistance method for assisting a driver D1 in driving a vehicle C1. This driving assistance method includes a detection process (steps S400 and S410) for detecting at least one of the driver D1's behavior and an object in the vehicle C1, an estimation process (steps S505, S506, S519, S601, S602, and S603) for estimating that the stopped vehicle C1 will start moving or that the moving vehicle C1 will stop, and a specific driving process (steps S506, S519, S601, S602, and S603) for estimating that the stopped vehicle C1 will start moving, based on the detection result of the detection process, in which the driver D1 performs a specific driving operation while driving the vehicle C1 while performing a specific act other than an act related to driving, after the vehicle C1 starts moving. The program according to the present embodiment includes a determination process (steps S507, S508, S511, S512, S520 to S522, S525, S526) for determining whether the driver D1 is likely to perform specific driving before the vehicle C1 stops, or, if it is estimated that the traveling vehicle C1 will stop, determining whether the driver D1 is likely to perform specific driving before the vehicle C1 stops based on the detection results of the detection process, and an output process (steps S515, S529) for causing an output unit to output driving assistance information for causing the driver D1 to stop the specific driving if it is determined in this determination process that the driver D1 is likely to perform the specific driving. The program according to the present embodiment is a program that causes a computer to execute each of these processes. In other words, the program according to the present embodiment is a program that causes a computer to realize each function that can be executed by the information processing system 100.

[0195] According to this configuration, if it is determined that the driver D1 is likely to perform a specific driving behavior after the vehicle starts traveling or while traveling just before stopping, it is possible to output driving assistance information to stop the specific driving behavior. This makes it possible to appropriately prevent specific behavior that may cause specific driving behavior after the vehicle starts traveling or while traveling just before stopping, thereby reducing the risk of an accident caused by the specific behavior.

[0196] In the driving assistance method according to this embodiment, the detection process (steps S400 and S410) detects, based on an image captured inside the vehicle C1, at least one of the following: the driver D1 is performing a specific behavior that may induce the driver D1 to look away; a specific object that may induce the driver D1 to look away is present around the hands of the driver D1; and the specific object is present around the steering wheel of the vehicle C1. The determination process (steps S507, S508, S511, S512, S520 to S522, S525, and S526) determines that there is a high possibility of the driver D1 performing a specific driving behavior in at least one of the following cases: the driver D1 is performing a specific behavior; the specific object is present around the hands of the driver D1; and the specific object is present around the steering wheel of the vehicle C1.

[0197] With this configuration, it is possible to appropriately determine the possibility of performing a specific driving operation based on specific actions that may induce driver D1 to look away, specific objects that may induce driver D1 to look away, etc.

[0198] In the driving assistance method according to this embodiment, the detection process (steps S400 and S410) detects the hands and line of sight of the driver D1 and the presence of a specific object in a specific location (e.g., a location within reach of the driver D1) in the vehicle C1 based on an image captured inside the vehicle C1. In addition, the determination process (steps S507, S508, S511, S512, S520 to S522, S525, and S526) determines, when a specific object is detected in a specific location, whether or not the driver D1 is likely to gaze at or operate the specific object based on the transition of the driver D1's hands and line of sight. If it is determined that the driver D1 is likely to gaze at or operate the specific object, it is determined that the specific driving is likely to be performed.

[0199] According to this configuration, when it is estimated that a stopped vehicle C1 will start moving, it is possible to appropriately determine the possibility of the driver D1 performing a specific driving operation based on the driver's hands and line of sight.

[0200] In the driving assistance method of this embodiment, if it is estimated that the vehicle C1 will stop while in motion, and if the judgment process (steps S507, S508) determines that there is a high possibility that the specific behavior will be continued and specific driving will be performed until the vehicle C1 stops, the method further includes a decision process (step S509) that determines driving assistance information (warning level WL4) including content to cause the driver D1 to immediately stop the specific behavior.

[0201] According to this configuration, if it is determined that there is a high possibility that the driver D1 will continue to perform the specific behavior and drive in a specific manner until the vehicle C1 stops, it is possible to determine the content of a strong warning to make the driver D1 immediately stop the specific behavior.

[0202] The driving assistance method according to this embodiment further includes a signal detection process (step S502) for detecting a traffic light and its indication ahead of the vehicle C1 based on an image captured of the area ahead of the vehicle C1. In the estimation process (steps S505, S506, and S519), when the signal indicating a change from "start prohibition" to "start permission" is detected while the vehicle C1 is stopped in front of the traffic light, the method estimates that the stopped vehicle C1 will start. On the other hand, when the signal indicating "start prohibition" is detected while the vehicle C1 is traveling in front of the traffic light, the method estimates that the traveling vehicle C1 will stop. The driving assistance information includes information for stopping the specific driving and information for notifying the driver D1 of the change in the signal indicating the traffic light.

[0203] This configuration makes it possible to appropriately estimate whether a stopped vehicle C1 will start moving or whether a moving vehicle C1 will stop based on the traffic light indication. Furthermore, when such an estimation is made, it is possible to notify the driver D1 that the traffic light indication has changed by including the change in the driving assistance information.

[0204] In the driving assistance method according to this embodiment, the detection process (step S400) detects the facial orientation and line of sight of occupants seated in seats other than the driver's seat of the vehicle C1 based on an image captured inside the vehicle C1. The method also includes a determination process (steps S509, S510, S513, S514, S523, S524, S527, S528, S604 to S607) of determining whether at least one of the occupants is looking toward a traffic light at a timing before the driving assistance information is output based on the detection result of the detection process (step S400) and the detection result of the signal detection process (step S502), and determining whether at least one of the occupants is looking toward the traffic light if it is determined that at least one occupant is looking toward the traffic light, to delay the timing of outputting the driving assistance information or to stop the output of the driving assistance information.

[0205] According to this configuration, when another occupant has certainly noticed the change in the traffic light, there is a high possibility that the occupant will provide driving assistance to the driver D1, so it is possible to delay the output timing of the driving assistance information or stop the output of the driving assistance information, thereby reducing the possibility that the output of the driving assistance information will cause the driver D1 to feel uncomfortable or uneasy.

[0206] In the driving assistance method according to this embodiment, the detection process (step S400) detects the behavior of occupants seated in seats other than the driver's seat of the vehicle C1 based on images captured inside the vehicle C1. Even if the determination process (steps S507, S508, S511, S512, S520 to S522, S525, and S526) determines that the driver D1 is likely to perform specific driving, if the specific behavior related to the determined specific driving is an action of the driver D1 and other occupants handing over a predetermined object (e.g., an ETC card, a bag, or a wallet) based on the detection result of the detection process (if the specific behavior is determined to be an action that is performed only temporarily and will not continue), the method further includes a determination process (steps S509, S510, S513, S514, S523, S524, S527, S528, and S604 to S607) for stopping the output of driving assistance information or reducing the output content of the driving assistance information.

[0207] According to this configuration, when an action that is only performed temporarily and does not continue between the driver D1 and another occupant (for example, an action in which another occupant hands over an ETC card, a bag, or the like to the driver D1) is performed, it is possible to stop the output of the driving assistance information or to moderate the content of the driving assistance information, thereby making it possible to reduce the possibility that the driver D1 will feel uncomfortable or uneasy after the action.

[0208] In the driving assistance method according to this embodiment, the detection process (step S516) detects the driver D1's line of sight and facial expression based on an image captured inside the vehicle C1. The method also includes a setting process (steps S517 and S518) for setting the output content of the driving assistance information to be less severe from the next time onward if the driver D1's line of sight is directed toward the audio output unit 171 or the display unit 172 for a certain period of time after the output of the driving assistance information and if the driver D1's facial expression is negative toward the driving assistance information for that certain period of time. For example, if the driver D1 has such a negative reaction (e.g., the driver D1's line of sight is directed toward the audio output unit 171 or the display unit 172 and has a negative facial expression), the setting unit 145 stores negative reaction information related to the negative reaction in the driver response history information DB 230 (see FIG. 7 ). The setting unit 145 also changes the content of the driving assistance setting information DB 220 (see FIG. 6 ) based on the negative reaction information stored in the driver response history information DB 230.

[0209] According to this configuration, the output content of the driving assistance information from the next time onwards is softened in response to the driver D1's negative reaction to the driving assistance information, thereby reducing the possibility of causing discomfort or annoyance to the driver D1 from the next time onwards.

[0210] The driving assistance method according to this embodiment further includes a determination process (steps S509, S510, S513, S514, S523, S524, S527, S528, S604 to S607) for determining output content of driving assistance information based on risk levels 1 to 4 indicating the likelihood of specific driving behavior determined in the determination process (steps S507, S508, S511, S512, S520 to S522, S525, and S526) when it is estimated that the stopped vehicle C1 will start moving or when it is estimated that the moving vehicle C1 will stop. For example, the output content of the driving assistance information can be determined based on the notification level list 200 (see FIG. 4), the warning level list 210 (see FIG. 5), and the notification level list 240 (see FIG. 13).

[0211] This configuration makes it possible to determine the output content of the driving support information appropriate to the behavior of the driver D1, thereby reducing the possibility of the driver D1 feeling uncomfortable or uneasy.

[0212] In the driving assistance method according to this embodiment, the output unit 170 is at least one of an audio output unit 171 that outputs the driving assistance information by voice and a display unit 172 that displays and outputs the driving assistance information. The driving assistance information may also be output using the display unit 31, an audio output unit, or an output unit of another device present in the vehicle C1 of the electronic device 30 carried by the driver D1.

[0213] According to this configuration, it is possible to output driving assistance information from the most suitable output device according to the behavioral state of the driver D1, etc.

[0214] The information processing system 100 is a driving assistance device that assists a driver D1 in driving a vehicle C1. The information processing system 100 includes an object detection unit 123 and a behavior detection unit 124 (an example of a detection unit) that detect at least one of the behavior of the driver D1 and an object in the vehicle C1, an estimation unit 141 that estimates that the stopped vehicle C1 will start moving or that the moving vehicle C1 will stop, and a determination unit 142 that, when it is estimated that the stopped vehicle C1 will start moving, determines whether or not there is a high possibility that the driver D1 will perform a specific driving operation, in which the driver D1 drives while performing a specific act other than an act related to driving, after the vehicle C1 starts moving, based on the detection result by the behavior detection unit 124, and when it is estimated that the moving vehicle C1 will stop, determines whether or not there is a high possibility that the driver D1 will perform the specific driving operation before the vehicle C1 stops, based on the detection result by the behavior detection unit 124, and an output control unit 144 that, when it is determined by the determination unit 142 that there is a high possibility that the driver D1 will perform the specific driving operation, outputs driving assistance information to cause the driver D1 to stop the specific driving from the output unit 170 and at least one of the electronic devices 30 (an example of an output unit). The information processing system 100 may be configured with one device or multiple devices. Alternatively, instead of the information processing system 100, an information processing system may be configured with multiple devices that can execute the processes realized by the information processing system 100.

[0215] According to this configuration, if it is determined that the driver D1 is likely to perform a specific driving behavior after the vehicle starts traveling or while traveling just before stopping, it is possible to output driving assistance information to stop the specific driving behavior. This makes it possible to appropriately prevent specific behavior that may cause specific driving behavior after the vehicle starts traveling or while traveling just before stopping, thereby reducing the risk of an accident caused by the specific behavior.

[0216] Note that each processing procedure shown in this embodiment is an example for realizing this embodiment, and the order of some of the processing procedures may be changed within the scope that makes it possible to realize this embodiment, and some of the processing procedures may be omitted or other processing procedures may be added.

[0217] Each process in this embodiment is executed based on a program that causes a computer to execute various processing procedures. This embodiment can also be understood as an embodiment of a program that realizes the function of executing each process and a recording medium that stores the program. For example, an update process for adding a new function to the driving assistance device can store the program in the storage device of the driving assistance device. This makes it possible to cause the updated driving assistance device to perform each process described in this embodiment.

[0218] Although the embodiments of the present invention have been described above, the above embodiments merely show application examples of the present invention, and are not intended to limit the technical scope of the present invention to the specific configurations of the above embodiments.

[0219] This application claims priority based on Japanese Patent Application No. 2023-155677 filed with the Japan Patent Office on September 21, 2023, the entire contents of which are incorporated herein by reference.

Claims

1. A driver assistance method that assists the driver of a vehicle, A detection process that detects at least one of the driver's actions and an object inside the vehicle, An estimation process that estimates whether the stationary vehicle will start moving, or whether the moving vehicle will stop, When it is estimated that the stationary vehicle will start moving, a determination process is performed to determine, based on the detection result of the detection process, whether there is a high probability that the driver will perform a specific driving action, which involves driving while performing a specific action other than driving, after the vehicle starts moving, or when it is estimated that the moving vehicle will stop, a determination process is performed to determine, based on the detection result of the detection process, whether there is a high probability that the driver will perform the specific driving action before the vehicle stops. If the determination process determines that there is a high probability of performing the specified operation, the output process includes causing the driver to output driver support information from the output unit to stop the specified operation. Driving assistance methods.

2. A driving assistance method according to claim 1, In the detection process, based on images captured inside the vehicle, at least one of the following is detected: the driver is performing a specific action that may induce the driver to look away; a specific object that may induce the driver to look away is present around the driver's hand; and the specific object is present around the vehicle's steering wheel. In the determination process, it is determined that there is a high probability of performing the specific driving action if at least one of the following is detected: the driver is performing the specific action; the specific object is detected to be present around the driver's hand; or the specific object is detected to be present around the vehicle's steering wheel. Driving assistance methods.

3. A driving assistance method according to claim 2, In the detection process, based on images captured of the interior of the vehicle, the driver's hands and line of sight, and the presence of the specific object at a specific location inside the vehicle are detected. In the aforementioned determination process, When the specified object is detected at the specified location, it is determined, based on the driver's hand and gaze transitions, whether or not there is a high probability that the driver will gaze at or operate the specified object. If it is determined that there is a high probability that the driver will gaze upon or operate the specified object, then it is determined that there is a high probability that the specified operation will be performed. Driving assistance methods.

4. A driving assistance method according to any one of claims 1 to 3, The process further includes a signal detection process that detects a signal and its indication located in front of the vehicle based on an image captured in front of the vehicle, In the estimation process described above, when it is detected that the signal's instruction has changed from "no departure" to "departure permitted" while the vehicle is stopped before the signal, it is estimated that the stopped vehicle will depart, or when it is detected that the signal's instruction has changed from "no departure" while the vehicle is traveling before the signal, it is estimated that the traveling vehicle will stop. The aforementioned driving assistance information includes information for stopping the specific driving and information for informing the driver of changes in the signal indications. Driving assistance methods.

5. A driving assistance method according to claim 4, In the detection process, based on images taken inside the vehicle, the direction of the faces and gaze of occupants seated in seats other than the driver's seat of the vehicle are detected. The process further includes, at a timing prior to outputting the aforementioned driving assistance information, a decision process that determines whether at least one of the occupants was looking in the direction of the traffic signal based on the detection result of the detection process and the detection result of the traffic signal detection process, and if it is determined that at least one was looking in the direction of the traffic signal, a decision process that determines whether to delay the timing of outputting the aforementioned driving assistance information or to stop outputting the aforementioned driving assistance information. Driving assistance methods.

6. A driving assistance method according to any one of claims 1 to 3, In the detection process, based on images taken inside the vehicle, the actions of occupants seated in seats other than the driver's seat of the vehicle are detected. Even if the determination process determines that there is a high probability of performing the specified driving, if, based on the detection results of the detection process, the specified act related to the determined specified driving is determined to be an act in which the driver and the passengers hand over a predetermined object, the determination process further includes a decision process to stop outputting the driving support information or to mitigate the content of the output of the driving support information. Driving assistance methods.

7. A driving assistance method according to any one of claims 1 to 3, In the detection process, the driver's gaze and facial expression are detected based on the images taken inside the vehicle. The system further includes a setting process that, if the driver's gaze is directed towards the output unit for a certain period of time after the output of the aforementioned driving assistance information, and the driver's facial expression during that period is negative towards the driving assistance information, then the system will be configured to mitigate the content of the driving assistance information output in subsequent instances. Driving assistance methods.

8. A driving assistance method according to any one of claims 1 to 3, The process further includes a decision process that determines the content of the output of the driving support information based on the level indicating the likelihood of performing the specific driving determined in the determination process, when it is estimated that the stationary vehicle will start moving, or when it is estimated that the moving vehicle will stop. Driving assistance methods.

9. A driving assistance method according to any one of claims 1 to 3, The output unit is at least one of the following: an audio output unit that outputs the driving assistance information as sound; a display unit that displays and outputs the driving assistance information; and an electronic device held by the driver. Driving assistance methods.

10. A driver assistance device that assists the driver of a vehicle, A detection unit that detects at least one of the driver's actions and an object inside the vehicle, An estimation unit that estimates whether the stationary vehicle will start moving or whether the moving vehicle will stop, A determination unit that, when it is estimated that the stationary vehicle will start moving, determines, based on the detection result of the detection unit, whether there is a high probability that the driver will perform a specific driving action, which involves driving while performing a specific action other than driving, after the vehicle starts moving, or when it is estimated that the moving vehicle will stop, determines, based on the detection result of the detection unit, whether there is a high probability that the driver will perform the specific driving action before the vehicle stops, The system includes an output control unit that, when the determination unit determines that there is a high probability of performing the specified operation, causes the driver to output driving support information from the output unit to stop the specified operation. Driving assistance system.

Images