Notification device, information processing device and program

Abstract

To provide a notification device capable of accurately detecting reduction in concentration of a driver to driving and notifying the driver of an attention evocation, an information processing device and a program.SOLUTION: A notification device comprises a controller. The controller acquires sounds in a vehicle cabin which are collected by a microphone which is loaded in a vehicle. In a case where inquiry information indicating an inquiry is included in the acquired sounds, the controller notifies a driver of the vehicle of an attention evocation.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The disclosed embodiments relate to a notification device, an information processing device, and a program. [Background technology]

[0002] Various technologies have been proposed to detect the driving state of a vehicle driver and issue a warning based on the driving state. Specifically, a technology is known that compares the driving behavior pattern when the vehicle passes through an intersection with a reference driving behavior pattern set for each intersection, and detects and notifies the driver that the driver is in an unsafe driving state based on the comparison result (see, for example, Patent Document 1). Another known technology is one that calculates the degree of interference with the driver's safe driving based on the voice in the vehicle and the situation around the vehicle and notifies the driver (see, for example, Patent Document 2). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-061084 [Patent Document 2] Japanese Patent Publication No. 2020-064553 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the conventional technology does not accurately detect a decline in the driver's concentration on driving and issue a warning. The conventional technology described in Patent Document 1 detects the driver's driving state using the vehicle's behavior, but does not take into consideration that the driver's concentration on driving may be reduced due to factors other than the vehicle's behavior. Furthermore, the conventional technology described in Patent Document 2 calculates the degree of interference with safe driving based on the conversation contained in the audio within the vehicle and detects the driver's driving state, but does not consider that the content of the conversation may significantly reduce the driver's concentration on driving.

[0005] One aspect of the embodiment has been made in consideration of the above, and aims to provide a notification device, an information processing device, and a program that can accurately detect a decline in a driver's concentration on driving and notify them of a warning. [Means for solving the problem]

[0006] In order to solve the above problems and achieve the object, a notification device according to one aspect of the embodiment includes a controller. The controller acquires audio from within a vehicle cabin collected by a microphone mounted on the vehicle. When the acquired audio includes question information indicating a question, the controller notifies a driver of the vehicle to pay attention. [Effects of the Invention]

[0007] In one aspect of the embodiment, a voice in the vehicle cabin is acquired, and if the acquired voice includes question information indicating a question, a warning is issued to the driver of the vehicle. In this way, by detecting questions that may be a factor in reducing the driver's concentration on driving, it is possible to accurately detect a reduction in the driver's concentration on driving and issue a warning. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram for explaining an outline of a notification method performed by a notification system including a notification device according to the first embodiment. [Figure 2] FIG. 2 is a block diagram showing an example of the configuration of the notification system according to the first embodiment. [Figure 3] FIG. 3 is a block diagram showing an example of the configuration of the in-vehicle device according to the first embodiment. [Figure 4] FIG. 4 is a block diagram showing an example of the configuration of the information processing device according to the first embodiment. [Figure 5] FIG. 5 is a flowchart showing an example of processing executed by the controller of the in-vehicle device according to the first embodiment. [Figure 6]FIG. 6 is a diagram illustrating an example of an output unit that issues a warning notice. [Figure 7] FIG. 7 is a flowchart showing an example of processing executed by the processing device controller of the information processing device according to the first embodiment. [Figure 8] FIG. 8 is a flowchart showing an example of processing executed by the controller of the in-vehicle device according to the first embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of an output unit that issues a notice calling attention according to the driving risk level. [Figure 10] FIG. 10 is a block diagram showing an example of the configuration of an in-vehicle device according to the second embodiment. [Figure 11] FIG. 11 is a flowchart showing an example of processing executed by the controller of the in-vehicle device according to the second embodiment. [Figure 12] FIG. 12 is a diagram showing an example of an output unit that issues a notice calling attention according to the driving risk level in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of a notification device, an information processing device, and a program disclosed in the present application will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to the following embodiments.

[0010] (First embodiment) First, an overview of a notification method performed by a notification system including a notification device according to the first embodiment will be described below with reference to Fig. 1. Fig. 1 is a diagram for explaining an overview of a notification method performed by a notification system including a notification device according to the first embodiment. In the example of Fig. 1, a vehicle A is provided with a front seat C1 and a rear seat C2. A driver D sits in the front seat C1 of the vehicle A, and a passenger E sits in the rear seat C2. The passenger E is not limited to sitting in the rear seat C2, and may be sitting in another seat of the vehicle A, such as a passenger seat.

[0011] As shown in Fig. 1, the notification system 1 according to the first embodiment is a system that detects the driving state of a driver D of a vehicle A and issues a warning based on the driving state. The notification system 1 according to this embodiment is able to accurately detect a decrease in the driver D's concentration on driving and issue a warning.

[0012] The notification system 1 includes an in-vehicle device 10 and an information processing device 50. The in-vehicle device 10 is an example of a notification device.

[0013] The in-vehicle device 10 is mounted on a vehicle A. The in-vehicle device 10 is a device having, for example, a camera, a microphone, a GPS (Global Positioning System) sensor, a storage device, a microcomputer, etc. The in-vehicle device 10 also has a communication function for communicating with an information processing device 50 via a communication network. Note that, for example, a drive recorder can be used as the in-vehicle device 10.

[0014] The camera of the in-vehicle device 10 captures images of the outside of the vehicle A and images (in-vehicle images) of the interior B of the vehicle A. Specifically, the camera captures images of the periphery of the vehicle A and images of the driver D, passenger E, etc. in the interior B of the vehicle.

[0015] The microphone of the in-vehicle device 10 collects and acquires voices from inside the vehicle compartment B (step S1). The voices collected by the microphone include at least one of the voices of the driver D and the passenger E, but are not limited to these.

[0016] The in-vehicle device 10 determines whether the acquired voice from inside the vehicle cabin B includes inquiry information indicating a question (step S2). The inquiry information may be information indicating a question from the driver D to the passenger E, or information indicating a question from the passenger E to the driver D. An inquiry is, for example, a conversation that seeks an answer or an opinion from the person asking the question. Therefore, an inquiry can also be said to be a question. Examples of questions are "Where are you going?", "What are you eating?", "The scenery is beautiful, isn't it?", etc.

[0017] If the voice from inside the vehicle cabin B contains question information, the in-vehicle device 10 calculates a decline level indicating the degree to which the driver D's concentration on driving has declined (step S3). The decline level is expressed, for example, as a numerical value, and an increase in the numerical value indicates an increase in the degree of decline in concentration (in other words, a further decline in concentration). The decline level increases as the number of times question information is detected in the voice from inside the vehicle cabin B increases, which will be described later. Note that if the voice from inside the vehicle cabin B does not contain question information, the in-vehicle device 10 does not calculate the decline level, and leaves it at an initial value (for example, 0).

[0018] Next, the in-vehicle device 10 transmits driving information including the calculated degradation level to the information processing device 50 (step S4). The driving information includes, in addition to the degradation level, for example, an outside image of vehicle A, an inside image of vehicle A, position information of vehicle A, vehicle speed, etc., but these are merely examples and are not limited thereto.

[0019] The information processing device 50 may be configured as a cloud server that provides cloud services via a communication network such as the Internet or a mobile phone network, or may be configured to perform distributed processing using multiple servers. The information processing device 50 also has a communication function for communicating with the in-vehicle device 10 and the like via the communication network.

[0020] The information processing device 50 calculates the driving risk level of the driver D according to the driving information including the deterioration level received from the in-vehicle device 10 (step S5). The driving risk level is information indicating the degree to which the driving state of the driver D is in a dangerous state. The driving risk level is expressed, for example, as a numerical value, and an increase in the numerical value indicates that the driving state of the driver D is in a more dangerous state.

[0021] The driving risk level is calculated using the reduction level calculated when the voice in the vehicle cabin B contains question information. More specifically, the driving risk level is calculated to be proportional to the reduction level. In other words, the driving risk level is calculated to increase as the reduction level increases. In this way, the driving risk level corresponds to the reduction level. Note that the driving risk level is calculated based on, in addition to the reduction level, for example, the location where vehicle A is traveling, the surrounding conditions of vehicle A, the traveling state of vehicle A, etc., which will be described later.

[0022] The information processing device 50 transmits information indicating the calculated driving risk level to the in-vehicle device 10 (step S6). The in-vehicle device 10 issues a warning based on the driving risk level received from the information processing device 50 (step S7). Specifically, when the driving risk level is relatively high, the in-vehicle device 10 estimates that the decrease level has increased due to a question from the driver D to the passenger E or a question from the passenger E to the driver D, and that this increase has led to an increase in the driving risk level, and issues a warning. Examples of such a warning notification include, but are not limited to, a display notification that a decrease in concentration on driving has been detected, a message urging the driver to concentrate on driving, or a voice notification. Note that the driving risk level may be transmitted to a terminal device 100 (see FIG. 2, which will be described later) used by the driver, an administrator terminal device 200 (see FIG. 2) used by an administrator of vehicle A, or the like.

[0023] The in-vehicle device 10 according to this embodiment acquires voice from inside the vehicle compartment B, and if the acquired voice includes question information, issues a warning to the driver D of the vehicle A. This allows the in-vehicle device 10 to accurately detect a decrease in the driver D's concentration on driving and issue a warning. Specifically, when the driver D asks a passenger E a question, the driver D is likely to lose concentration on driving, for example, by concentrating on listening to the passenger E's answer. Furthermore, when the passenger E asks the driver D a question, the driver D is likely to lose concentration on driving, for example, by concentrating on thinking of an answer to the passenger E. In this way, by detecting a question that could be a factor in decreasing the driver D's concentration on driving, it is possible to accurately detect a decrease in the driver D's concentration on driving and issue a warning.

[0024] The in-vehicle device 10 is configured to issue a warning based on the driving risk level received from the information processing device 50, but this is not limited to this, and the warning may be issued, for example, when the calculated deterioration level becomes relatively high.

[0025] Fig. 2 is a block diagram showing an example of the configuration of a notification system 1 according to the first embodiment. Note that the block diagrams such as Fig. 2 show only the components necessary to explain the features of the embodiment, and general components are omitted.

[0026] The notification system 1 includes the above-described in-vehicle device 10, an information processing device 50, a terminal device 100, and an administrator terminal device 200. The in-vehicle device 10, the information processing device 50, the terminal device 100, and the administrator terminal device 200 are communicably connected via a communication network N.

[0027] The terminal device 100 is a terminal owned by the driver, such as a smartphone, a notebook PC (Personal Computer), a desktop PC, or a tablet terminal. The terminal device 100 is notified of the driving risk level as described above. This allows the driver to check the driving risk level of his or her own driving, and also makes it possible to encourage the driver to further improve his or her awareness of safe driving, for example.

[0028] The administrator terminal device 200 is a terminal owned by the administrator of vehicle A, and may be, for example, a smartphone, a notebook PC, a desktop PC, or a tablet terminal. The administrator terminal device 200 is notified of the driving risk level as described above. This allows the administrator to check the driving risk level of the driver. Furthermore, based on the driving risk level, the administrator can encourage the driver to, for example, further improve their awareness of safe driving.

[0029] Next, a configuration example of the in-vehicle device 10 according to the first embodiment will be described with reference to Fig. 3. Fig. 3 is a block diagram showing a configuration example of the in-vehicle device 10 according to the first embodiment.

[0030] As shown in FIG. 3, the in-vehicle device 10 includes an exterior camera 11, an interior camera 12, a microphone 13, a GPS sensor 14, an output unit 15, a controller (control unit) 20, a memory unit 30, and a communication unit 40.

[0031] The exterior camera 11 is installed at an appropriate position on the vehicle and captures images of the exterior of the vehicle. The exterior camera 11 captures images of the area around the vehicle, including the area ahead of the vehicle, as exterior images. The exterior camera 11 outputs the captured exterior images to the controller 20.

[0032] The interior camera 12 is installed at an appropriate position in the vehicle and captures video of the interior of the vehicle (interior video). The interior video includes, for example, video of the faces of the driver and passengers. The interior camera 12 outputs the captured interior video to the controller 20. The exterior camera 11 and the interior camera 12 are cameras equipped with, for example, a lens and an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), but are not limited to this.

[0033] The microphone 13 is installed in the vehicle cabin and collects (collects) voices within the cabin. The voices within the cabin include, for example, voices spoken by the driver and passengers. If a hands-free call function is available in the vehicle, the voices within the cabin may also include voices during the hands-free call. The microphone 13 outputs the collected voices to the controller 20.

[0034] The GPS sensor 14 receives radio waves transmitted from GPS satellites, acquires vehicle position information based on the received radio waves, and outputs the acquired position information to the controller 20.

[0035] The output unit 15 outputs various information such as driving risk level, attention-calling notifications, etc. For example, the output unit 15 includes at least one of a display unit such as a display and an audio output unit such as a speaker, and outputs various information to the driver of the vehicle.

[0036] The storage unit 30 is realized by a storage device such as a ROM (Read Only Memory), a RAM (Random Access Memory), a flash memory, etc. The storage unit 30 stores map information 31, degradation level information 32, various data, various programs, etc.

[0037] The map information 31 is information about a map including the route (road) on which the vehicle travels. The map information 31 includes various information about the route, such as information indicating the locations of crosswalks, intersections, stop signs, curves, and steep slopes, and legal speed limits. The degradation level information 32 is information about the calculated degradation level.

[0038] The communication unit 40 is a communication interface that transmits and receives information to and from the information processing device 50 and the like.

[0039] The controller 20 corresponds to a so-called processor. The controller 20 is realized by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphical Processing Unit), or the like. The controller 20 executes a program according to an embodiment (not shown) stored in the storage unit 30, using RAM as a work area. Note that the controller 20 may be partially or entirely configured by hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0040] The controller 20 executes various processes such as a process of acquiring various data such as voice in the vehicle cabin, a process of determining whether the voice includes question information, a process of calculating the degradation level, and a process of transmitting driving information including the degradation level to the information processing device 50. These various processes will be described later with reference to Fig. 5. The controller 20 also executes a process of issuing a warning notification in accordance with the driving risk level received from the information processing device 50, which will be described later with reference to Fig. 8.

[0041] Next, an example of the configuration of the information processing device 50 according to the first embodiment will be described with reference to Fig. 4. Fig. 4 is a block diagram showing an example of the configuration of the information processing device 50 according to the first embodiment.

[0042] As shown in FIG. 4, the information processing device 50 includes a communication unit 60, a processing device controller (control unit) 70, and a storage unit 80.

[0043] The communication unit 60 is a communication interface that transmits and receives information to and from the in-vehicle device 10 and the like.

[0044] The storage unit 80 is realized by a storage device such as a ROM, a RAM, or a flash memory. The storage unit 80 stores map information 81, driving risk level information 82, various data, various programs, and the like. The map information 81 is map information including the driving route (road) on which the vehicle travels, similar to the map information 31 of the in-vehicle device 10. The driving risk level information 82 is information on the calculated driving risk level.

[0045] The processing device controller 70 corresponds to a so-called processor. The processing device controller 70 is realized by a CPU, MPU, GPU, etc. The processing device controller 70 executes a program according to an embodiment (not shown) stored in the storage unit 80, using RAM as a work area. Note that the processing device controller 70 may be partially or entirely configured by hardware such as an ASIC or FPGA.

[0046] The processing device controller 70 executes various processes such as a process of calculating the driving risk level in accordance with the driving information including the deterioration level received from the in-vehicle device 10. These various processes will be described later with reference to FIG.

[0047] Next, the processing executed by the in-vehicle device 10 and the information processing device 50 will be described in detail with reference to Fig. 5 and subsequent figures. Fig. 5 is a flowchart showing an example of processing executed by the controller 20 of the in-vehicle device 10 according to the first embodiment. Note that the processing shown in Fig. 5 is repeatedly executed every time a predetermined time elapses, but is not limited to this.

[0048] 5, the controller 20 of the in-vehicle device 10 acquires various data (step S100). For example, the controller 20 acquires an outside image captured by the exterior camera 11. The controller 20 acquires an inside image captured by the interior camera 12. The controller 20 acquires sound (sound data) from within the vehicle cabin collected by the microphone 13. The controller 20 acquires vehicle position information from the GPS sensor 14.

[0049] Next, the controller 20 determines whether or not there is voice (specifically, conversation) in the vehicle cabin based on the voice data acquired from the microphone 13 (step S101). If it is determined that there is voice in the vehicle cabin (step S101, Yes), the controller 20 sets the conversation flag to "1" and sets the conversation timer to "N seconds" (step S102). The conversation flag is a flag indicating that a conversation is occurring in the vehicle cabin. When the conversation flag is "1," it indicates that a conversation is occurring, and when the conversation flag is "0," it indicates that a conversation is not occurring. The conversation timer is a timer used in a process of determining whether or not the conversation has ended. The conversation timer is set to "N seconds" and counts down each time voice is detected in the vehicle cabin. Then, the controller 20 determines that the conversation has ended when the conversation timer reaches "0," in other words, when "N seconds" have elapsed since voice (conversation) was no longer detected in the vehicle cabin. Note that N seconds is set to a time period at which it can be determined that the conversation has ended, but is not limited to this and can be set to any value.

[0050] Next, the controller 20 performs a voice analysis process on the voice in the vehicle cabin (step S103). For example, the controller 20 converts the voice in the vehicle cabin into text using an arbitrary analysis method. The controller 20 performs analysis on the converted text (sentence), such as word analysis, phrasing analysis, expression analysis, and emotion analysis. For example, emotion recognition AI (Artificial Intelligence) for the sentence is stored in the memory unit 30, and the controller 20 performs analysis, such as word analysis, phrasing analysis, expression analysis, and emotion analysis, using the emotion recognition AI. The emotion recognition AI is a model trained to detect various words, phrases, expressions, emotions, etc. from the voice or text in the vehicle cabin. If the detected words, phrases, expressions, etc. contain question information, in other words, if the voice in the vehicle cabin contains question information, the controller 20 extracts the question information.

[0051] Next, the controller 20 determines whether or not the voice in the vehicle cabin includes a question from a passenger to the driver, based on the analysis result of the voice analysis process (step S104).

[0052] It should be noted that any identification method such as voice recognition or face recognition can be used to identify whether the voice in the vehicle cabin belongs to the driver or a passenger. In the case of voice recognition, for example, the driver's voice data is stored in advance in the storage unit 30, and the controller 20 identifies the voice in the vehicle cabin as belonging to the driver if the voice in the vehicle cabin matches the voice data, and identifies the voice in the vehicle cabin as belonging to a passenger if they do not match. It should be noted that it is also possible to store the passenger's voice data in advance in the storage unit 30, and the controller 20 identifies the voice in the vehicle cabin as belonging to a passenger if the voice in the vehicle cabin matches the voice data.

[0053] In the case of face authentication, for example, the memory unit 30 stores face data of the driver in advance, and the controller 20 determines that the voice in the vehicle cabin is that of the driver if the face data matches the face with the mouth moving in the vehicle interior video. The controller 20 determines that the voice in the vehicle cabin is that of a passenger if the face data does not match the face with the mouth moving in the vehicle interior video. Note that the memory unit 30 may store face data of the passenger in advance, and the controller 20 may determine that the voice in the vehicle cabin is that of a passenger if the face data matches the face with the mouth moving in the vehicle interior video.

[0054] When it is determined that the voice in the vehicle cabin includes a question from a passenger to the driver (step S104, Yes), the controller 20 calculates a reduction level (step S105). Specifically, the controller 20 calculates a new reduction level by adding a preset value X1 to the current reduction level. More specifically, the controller 20 reads the current reduction level from the reduction level information 32 stored in the storage unit 30, and stores the new reduction level obtained by adding the value X1 to the current reduction level as the reduction level information 32 in the storage unit 30. In other words, the controller 20 updates the reduction level information 32 in the storage unit 30 to the reduction level calculated in step S105. The initial value of the reduction level is 0 (zero).

[0055] If it is determined that the voice in the vehicle cabin does not include a question from a passenger to the driver (No at step S104), the controller 20 skips the process at step S105.

[0056] Next, the controller 20 determines whether the voice in the vehicle cabin includes a question from the driver to the passenger (step S106). If the controller 20 determines that the voice in the vehicle cabin includes a question from the driver to the passenger (step S106, Yes), the controller 20 calculates a reduction level (step S107). Specifically, the controller 20 adds a preset value X2 to the current reduction level to calculate a new reduction level. More specifically, the controller 20 reads the current reduction level from the reduction level information 32 stored in the storage unit 30, and stores the new reduction level obtained by adding the value X2 to the current reduction level as the reduction level information 32 in the storage unit 30. In other words, the controller 20 updates the reduction level information 32 in the storage unit 30 to the reduction level calculated in step S107.

[0057] The value X1 added in step S105 and the value X2 added in step S107 are set to arbitrary values. Furthermore, the value X1 and the value X2 may be the same value or different values.

[0058] If it is determined that the voice in the vehicle cabin does not include a question from the driver to the passenger (No in step S106), the controller 20 skips the process of step S107.

[0059] Next, the controller 20 reads the degradation level from the degradation level information 32 stored in the storage unit 30, and determines whether the read degradation level is equal to or greater than a threshold level (step S108). The threshold level is set to a value that indicates, for example, an increase in the number of questions from passengers to the driver, or from the driver to passengers, which can be used to estimate a relatively large decline in the driver's concentration on driving. If the controller 20 determines that the degradation level is equal to or greater than the threshold level (step S108, Yes), it issues a warning to the driver (step S109).

[0060] The notification of the alert in step S109 will now be described with reference to Fig. 6. Fig. 6 is a diagram showing an example of the output unit (display unit) 15 where the alert notification is made.

[0061] 6, the controller 20 displays a warning notice in the display field 15a of the output unit (display unit) 15 of the in-vehicle device 10. In the display field 15a, a message encouraging the driver to concentrate on driving, such as "Are you losing concentration on driving? Please concentrate on driving," is displayed. This enables the controller 20 to encourage the driver to concentrate on driving and drive safely.

[0062] Returning to the description of FIG. 5, if it is determined that the decrease level is not equal to or greater than the threshold level (No at step S108), the controller 20 skips the process at step S109.

[0063] When it is determined that there is no voice (specifically, conversation) in the vehicle cabin (step S101, No), the controller 20 determines whether the conversation has ended (step S110). Specifically, the controller 20 determines whether the conversation timer has reached 0.

[0064] If it is determined that the conversation has not ended (step S110, No), in other words, if it is determined that the conversation timer is not 0, the controller 20 returns to the processing of step S100.

[0065] On the other hand, when it is determined that the conversation has ended (step S110, Yes), in other words, when it is determined that the conversation timer has reached 0, the controller 20 transmits driving information including the calculated degradation level to the information processing device 50 (step S111). The driving information includes, in addition to the degradation level, an outside image of the vehicle, an inside image of the vehicle, position information of the vehicle, vehicle speed, etc. at the time when the degradation level was calculated.

[0066] The vehicle speed is calculated based on the vehicle's position information acquired from the GPS sensor 14. For example, the controller 20 calculates the distance traveled by the vehicle based on the vehicle's position information, and detects the vehicle speed by dividing the calculated distance traveled by the time required for the movement. Although the above describes an example in which the controller 20 calculates the vehicle speed based on the vehicle's position information, the present invention is not limited to this, and the controller 20 may be configured to acquire vehicle speed information from a vehicle speed sensor (not shown) mounted on the vehicle, for example.

[0067] Next, since the current conversation has ended, the controller 20 executes a process of resetting various data for the current conversation (step S112). Specifically, the controller 20 resets the conversation flag to "0" and also resets the degradation level to "0 (initial value)". Furthermore, the controller 20 transmits to the information processing device 50 a reset request for the driving risk level calculated based on the driving information including the degradation level calculated for the current conversation. In response to the reset request, the information processing device 50 thereby executes a process of resetting the driving risk level calculated for the current conversation at an appropriate timing.

[0068] 5, the controller 20 transmits the driving information to the information processing device 50 at the timing when the conversation ends, but this is not limited to this. That is, the controller 20 may transmit the driving information at any timing, for example, by collectively transmitting a plurality of pieces of driving information including the calculated deterioration level to the information processing device 50 between the start and end of driving of the vehicle.

[0069] Next, a process performed by the processing device controller 70 of the information processing device 50 when the processing device controller 70 receives driving information including a degradation level from the in-vehicle device 10 will be described with reference to Fig. 7. Fig. 7 is a flowchart showing an example of a process performed by the processing device controller 70 of the information processing device 50 according to the first embodiment. Note that the process shown in Fig. 7 is performed at the timing when the driving information is received from the in-vehicle device 10, but is not limited to this.

[0070] 7, the processing device controller 70 determines whether the decline level included in the received driving information is greater than 0 (step S200). Specifically, the process of step S200 is a process of determining whether it is estimated that the driver's concentration on driving has declined due to a question from a passenger to the driver or a question from the driver to a passenger.

[0071] If the processing device controller 70 determines that the decline level is not greater than 0 (step S200, No), in other words, if it determines (estimates) that the decline level is 0 and that the driver's concentration on driving has not declined due to the question, it skips subsequent processing.

[0072] When it is determined that the decline level is greater than 0 (step S200, Yes), in other words, when it is determined (estimated) that the driving concentration has declined due to the question, the processing device controller 70 proceeds to step S201 and thereafter to calculate the driving risk level. Specifically, the processing device controller 70 calculates the driving risk level according to the decline level, the location where the vehicle is traveling, the conditions around the vehicle, the driving state of the vehicle (for example, vehicle speed), etc.

[0073] In detail, the processing device controller 70 sets the value of the decrease level to the driving risk level and calculates the driving risk level (step S201). Specifically, the controller 20 sets the value of the decrease level received from the in-vehicle device 10 to the driving risk level and stores the set driving risk level in the storage unit 30 as driving risk level information 82.

[0074] Next, the processing device controller 70 determines whether the vehicle was traveling near a pedestrian crossing at the time when the decline level was calculated (in other words, the time when a decline in concentration on driving was estimated) (step S202). Specifically, the processing device controller 70 determines whether the vehicle was traveling near a pedestrian crossing based on the vehicle's position information included in the driving information and the map information 81 in the storage unit 80. The area around a pedestrian crossing is an example of a driving location where the driver should concentrate more on driving and be more conscious of safe driving. Such a driving location is set in advance. Note that the area around a pedestrian crossing is an example of a predetermined location. The predetermined location is not limited to the area around a pedestrian crossing, but may be other locations such as an intersection, a stop sign, a curved road, a steep slope, etc.

[0075] When it is determined that the vehicle was traveling near a pedestrian crossing (step S202, Yes), the processing device controller 70 calculates a driving risk level (step S203). Specifically, the processing device controller 70 calculates a new driving risk level by adding a preset value Y1 to the current driving risk level. More specifically, the processing device controller 70 reads the current driving risk level from the driving risk level information 82 stored in the storage unit 80, and stores the new driving risk level obtained by adding the value Y1 to the current driving risk level as the driving risk level information 82 in the storage unit 80. In other words, the processing device controller 70 updates the driving risk level information 82 in the storage unit 80 to the driving risk level calculated in step S203.

[0076] If it is determined that the vehicle is not traveling near a pedestrian crossing (step S202, No), the processing device controller 70 skips the process of step S203.

[0077] Next, the processing device controller 70 determines whether or not there were pedestrians, bicycles, etc. near the vehicle at the time when the decline level was calculated (in other words, the time when a decline in concentration on driving was estimated) (step S204). Specifically, the processing device controller 70 analyzes the outside-of-vehicle video included in the driving information using any video analysis method, and detects the presence or absence of pedestrians, etc. from the analysis results. A situation in which pedestrians, etc. are present near the vehicle is an example of a situation around the vehicle in which the driver should concentrate more on driving and be more conscious of safe driving. Such a situation is set in advance.

[0078] When it is determined that a pedestrian or the like is present near the vehicle (step S204, Yes), the processing device controller 70 calculates a driving risk level (step S205). Specifically, the processing device controller 70 calculates a new driving risk level by adding a preset value Y2 to the current driving risk level. More specifically, the processing device controller 70 reads the current driving risk level from the driving risk level information 82 stored in the storage unit 80, and stores the new driving risk level obtained by adding the value Y2 to the current driving risk level as the driving risk level information 82 in the storage unit 80. In other words, the processing device controller 70 updates the driving risk level information 82 in the storage unit 80 to the driving risk level calculated in step S205.

[0079] If it is determined that there is no pedestrian or the like near the vehicle (step S204, No), the processing device controller 70 skips the process of step S205.

[0080] Next, the processing device controller 70 determines whether the vehicle speed was greater than a threshold vehicle speed at the time when the decline level was calculated (in other words, the time when a decline in concentration on driving was estimated) (step S206). The threshold vehicle speed is set to a value that makes it possible to estimate that the vehicle is in a dangerous driving state when the vehicle speed is greater than this value. The threshold vehicle speed is set to, for example, the legal speed of the road on which the vehicle is traveling, but is not limited to this and can be set to any value.

[0081] The processing device controller 70 compares the vehicle speed information included in the driving information with a preset threshold vehicle speed, and if it is determined that the vehicle speed is greater than the threshold vehicle speed (step S206, Yes), calculates a driving risk level (step S207). Specifically, the processing device controller 70 adds a preset value Y3 to the current driving risk level to calculate a new driving risk level. More specifically, the processing device controller 70 reads the current driving risk level from the driving risk level information 82 stored in the storage unit 80, and stores the new driving risk level obtained by adding the value Y3 to the current driving risk level as the driving risk level information 82 in the storage unit 80. In other words, the processing device controller 70 updates the driving risk level information 82 in the storage unit 80 to the driving risk level calculated in step S207.

[0082] If it is determined that the vehicle speed is not greater than the threshold vehicle speed (step S206, No), that is, if the vehicle speed is equal to or less than the threshold vehicle speed, the processing device controller 70 skips the process of step S207.

[0083] The value Y1 added in step S203, the value Y2 added in step S205, and the value Y3 added in step S207 are set to arbitrary values. The values ​​Y1, Y2, and Y3 may be the same value or different values.

[0084] Next, the processing device controller 70 transmits information on the calculated driving risk level to the in-vehicle device 10 (step S208). The timing at which the processing device controller 70 transmits information on the driving risk level to the in-vehicle device 10 can be set arbitrarily, for example, at the time when the driving risk level is calculated, or at the time after the vehicle driving is completed.

[0085] In the above example, the information processing device 50 identifies the location where the vehicle is traveling, the surrounding conditions of the vehicle, and the traveling state of the vehicle at the time the degradation level is calculated, but some or all of these identifications may be performed by the in-vehicle device 10.

[0086] Next, the process performed by the controller 20 of the in-vehicle device 10 when it receives information on the driving risk level calculated by the information processing device 50 will be described with reference to Fig. 8. Fig. 8 is a flowchart showing an example of the process performed by the controller 20 of the in-vehicle device 10 according to the first embodiment. Note that the process shown in Fig. 8 is repeatedly performed every time a predetermined time elapses, but is not limited to this.

[0087] 8, the controller 20 determines whether or not the driving risk level has been received from the information processing device 50 (step S300). If it is determined that the driving risk level has not been received (step S300, No), the controller 20 ends the process.

[0088] When it is determined that the driving risk level has been received (Yes at step S300), the controller 20 issues a warning in accordance with the received driving risk level (step S301).

[0089] The notification of the attention call in step S301 will now be described with reference to Fig. 9. Fig. 9 is a diagram showing an example of the output unit (display unit) 15 that has issued a notification of the attention call according to the driving risk level.

[0090] As shown in FIG. 9 , the controller 20 displays the driving risk level in the display field 15b of the output unit 15 of the in-vehicle device 10. The controller 20 also displays a warning notice corresponding to the driving risk level in the display field 15c of the output unit 15 of the in-vehicle device 10. For example, the display field 15c may display a message such as, "When you were driving near a pedestrian crossing, your concentration on driving may have decreased due to questions you asked your passengers and the passengers' questions. Please concentrate on driving and drive safely." That is, the display field 15c displays a message indicating that a decrease in concentration on driving has been detected and encouraging the driver to concentrate on driving. This allows the controller 20 to encourage the driver to concentrate on driving safely.

[0091] As described above, the in-vehicle device (an example of a notification device) 10 according to the first embodiment includes the controller 20. The controller 20 acquires voices within the vehicle cabin collected by the microphone 13 mounted on the vehicle. If the acquired voice contains question information indicating a question, the controller 20 notifies the driver of the vehicle to call attention.

[0092] In this way, the in-vehicle device 10 acquires voices within the vehicle cabin, and if the acquired voices contain question information indicating a question, it issues a warning to the driver of the vehicle. By detecting questions that may be a factor in reducing the driver's concentration on driving, the in-vehicle device 10 can accurately detect a reduction in the driver's concentration on driving and issue a warning.

[0093] The question information includes at least one of information indicating a question from the driver to the passenger and information indicating a question from the passenger to the driver. This allows the in-vehicle device 10 to detect questions from the driver to the passenger and questions from the passenger to the driver, which may be factors that decrease the driver's concentration on driving, and therefore allows for more accurate detection of a decrease in the driver's concentration on driving.

[0094] Furthermore, the controller 20 issues a warning when the vehicle travels through a preset location (for example, a crosswalk, etc.). Specifically, when the decline level is calculated (in other words, when a decline in concentration on driving is estimated), the controller 20 adds up the driving risk level when the vehicle travels through a preset location, and issues a warning according to the added driving risk level.

[0095] This allows the in-vehicle device 10 to issue a warning when, for example, the driver's concentration on driving is reduced due to the question when the vehicle is traveling across a pedestrian crossing.

[0096] (Second embodiment) Next, a notification system 1 including an in-vehicle device 10 according to a second embodiment will be described with reference to Fig. 10. Fig. 10 is a block diagram showing an example of the configuration of the in-vehicle device 10 according to the second embodiment. Note that, in the following, configurations common to the first embodiment will be assigned the same reference numerals and descriptions thereof may be omitted.

[0097] In the in-vehicle device 10 according to the second embodiment, if the sound in the vehicle cabin (for example, the sound of music, radio, television, etc.) contains a keyword that affects the driver's emotions, a warning is issued.

[0098] More specifically, in the in-vehicle device 10 according to the second embodiment, keyword information 33 is stored in the storage unit 30 as shown in FIG. 10. The keyword information 33 is information relating to keywords that affect the emotions of the driver, and is stored (registered) in advance in the storage unit 30. Specifically, when the driver hears a keyword, the keyword may affect the driver's emotions and reduce the driver's ability to concentrate on driving. Note that being affected by emotions includes, but is not limited to, being emotionally stirred, being interested, or being concerned, for example.

[0099] Examples of keywords are words that may interest or concern the driver, such as "work," "weather," "traffic jam," "music title," and "music lyrics." Other examples of keywords are words that may stir the driver's emotions or pique the driver's interest, such as "favorite team name," "goal," "home run," "comeback," and "tie" in sports broadcasts. Note that the above keywords are merely examples and are not limiting, and can be set arbitrarily.

[0100] The controller 20 executes various processes such as a process of determining whether or not a keyword is included in the voice in the vehicle cabin, and a process of calculating a reduction level when a keyword is included. Such processes will be described with reference to Fig. 11. Fig. 11 is a flowchart showing an example of a process executed by the controller 20 of the in-vehicle device 10 according to the second embodiment. Note that the process shown in Fig. 11 is repeatedly executed every time a predetermined time elapses, but is not limited to this.

[0101] 11, the controller 20 of the in-vehicle device 10 acquires various data (step S400). Step S400 is the same process as step S100. Specifically, the controller 20 acquires outside vehicle images from the outside vehicle camera 11, inside vehicle images from the inside vehicle camera 12, inside vehicle audio (audio data) from the microphone 13, and vehicle position information from the GPS sensor 14.

[0102] Next, the controller 20 executes a voice analysis process on the voice in the vehicle cabin (step S401). Step S401 is the same process as step S103. Specifically, the controller 20 converts the voice in the vehicle cabin into text using any analysis method. The controller 20 performs analysis of the converted text (sentence), such as word analysis, word usage analysis, expression analysis, and emotion analysis, using emotion recognition AI.

[0103] Next, the controller 20 determines whether or not a new keyword is included in the words, phrases, expressions, etc. detected in step S401, in other words, whether or not a new keyword is included in the voice in the vehicle cabin (step S402). Here, a new keyword means a keyword that is detected for the first time in this process.

[0104] If it is determined that the voice in the vehicle cabin contains a new keyword (step S402, Yes), the controller 20 sets the keyword timer to "Msec" (step S403). The keyword timer is a timer used in a process of determining whether a certain time has elapsed since the detection of a new keyword and the possibility of detecting a new keyword has decreased. The keyword timer is set to "Msec" and counts down each time a keyword is detected in the voice. Then, when the keyword timer reaches "0," in other words, when "Msec" has elapsed since a new keyword was no longer detected, the controller 20 determines that the possibility of detecting a new keyword is low. Note that Msec is set to a time period that allows it to be determined that the possibility of detecting a new keyword is low, but is not limited to this and can be set to any value.

[0105] Next, the controller 20 calculates a reduction level in response to the fact that the voice in the vehicle cabin contains a keyword (step S404). Specifically, the controller 20 calculates a new reduction level by adding a preset value Z1 to the current reduction level. More specifically, the controller 20 reads the current reduction level from the reduction level information 32 stored in the storage unit 30, and stores the new reduction level obtained by adding the value Z1 to the current reduction level as the reduction level information 32 in the storage unit 30. In other words, the controller 20 updates the reduction level information 32 in the storage unit 30 to the reduction level calculated in step S404.

[0106] Next, the controller 20 determines whether or not the movement of the driver's mouth includes a preset movement (step S405). Specifically, the controller 20 analyzes the in-vehicle video using any video analysis method and extracts the movement of the driver's mouth from the analysis result. The controller 20 determines whether or not the extracted movement of the driver's mouth includes a preset movement. More specifically, step S405 is a process of determining whether or not the driver's emotions (mood) are elevated, for example, by humming to music (audio) playing in the vehicle, and whether or not there is a possibility that the driver's concentration on driving is decreasing. Therefore, the preset movement may be, for example, a state in which the driver's mouth is continuously moving, but is not limited to this.

[0107] When it is determined that the movement of the driver's mouth includes a preset movement (step S405, Yes), the controller 20 calculates a reduction level (step S406). Specifically, the controller 20 calculates a new reduction level by adding a preset value Z2 to the current reduction level. More specifically, the controller 20 reads the current reduction level from the reduction level information 32 stored in the storage unit 30, and stores the new reduction level obtained by adding the value Z2 to the current reduction level as the reduction level information 32 in the storage unit 30. In other words, the controller 20 updates the reduction level information 32 in the storage unit 30 to the reduction level calculated in step S406.

[0108] The value Z1 added in step S404 and the value Z2 added in step S406 are set to arbitrary values. Furthermore, the value Z1 and the value Z2 may be the same value or different values.

[0109] If it is determined that the movement of the driver's mouth does not include the preset movement (No at step S405), the controller 20 skips the process at step S406.

[0110] In step S405, an example is shown in which it is determined whether or not the driver's concentration on driving is likely to be declining based on the driver's mouth movements, but the present invention is not limited to this. For example, it may be determined whether or not the driver's emotion detected by emotion analysis using emotion recognition AI in step S401 is an emotion that indicates that the driver is feeling excited, such as "joy." Then, the controller 20 may determine that the driver's concentration on driving is likely to be declining if the driver's emotion is "joy," for example.

[0111] Next, the controller 20 reads out the decrease level from the decrease level information 32 stored in the storage unit 30, and determines whether the read decrease level is equal to or greater than the threshold level (step S407). If it is determined that the decrease level is equal to or greater than the threshold level (step S407, Yes), the controller 20 notifies the driver of a warning (step S408; see FIG. 6).

[0112] If it is determined that the decrease level is not equal to or greater than the threshold level (No at step S407), the controller 20 skips the process at step S408.

[0113] If it is determined that the voice in the vehicle cabin does not contain a new keyword (step S402, No), the controller 20 determines whether the keyword timer has reached 0 (step S409). If it is determined that the keyword timer has not reached 0 (step S409, No), the controller 20 returns to the processing of step S400.

[0114] On the other hand, when it is determined that the keyword timer has reached 0 (Yes at step S409), the controller 20 transmits the driving information including the calculated deterioration level to the information processing device 50 (step S410).

[0115] Next, since the possibility of detecting a new keyword has decreased, the controller 20 executes a process of resetting various data in the current process (step S411). Specifically, the controller 20 resets the degradation level to "0 (initial value)" and transmits to the information processing device 50 a reset request for the driving risk level calculated based on the driving information including the degradation level calculated in the current process.

[0116] The process (e.g., the process of calculating the driving risk level) performed by the information processing device 50 that receives the driving information including the deterioration level from the in-vehicle device 10 is the same as that in the first embodiment (see FIG. 7), and therefore will not be described here. The information processing device 50 transmits information on the calculated driving risk level to the in-vehicle device 10.

[0117] When the in-vehicle device 10 receives the driving risk level from the information processing device 50, the in-vehicle device 10 executes the same process as in the first embodiment (see FIG. 8). Specifically, when it is determined in step S300 that the driving risk level has been received, the controller 20 proceeds to step S301, and issues a warning notice according to the received driving risk level.

[0118] The notification of the attention call in step S301 will now be described with reference to Fig. 12. Fig. 12 is a diagram showing an example of the output unit (display unit) 15 that issues a notification of the attention call according to the driving risk level in the second embodiment.

[0119] As shown in FIG. 12 , the controller 20 displays the driving risk level in the display field 15b of the output unit 15 of the in-vehicle device 10. The controller 20 also displays a warning notice corresponding to the driving risk level in the display field 15d of the output unit 15 of the in-vehicle device 10. For example, the display field 15d may display a message such as, "When you were driving near a pedestrian crossing, your concentration on driving may have been reduced by the sounds of music, radio, television, etc. Please concentrate on driving and drive safely." That is, the display field 15d displays a message indicating that a decrease in concentration on driving has been detected and urging the driver to concentrate on driving. This allows the controller 20 to urge the driver to concentrate on driving safely.

[0120] As described above, the controller 20 of the in-vehicle device 10 according to the second embodiment issues a warning notice when the voice contains a preset keyword that affects the driver's emotions.

[0121] In this way, the in-vehicle device 10 can accurately detect a decrease in the driver's concentration on driving and issue a warning by detecting keywords that may be factors that decrease the driver's concentration on driving.

[0122] Furthermore, the controller 20 according to the second embodiment issues a warning when the driver's mouth movements include a preset movement. This allows the controller 20 to accurately detect that the driver's emotions (mood) are elevated, for example, when the driver is humming along to music (audio) playing in the vehicle, and that this may decrease the driver's concentration on driving.

[0123] (Third embodiment) Next, a notification system 1 according to a third embodiment will be described. In the third embodiment, the first embodiment and the second embodiment are combined. Although not shown in the drawings, in the third embodiment, the reduction level calculated in response to the question contained in the voice in the vehicle cabin in the first embodiment is referred to as a "first reduction level," and the reduction level calculated in response to the keyword contained in the voice in the vehicle cabin in the second embodiment is referred to as a "second reduction level."

[0124] Then, the controller 20 according to the third embodiment calculates a value obtained by adding the first and second deterioration levels as a “deterioration level.” As in the first and second embodiments, the controller 20 according to the third embodiment executes various processes, such as a process of issuing a warning notification in accordance with the calculated “deterioration level,” and a process of transmitting driving information including the calculated “deterioration level” to the information processing device 50.

[0125] As a result, the third embodiment can achieve the same effects as the first embodiment and the second embodiment. That is, the third embodiment can accurately detect a decrease in the driver's concentration on driving and issue a warning to alert the driver.

[0126] In the above-described embodiments, the in-vehicle device 10 and the information processing device 50 cooperate to execute various processes, but the present invention is not limited to this. That is, the in-vehicle device 10 may be configured to perform part or all of the processes performed by the information processing device 50. Furthermore, the in-vehicle device 10 may be configured to perform part or all of the processes performed by the information processing device 50.

[0127] Further advantages and modifications will readily occur to those skilled in the art. Therefore, the invention in its broader aspects is not limited to the specific details and representative embodiments shown and described above. Accordingly, various modifications may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and their equivalents. [Explanation of symbols]

[0128] 10 Onboard equipment 20 Controller 50 Information processing equipment 70 Processing device controller

Claims

1. Acquires the sound from inside the vehicle collected by a microphone installed in the vehicle, a controller that notifies a driver of the vehicle of a warning when the acquired voice includes question information indicating a question; A notification device comprising:

2. The question information is The information includes at least one of information indicating a question from the driver to the passenger and information indicating a question from the passenger to the driver. The notification device of claim 1 .

3. The controller notifying the warning when the vehicle is traveling in a predetermined location; The notification device of claim 1 .

4. The controller If the voice includes a keyword that is set in advance and that affects the driver's emotions, the driver is notified of the warning. The notification device of claim 1 .

5. The controller When the movement of the driver's mouth includes a predetermined movement, the driver is notified of the warning. The notification device of claim 1 .

6. When the voice collected by the microphone mounted on the vehicle from within the vehicle includes question information indicating a question, a process of issuing a warning to the driver of the vehicle is executed. Information processing device.

7. Acquires the sound from inside the vehicle collected by a microphone installed in the vehicle, If the acquired voice includes question information indicating a question, notifying a driver of the vehicle of a warning; A program that causes a computer to execute the following.

Images