Notification control device and notification control method

Abstract

This technology provides a way to help occupants understand the in-vehicle functions. [Solution] The notification control device 10 is mounted on a vehicle and includes a notification unit 30 capable of outputting a first notification relating to the vehicle's in-vehicle functions and a second notification subordinate to the first notification, and a decision unit 26 that decides to output a second notification if it is estimated that the vehicle occupant does not understand the content of the first notification after the first notification has been output.

Description

Technical Field

[0001] The present invention relates to a technology for notifying a passenger about in-vehicle functions.

Background Art

[0002] Patent Document 1 discloses a vehicle control system including an information terminal device that displays instructions regarding a vehicle operation method, and a server capable of communicating with the vehicle and the information terminal device. This information terminal device includes a display control unit that displays first content teaching an operation method of a specific function of the vehicle and second content teaching an operation method corresponding to an event occurring during the provision of the specific function, and a determination unit that determines a viewing result of the second content by a user of the vehicle. The specific function is a function related to vehicle control in autonomous driving. The server includes a generation unit that generates release information for releasing a function limitation of a specific function of the vehicle, which is preset, based on the determination result of the determination unit. The vehicle control unit of the vehicle releases the function limitation of the specific function based on the release information.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the technology described in Patent Document 1, the user can release the limitation of the specific function by viewing the first content. However, due to the sophistication of in-vehicle functions, there is a possibility that even if the user has viewed the first content, the user may not understand the content of the specific function when using the specific function.

[0005] An object of the present invention is to provide a technology capable of promoting the understanding of in-vehicle functions for passengers.

Means for Solving the Problems

[0006] To solve the above problems, a notification control device according to one aspect of the present invention includes a notification unit mounted in a vehicle and capable of outputting a first notification relating to the vehicle's in-vehicle functions and a second notification subordinate to the first notification, and a decision unit that, after the first notification has been output, determines to output the second notification if it is estimated that the vehicle occupant does not understand the content of the first notification.

[0007] Another aspect of the present invention is a notification control method. This method is a notification control method in which each step is performed by a computer, comprising the steps of: a notification unit mounted in a vehicle outputting a first notification relating to an in-vehicle function of the vehicle; and, if it is presumed after the first notification has been output that the occupant of the vehicle does not understand the content of the first notification, the notification unit outputting a second notification relating to an in-vehicle function that is subordinate to the first notification. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide a technology that can promote understanding of in-vehicle functions among occupants. [Brief explanation of the drawing]

[0009] [Figure 1] This diagram shows the functional configuration of the notification control system in the embodiment. [Figure 2] This is a flowchart of the notification control method in the embodiment. [Figure 3] This flowchart shows the notification control process after the second notification is output. [Modes for carrying out the invention]

[0010] Figure 1 shows the functional configuration of the notification control system 1 of the embodiment. Each function of the notification control system 1 can be configured in hardware terms using circuit blocks, memory, and other LSIs, and in software terms using system software and application programs loaded into memory. Therefore, it will be understood by those skilled in the art that each function of the notification control system 1 can be implemented in various ways using hardware alone, software alone, or a combination thereof, and is not limited to any one of these.

[0011] The notification control system 1 notifies the vehicle occupants of information regarding in-vehicle functions. Since there are various types of in-vehicle functions, the occupants may not understand the information provided in the initial notification alone. Therefore, the notification control system 1 monitors the occupants' state after the initial notification and, if it estimates that the occupants did not understand the initial notification, it provides additional information to help them understand the initial notification, such as supplementary explanations of the in-vehicle functions. This makes it easier for occupants to understand notifications regarding in-vehicle functions.

[0012] The notification control system 1 comprises a notification control device 10, an in-vehicle camera 12, an in-vehicle sensor 14, and a server device 16. The notification control device 10, the in-vehicle camera 12, and the in-vehicle sensor 14 are mounted on the vehicle. This vehicle may be capable of autonomous driving control.

[0013] The in-vehicle camera 12 captures images of occupants in the vehicle and sends the captured images to the notification control device 10. The occupants may be the driver. The in-vehicle sensors 14 include a sound detection sensor, an operation detection sensor, and a driving state detection sensor, and send the detection results to the notification control device 10. The sound detection sensor detects sounds inside the vehicle and the voices of occupants. The operation detection sensor detects operations performed on the touch panel. The driving state detection sensor detects the driving state of the vehicle. The detection results of the driving state detection sensor can be used to detect abnormalities in the in-vehicle functions. The in-vehicle sensors 14 also detect information to determine whether to illuminate a warning light.

[0014] The server device 16 is capable of communicating with the notification control device 10 and holds explanatory information regarding in-vehicle functions. The server device 16 holds a so-called instruction manual and holds explanatory information for each in-vehicle function for each identification information. The explanatory information for in-vehicle functions has a hierarchical structure, with a first-stage explanatory information and a second-stage explanatory information for each in-vehicle function. The first-stage explanatory information includes the content to be explained first, and the second-stage explanatory information includes content that supplements the first-stage explanatory information. The explanatory information for in-vehicle functions may be at least one of text information and audio information. The server device 16 transmits the explanatory information for in-vehicle functions to the notification control device 10 in response to a request from the notification control device 10.

[0015] The notification control device 10 comprises an acquisition unit 20, an estimation unit 22, a communication unit 24, a determination unit 26, a generation unit 28, and a notification unit 30. The acquisition unit 20 acquires detection results from the in-vehicle camera 12 and the in-vehicle sensor 14. The acquisition unit 20 also acquires signals indicating abnormalities in the in-vehicle functions.

[0016] The decision unit 26 decides to output a first notification regarding the vehicle's onboard functions when it receives instructions from the occupant or when it detects an abnormality in an onboard function. For example, if the occupant requests an explanation of an onboard function via touch input or voice, the decision unit 26 decides to output a first notification. When the occupant is setting an onboard function via touch input, the decision unit 26 may decide to output the onboard function setting screen as the first notification. Furthermore, if the detection result of the onboard sensor 14 indicates an abnormality in any onboard function, the decision unit 26 decides to illuminate a warning light as the first notification.

[0017] The generation unit 28 generates a first notification according to the decision result of the determination unit 26. The notification unit 30 outputs the generated first notification. The notification unit 30 includes a display unit 32 and a sound output unit 34. The display unit 32 includes a display and warning lights. The display can show images such as navigation information, map information, and explanatory information. The sound output unit 34 is, for example, a speaker.

[0018] After the first notification is executed, if it is estimated that the occupant does not understand the content of the first notification, the determination unit 26 determines to output a second notification that is subordinate to the first notification. The second notification is explanatory information of the same in-vehicle function as the subordinate first notification and is content that supplements the first notification. Further, the second notification may include information prompting connection to the operator in order to explain the first notification.

[0019] Upon receiving the output determination of the second notification, the communication unit 24 requests the content of the second notification from the server device 16 and receives the information of the second notification from the server device 16. The communication unit 24 transmits the recognition information of the in-vehicle function corresponding to the first notification and the request for the second notification to the server device 16. The generation unit 28 generates the second notification upon receiving the output determination of the second notification. The notification unit 30 outputs the generated second notification as an image or voice. Thereby, the occupant can receive explanatory information supplementing the first notification and grasp the content of the first notification. Only when it is estimated that the occupant does not understand the content of the first notification, unnecessary output for the notification unit 30 to output the second notification can be suppressed. For example, when the navigation function is being executed, the number of times the navigation display is switched to the second notification can be reduced.

[0020] The estimation unit 22 estimates the state of the occupant immediately after the first notification is output and estimates whether the occupant understands the content of the first notification. The estimation unit 22 extracts the occupant from the captured image using a neural network method, for example, a deep learning method, and analyzes the captured image to estimate the emotion of the occupant. The estimation unit 22 tracks the face image of the occupant shown in consecutive captured images and calculates the anxiety level of the occupant as a score based on the change in the face image of the occupant. The estimation unit 22 estimates, by image analysis, an expression of the occupant showing a sense of uneasiness such as confusion, disarray, dissatisfaction, etc.

[0021] The estimation unit 22 sets a large number of points on, for example, the face image of the occupant using a mesh, tracks each point on the face image, and calculates an instability score based on the displacement of each point. For example, the estimation unit 22 sets points in 20 columns each in the vertical and horizontal directions on the face image of the occupant and tracks the points immediately after the first notification. When the calculated instability score is greater than or equal to a predetermined threshold, the estimation unit 22 estimates that the occupant does not understand the content of the first notification. The estimation unit 22 sends the estimation result to the determination unit 26. Instead of calculating only one score, i.e., the instability, the estimation unit 22 may calculate a plurality of scores such as perplexity, confusion, dissatisfaction, etc. When the total score of these is greater than or equal to a predetermined threshold, the estimation unit 22 estimates that the occupant does not understand the content of the first notification. The estimation unit 22 may calculate the instability score using a learning model that estimates the expression from the face image.

[0022] The estimation unit 22 estimates whether the occupant understands the content of the first notification based on the expression of the occupant shown in the captured image and the movement of the occupant looking at the display unit 32. That is, the estimation unit 22 estimates whether the occupant understands the content of the first notification based on the expression and movement of the occupant. For example, when the display unit 32 is displaying the first notification, if the occupant makes an action of gazing at the display unit 32 or confirming the display unit 32 multiple times, and the instability calculated based on the expression of the occupant is greater than or equal to a predetermined threshold, the estimation unit 22 estimates that the occupant does not understand the content of the first notification. Thereby, the estimation unit 22 can accurately estimate that the occupant does not understand.

[0023] After the second notification, the notification unit 30 asks the occupant whether they understand the first notification. For example, after the second notification, the notification unit 30 outputs a question such as "Was the second notification necessary?" on at least one of the display unit 32 and the sound output unit 34, and outputs a question to confirm whether the occupant understood the first notification. The vehicle-mounted sensor 14 acquires the occupant's answer to the question. The estimation unit 22 learns a learning model for calculating the instability score based on the occupant's answer result to the question. Thereby, the estimation accuracy of the necessity of the second notification can be improved.

[0024] The second notification may include not only explanatory information about the in-vehicle functions but also instructions on how to connect a call to an operator. This would allow the occupant to learn detailed instructions on how to operate the in-vehicle functions corresponding to the first notification by speaking with an operator.

[0025] Figure 2 is a flowchart of the notification control method of the embodiment. The acquisition unit 20 acquires detection results from the in-vehicle camera 12 and the in-vehicle sensor 14 (S10). The decision unit 26 decides to output a first notification based on the detection results of the in-vehicle sensor 14 (S12). The generation unit 28 receives the decision result and generates the first notification, and the notification unit 30 outputs the generated first notification (S14).

[0026] The estimation unit 22 detects the occupant's state for a predetermined time after outputting the first notification, based on the image captured by the in-vehicle camera 12 (S16). The estimation unit 22 analyzes the captured image to calculate the occupant's level of anxiety.

[0027] The estimation unit 22 determines whether the calculated occupant's anxiety level is above a predetermined threshold (S18). If the occupant's anxiety level is not above a predetermined threshold (N in S18), the process ends without outputting the second notification. If the occupant's anxiety level is above a predetermined threshold (Y in S18), the communication unit 24 transmits to the server device 16 the recognition information of the in-vehicle function corresponding to the first notification currently being output, and information requesting explanatory information of the in-vehicle function, and obtains the information of the second notification which is subordinate to the first notification (S20).

[0028] The estimation unit 22 analyzes the captured image while the first notification is displayed to determine whether the occupant performs an anxious action such as staring at or repeatedly checking the display unit 32 (S22). If the occupant does not perform an anxious action (N in S22), the process ends without outputting the second notification. If the occupant performs an anxious action (Y in S22), the decision unit 26 decides to output the second notification, and the notification unit 30 outputs the second notification (S24). Note that the decision process in S22 is not required. Also, if an anxious action by the occupant is detected, the display time of the second notification may be set to be longer than when no anxious action is detected.

[0029] Figure 3 is a flowchart showing the notification control process after the output of the second notification. The notification unit 30 outputs the second notification (S30). The decision unit 26 monitors the occupant's cancellation input for the second notification based on the detection result of the in-vehicle sensor 14 (S32). For example, the decision unit 26 monitors whether the occupant operates the touch panel to press a button to dismiss the second notification or gives a voice command to dismiss the display of the second notification.

[0030] If the second notification is not canceled by the occupant (N in S32), the notification unit 30 outputs the second notification until a predetermined time has elapsed since outputting the second notification (N in S34). Once the predetermined time has elapsed (Y in S34), the notification unit 30 stops outputting the second notification and the display unit 32 switches back to displaying the original screen (S36). The original screen is, for example, the navigation screen. If the second notification is canceled by the occupant (Y in S32), the display unit 32 switches back to displaying the original screen (S36).

[0031] After outputting the second notification, the notification unit 30 outputs a question to the crew to confirm whether they understood the first notification (S38). The estimation unit 22 learns a model to determine whether the crew understood based on their response (S40).

[0032] The present disclosure has been explained above based on the examples described. The present disclosure is not limited to the examples described above, and various modifications such as design changes can be made based on the knowledge of those skilled in the art. [Explanation of symbols]

[0033] 1 Notification control system, 10 Notification control device, 12 In-vehicle camera, 14 In-vehicle sensor, 16 Server device, 20 Acquisition unit, 22 Estimation unit, 24 Communication unit, 26 Determination unit, 28 Generation unit, 30 Notification unit, 32 Display unit, 34 Sound output unit.

Claims

1. A notification unit mounted in a vehicle and capable of outputting a first notification relating to the vehicle's onboard functions and a second notification subordinate to the first notification, A notification control device comprising: a determination unit that determines to output a second notification if it is estimated that the occupant of the vehicle does not understand the content of the first notification after the first notification has been output.

2. An acquisition unit that acquires captured images of the aforementioned crew members, The notification control device according to claim 1, further comprising: an estimation unit that analyzes the captured image to estimate the emotions of the occupant and estimates whether the occupant understands the content of the first notification.

3. The notification unit includes a display unit capable of displaying an image, The notification control device according to claim 2, characterized in that the estimation unit estimates whether the occupant understands the content of the first notification based on the occupant's facial expression shown in the captured image and the occupant's actions when viewing the display unit.

4. The notification control device according to claim 2, characterized in that the estimation unit tracks the occupant's facial expression when outputting the first notification based on the captured image, calculates a score indicating the occupant's level of anxiety, and estimates that the occupant does not understand the content of the first notification if the score indicating the level of anxiety is above a predetermined threshold.

5. A notification control method in which each step is performed by a computer, The process involves a notification unit installed in the vehicle outputting a first notification regarding the vehicle's in-vehicle functions, A notification control method characterized by including the step of, if it is presumed that the occupant of the vehicle does not understand the content of the first notification after the first notification has been output, the notification unit outputs a second notification that is subordinate to the first notification with respect to the in-vehicle function.

Images