Information notification method and information notification device

Abstract

To restrain a user from being confused by using a display for guiding the user's visual line in a direction of a target object which has not been noticed by the user.SOLUTION: In an information notification method, a direction of a visual line 50 of an occupant of a vehicle, and a target object 60 that should be noticed by the occupant are detected respectively. If the detected visual line 50 is determined to be not moving toward a position of the target object 60, markers 41, 82 are displayed at a target position on the visual line 50 relative to the detected target object 60, on a display unit 40 arranged within a visual field of the occupant.SELECTED DRAWING: Figure 3E

Description

Technical Field

[0001] The present invention relates to an information notification method and an information notification device.

Background Art

[0002] There has been proposed a technique in which a risk target to be noted by a driver of a vehicle is detected from an image or the like photographed by a camera, and a light-emitting spot located on the driver's line of sight with respect to the risk target in a linear light-emitting area of an instrument panel is lit (Patent Document 1). In this technique, the lit light-emitting spot guides the driver's line of sight in the direction of the risk target. When it is detected that the driver has directed the line of sight in the direction of the light-emitting spot, the light-emitting spot corresponding to the next risk target in the linear light-emitting area is lit.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the technique of Patent Document 1, when a risk target is detected from an image or the like photographed by a camera, the light-emitting spot is lit. Since this lighting is started regardless of the behavior of the driver who moves the line of sight to notice the risk target, there is a possibility that the driver may feel that the guidance of the line of sight has started inadvertently and may be confused.

[0005] A driver of a vehicle finely moves the line of sight to visually recognize an object in the field of view in various scenes. The driver visually captures the target to be noted while moving the line of sight and reflects it in the driving operation of the vehicle. Therefore, when a vehicle occupant such as a driver does not notice the target to be noted while moving the line of sight, it is desirable to enable the target to be grasped without confusion.

[0006] The present invention has been made in view of the above circumstances, and an object of the present invention is to suppress confusion of a vehicle occupant caused by a display that guides the occupant's line of sight in a direction of a target of interest that the occupant is not aware of.

Means for Solving the Problems

[0007] In order to solve the above-described problems, in an information notification method according to one aspect of the present invention, a line-of-sight direction of a vehicle occupant and a target of interest to be noted by the occupant are detected. When the target of interest is detected and the line-of-sight direction does not move toward a position on the line of sight with respect to the detected target of interest, first control is performed. In the first control, a marker indicating a position on the line of sight with respect to the target of interest is displayed on a display unit within the occupant's field of view.

Effects of the Invention

[0008] According to the present invention, it is possible to suppress confusion of a vehicle occupant caused by a display that guides the occupant's line of sight in a direction of a target of interest that the occupant is not aware of.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3A

Figure 3B

Figure 3C

Figure 3D

Figure 3E

Figure 3F

Figure 4

Figure 5

Figure 6

Figure 7A

Figure 7B

Figure 7C

Figure 7D

Figure 7E

Figure 7F

Figure 7G

Figure 7H

Figure 7I

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the description of the drawings, the same parts are denoted by the same reference numerals and the description thereof will be omitted.

[0011] (First Embodiment) With reference to FIG. 1, the configuration of the information notification device 10 according to the first embodiment will be described. As shown in FIG. 1, the information notification device 10 according to the present embodiment is mounted on a vehicle together with the peripheral monitoring device 30. The information notification device 10 shown in FIG. 1 executes the information notification method according to the first embodiment of the present invention. The information notification device 10 in FIG. 1 performs a display that guides the line of sight of the vehicle occupant in the direction of the target of interest that the vehicle occupant has not noticed.

[0012] The peripheral monitoring device 30 includes a surrounding recognition sensor 31 and an image generation unit 32.

[0013] The surrounding recognition sensor 31 may include, for example, a camera that captures the front area of the vehicle that can be seen by the vehicle occupant from inside the vehicle, a radar, and a lidar (LiDAR). The radar and the lidar output an output wave and receive a reflected wave from an object irradiated with the output wave, thereby detecting the position of an object existing in the front area of the vehicle. Since the radar outputs an electromagnetic wave with a long wavelength and the lidar outputs an electromagnetic wave with a short wavelength, ranges with low mutual detection sensitivity can complement each other.

[0014] The objects detected by the radar and the lidar may include pedestrians. The objects detected by the radar and the lidar are candidates for objects that the occupant should pay attention to. The objects to be the target of attention can be detected by the target object detection unit 11 of the information notification device 10 described later.

[0015] The image generation unit 32 generates a superimposed image of the video of the front area of the vehicle captured by the camera of the surrounding recognition sensor 31 and a synthetic image that simulates the objects in the front area of the vehicle detected by the radar and the lidar of the surrounding recognition sensor 31.

[0016] The information notification device 10 can be realized by using a microcomputer including a CPU (Central Processing Unit), a memory, and an input / output unit. A computer program for causing the microcomputer to function as the information notification device 10 is installed and executed in the microcomputer. Thereby, the microcomputer can function as a plurality of information processing units (11 to 17) provided in the information notification device 10.

[0017] Here, an example of realizing the information notification device by software is shown. Of course, it is also possible to prepare dedicated hardware for executing each information process and configure the information notification device 10. The dedicated hardware includes devices such as an application-specific integrated circuit (ASIC) arranged to execute the functions described in the embodiment and conventional circuit components.

[0018] Although the information notification device 10 and the peripheral monitoring device 30 have been described as different members, of course, the two devices 10 and 20 may be configured as one device. Alternatively, a plurality of information processing units (11 to 17) may be divided and configured using two or more different devices. Further, all or part of the plurality of information processing units (11 to 17) may be configured using an ECU (Electronic Control Unit) mounted on the vehicle.

[0019] The information notification device 10 includes, as part of a plurality of information processing units (11 to 17), an object detection unit 11, a line-of-sight direction detection unit 12, a line-of-sight determination unit 13, a corresponding position calculation unit 14, and a marker display unit 15.

[0020] The object detection unit 11 (attention target detection unit) analyzes the superimposed image generated by the image generation unit 32 of the peripheral monitoring device 30 and detects the position of an object (attention target) existing in the front area of the vehicle. The occupant is, for example, the driver of the vehicle. A passenger other than the driver may also be the occupant. When there are a plurality of objects in the front area of the vehicle, the object detection unit 11 detects the position of each object.

[0021] The object whose position is detected by the object detection unit 11 is an object that the occupant (especially the driver) should pay attention to for safe driving. The objects to be noted (targets of attention) can include, for example, pedestrians, obstacles, etc. The object can be detected by the radar and lidar of the surrounding recognition sensor 31 of the peripheral monitoring device 30.

[0022] Based on the positions of the objects detected by analyzing the superimposed image of the image generation unit 32, the object detection unit 11 detects the direction of each object as seen from the position of the occupant's eyes (viewpoint).

[0023] For example, the line-of-sight direction detection unit 12 detects the line-of-sight direction of the occupant based on the in-vehicle camera that captures the occupant's eyeballs and the relative position of the occupant's eyes (viewpoint) with respect to the in-vehicle camera from the eyeball center position and pupil center position in the captured image of the in-vehicle camera.

[0024] Based on the line-of-sight direction of the occupant detected by the line-of-sight direction detection unit 12, the line-of-sight determination unit 13 performs various determinations related to the occupant's line of sight. For this purpose, the line-of-sight determination unit 13 includes a switching detection unit 16 and a switching notification unit 17. These units 16 and 17 are also part of the plurality of information processing units (11 to 17).

[0025] The switching detection unit 16 detects that the line-of-sight direction of the occupant is moving toward the position on the line of sight with respect to the object of attention based on the line-of-sight direction of the occupant detected by the line-of-sight direction detection unit 12 in a state where the object detection unit 11 has detected an object that should be noted. The position on the line of sight with respect to the object of attention can be, for example, the position on the line of sight with respect to the object of attention of the display unit described later (target position).

[0026] When the switching detection unit 16 determines that it has not detected the movement of the occupant's line-of-sight direction toward the position on the line of sight with respect to the object of attention, the switching notification unit 17 outputs a notification signal for notifying a control switch to the corresponding position calculation unit 14. For example, the switching notification unit 17 may output a notification signal to the corresponding position calculation unit 14 when the switching detection unit 16 has not detected the movement of the occupant's line-of-sight direction toward the position on the line of sight with respect to the object of attention until a preset time has elapsed since the object of attention was first detected.

[0027] The corresponding position calculation unit 14 calculates the position of the marker to be displayed on the display unit. The corresponding position calculation unit 14 switches the position of the marker to be calculated according to whether a notification signal is input from the switching notification unit 17.

[0028] Specifically, before the input of the notification signal, the corresponding position calculation unit 14 calculates the position of the marker that indicates, with a predetermined delay amount, the position corresponding to the line-of-sight direction of the occupant detected by the line-of-sight direction detection unit 12 on the display unit of the marker (the corresponding position of the detected line-of-sight direction). After the input of the notification signal, the corresponding position calculation unit 14 calculates the position of the marker that guides the line of sight to the target object detected by the target object detection unit 11 on the display unit of the marker. The corresponding position calculation unit 14 can calculate the intersection position between the line of sight of the occupant and the display surface of the display unit as the position of the marker on the display unit.

[0029] The marker display unit 15 displays a marker indicating the position calculated by the corresponding position calculation unit 14 on the display unit inside the vehicle. The marker display unit 15 generates a video for displaying the image of the marker at the position calculated by the corresponding position calculation unit 14, and displays the generated video on the display unit. The marker can be of any shape such as a circle, polygon, symbol, character, etc. The character can be, for example, an avatar simulating an arbitrary item. The item simulated by the avatar can be a physically existing one or a virtual one represented by an image.

[0030] The display unit can be arranged within the field of view of the occupant. For example, the front windshield of the vehicle can be used as the display unit. When the front windshield is used as the display unit, the marker display unit 15 projects the image of the marker at the position calculated by the corresponding position calculation unit 14.

[0031] With reference to the flowchart of FIG. 2, an example of the information notification method executed by the information notification device 10 of the present embodiment will be described. The information notification device 10 can execute the processing of the flowchart of FIG. 2, for example, when the vehicle is in a drivable state.

[0032] The information notification device 10 detects the line-of-sight direction of the occupant by the line-of-sight direction detection unit 12 (step S1). The information notification device 10 causes the corresponding position calculation unit 14 to calculate the position of the intersection where the line of sight in the detected line-of-sight direction intersects the projection surface (the inner surface of the vehicle) of the windshield on which the marker display unit 15 projects the video (step S3). The information notification device 10 records the intersection position calculated by the corresponding position calculation unit 14 (step S5).

[0033] The information notification device 10 applies a low-pass filter to the locus of the recorded intersection positions to smooth the locus of the intersection positions (step S7). The information notification device 10 causes the marker display unit 15 to display a marker on the projection surface of the windshield with a preset delay time (delayed by a predetermined delay amount) in the smoothed locus of the intersection positions (step S9).

[0034] The information notification device 10 checks whether there is an object of attention (object of interest) around the line-of-sight direction of the occupant detected by the line-of-sight direction detection unit 12 (step S11).

[0035] If it cannot be confirmed that there is an object of attention around (NO in step S11), the process returns to step S1. If it is confirmed that there is an object of attention around (YES in step S11), the information notification device 10 checks whether the occupant's line of sight is was not intended for directed at the object of attention or not (step S13). The information notification device 10 can confirm that the occupant has directed their line of sight at the object of attention by the direction of the occupant's line of sight detected by the line-of-sight direction detection unit 12 changing from a direction other than the object of attention to the direction of the object of attention.

[0036] If the occupant's line of sight is was intended for not directed at the object of attention (NO in step S13), the process returns to step S1. If the occupant's line of sight is was not intended forIf so (YES in step S13), the information notification device 10 causes the marker display unit 15 to display a marker so as to guide the occupant's line of sight in the direction of the target of attention (step S15). In this guiding display, the information notification device 10 moves the marker that is showing the corresponding position on the projection plane behind the occupant's line of sight to the current corresponding position, and then further moves the marker to the position on the projection plane that is the intersection point with the line of sight when the occupant's line of sight is directed at the target of attention.

[0037] The information notification device 10 checks whether the occupant has directed their line of sight at the target of attention (step S17). If the occupant has not directed their line of sight at the target of attention (NO in step S17), the process returns to step S15. If the occupant has directed their line of sight at the target of attention (YES in step S17), the process returns to step S1.

[0038] In a vehicle equipped with the information notification device 10, by the second control, as shown in FIG. 3A, a circular marker 41 indicating the corresponding position in the line-of-sight direction existing on the occupant's line of sight 50 is normally displayed on the windshield 40. When the occupant's line of sight 50 does not move towards, for example, a pedestrian outside the vehicle detected by the radar and lidar of the surrounding recognition sensor 31, the marker 41 moves on the projection plane of the windshield 40 in accordance with the movement of the line of sight 50, as shown in FIG. 3B. The marker 41 moves on the locus 51 of the corresponding position according to the movement of the line of sight 50 so as to follow with a predetermined delay amount the corresponding position on the line of sight 50 on the projection plane.

[0039] Unless the occupant's line of sight 50 moves by a predetermined delay amount or more, the marker 41 that has followed the movement of the line of sight 50 with a predetermined delay amount catches up with the line of sight 50 and is displayed at the corresponding position on the line of sight 50, as shown in FIG. 3C.

[0040] Also, when the object detection unit 11 detects a pedestrian who is in a direction different from the direction of the driver's line of sight 50 as an object of attention 60, the first control is performed. In the first control, the marker display unit 15 moves the marker 41 on the projection surface toward the target position on the driver's line of sight 50 with respect to the object of attention 60 on the projection surface, as shown in FIG. 3D, regardless of the direction of the driver's line of sight 50. The display of the marker 41 by the first control can be performed to guide the driver's line of sight 50 to the object of attention 60.

[0041] When the marker 41 that has been following behind the driver's line of sight 50 moves on the projection surface toward the object of attention 60 in a direction different from the direction of the driver's line of sight 50, the marker display unit 15 may move the marker 41 linearly at the shortest distance. The marker display unit 15 may move the marker 41 such that the moving direction of the marker 41 continuously changes during the movement.

[0042] As shown in FIG. 3E, the marker display unit 15 may move the marker 41 on the projection surface through a path of a curved trajectory. The projection surface (the inner surface of the vehicle) of the windshield 40 is not a two-dimensional flat surface but has curvatures in the vertical direction and the vehicle width direction of the vehicle, respectively. The curve of the moving trajectory of the marker 41 shown in FIG. 3E is not a curve caused by the curvature of the projection surface. The marker display unit 15 moves the marker 41 on the projection surface along a trajectory that would be curved even if the marker 41 were displayed on a two-dimensional display surface.

[0043] The marker display unit 15 may move the marker 41 through a preset detour area 42 within the projection surface and toward the object of attention 60. The detour area 42 can be set, for example, near the outer edge of the projection surface.

[0044] Note that after the driver's line of sight 50 is directed to the object of attention 60 by the guidance of the marker 41, the marker 41 may be displayed again at a position that follows the movement of the line of sight 50 with a predetermined delay amount.

[0045] In the first control, the marker display unit 15 may display a marker with a display mode different from that of the marker 41 (Figs. 3A to 3D) in the second control. The marker with a different display mode may have a shape that is more visually prominent than the circular marker 41, such as the marker 43 shown in Fig. 3F. Further, the display mode may be, in addition to the shape of the marker, color, size, moving speed on the projection surface, etc.

[0046] The change in the display mode of the markers 41 and 43 may be performed when a preset time has elapsed since the detection of the object of attention 60 (target of attention) by the object detection unit 11, when a certain time has elapsed since the end of the guidance by the marker 41, or when the line of sight 50 of the occupant does not face the object of attention 60. For example, by changing the display mode of the markers 41 and 43, the occupant can be notified of the switch from the second control to the first control.

[0047] In the information notification device 10 of the present embodiment, when the object detection unit 11 detects the object of attention 60, if the line of sight 50 of the occupant moves toward the object of attention 60, the second control is continued, and the marker 41 is displayed with a predetermined delay amount at the corresponding position of the line of sight 50 of the occupant on the projection surface. If the line of sight 50 of the occupant does not move toward the object of attention 60 even when the object of attention 60 is detected, the control switches from the second control to the first control, and the markers 41 and 43 are displayed at the target position corresponding to the position of the object of attention 60 without delay.

[0048] Therefore, when the occupant notices the object of attention 60 and moves the line of sight 50 toward the object of attention 60, the control switches from the second control to the first control, the positions of the markers 41 and 43 move to the target position, and it is possible to suppress the occupant from being confused about the positions indicated by the markers 41 and 43.

[0049] Also, when switching from the second control to the first control, since the markers 41 and 43 reach the target position while moving on the projection surface, when guiding the line of sight 50 to the object of attention 60 at the target position, the occupant can smoothly move the line of sight 50 in accordance with the movement of the markers 41 and 43. Furthermore, when switching from the second control to the first control, the movement of the markers 41 and 43 can continuously guide the occupant's line of sight 50 from the display position in the second control to the target position.

[0050] When switching to the first control, if the markers 41 and 43 are moved to the target position such that the moving directions of the markers 41 and 43 change continuously, the occupant can be made to gently change the direction of the line of sight 50 toward the target position, rather than moving directly to the target position in a straight line at the shortest distance. Even when the markers 41 and 43 are moved to the target position via a path of a curved trajectory when switching to the first control, a similar effect can be obtained. Also, the movement path of the markers 41 and 43 can be determined by a well-known logic in the marker display unit 15, reducing the processing burden on the marker display unit 15.

[0051] When switching to the first control, if the markers 41 and 43 are moved to the target position passing through the detour area 42, it is possible to suppress the situation where the occupant is induced by the markers 41 and 43 and has to suddenly change the direction of the line of sight 50 greatly.

[0052] Also, by making the display modes of the markers 41 and 43 different between the first control and the second control, for example, the switching from the second control to the first control can be notified to the occupant by the display mode of the markers 41 and 43 displayed on the projection surface. Also, the meaning of the position where the markers 41 and 43 are displayed can be easily recognized by the occupant from the display mode of the markers 41 and 43.

[0053] (Second and Third Embodiments) Referring to FIG. 4, the configuration of the information notification device 10 according to the second and third embodiments will be described. The information notification device 10 according to the second and third embodiments can be realized using a microcomputer including a CPU (Central Processing Unit), a memory, and an input / output unit. The microcomputer of the information notification device 10 can function as a plurality of information processing units (11 to 22) provided in the information notification device 10.

[0054] Similar to the information notification device 10 of the first embodiment shown in FIG. 1, the information notification device 10 of the second and third embodiments has an object detection unit 11, a line-of-sight direction detection unit 12, a line-of-sight determination unit 13, a corresponding position calculation unit 14, and a marker display unit 15. The information notification device 10 of the second and third embodiments further has a travel information acquisition unit 18, a travel / visual recognition data storage unit 19, and a notification unit 20. Hereinafter, the parts different from the information notification device 10 of the first embodiment will be described.

[0055] The object that the object detection unit 11 of the second and third embodiments detects the position of is a landmark (a mark, an object of attention) in vehicle route guidance. Based on the vehicle route guidance information acquired by the travel information acquisition unit 18 described later, the object detection unit 11 can identify a landmark to guide the occupant in route guidance and detect the position of the identified landmark.

[0056] The landmark can include, for example, an intersection, a building at the corner of an intersection, etc. In order to identify the position of the landmark, the object detection unit 11 includes a position information acquisition unit 21. The position information acquisition unit 21 is also a part of the plurality of information processing units (11 to 22).

[0057] The position information acquisition unit 21 acquires the position information of the landmark to be guided in the guidance. The position information of the landmark includes, for example, information such as the latitude, longitude, altitude, etc. of each landmark. In order to acquire the position information of the landmark, the position information acquisition unit 21 has map information and the vehicle's position information. The map information includes node and link information regarding roads and intersections.

[0058] The position of the vehicle can be detected, for example, using the output of a GNSS (Global Navigation Satellite System) sensor. The GNSS sensor may be possessed by the information notification device 10 or the peripheral monitoring device 20, or may be possessed by the vehicle equipped with the information notification device 10.

[0059] The object detection unit 11 can identify landmarks in the superimposed image, for example, by analyzing the superimposed image generated by the image generation unit 22 of the peripheral monitoring device 20, based on the positions of the objects detected by the radar and lidar in the superimposed image and the position of the vehicle.

[0060] The line-of-sight determination unit 13 makes various determinations regarding the line of sight 50 of the occupant based on the direction of the line of sight of the occupant detected by the line-of-sight direction detection unit 12. The line-of-sight determination unit 13 of the second and third embodiments includes a focus detection unit 22 instead of the switching detection unit 16 and the switching notification unit 17 of the first embodiment. The focus detection unit 22 is also part of a plurality of information processing units (11 to 22).

[0061] The focus detection unit 22 determines whether the direction of the line of sight 50 of the occupant detected by the line-of-sight direction detection unit 12 has moved toward a landmark that the occupant should focus on in the vehicle route guidance.

[0062] The focus detection unit 22 determines whether the line of sight 50 of the occupant is moving toward any of the objects existing in the superimposed image of the image generation unit 32 in order to determine whether the direction of the line of sight 50 of the occupant has moved toward a landmark.

[0063] When the line-of-sight direction detected by the line-of-sight direction detection unit 12 approaches the direction of the landmark guided in the vehicle route guidance, the attention detection unit 22 can determine that the passenger's line of sight 50 has moved toward the landmark. The attention detection unit 22 can use the change in the speed (the moving speed of the line of sight) at which the line-of-sight direction detected by the line-of-sight direction detection unit 12 approaches the direction of the landmark as seen from the passenger's viewpoint as a condition for determining that the passenger's line of sight 50 has moved toward the landmark. The change in the moving speed of the line of sight 50 can be, for example, a decrease in the moving speed of the line of sight 50.

[0064] Based on the positional relationship between the positions corresponding to the landmark and the passenger's line of sight 50 in the marker display section by the marker display section 15, the attention detection unit 22 can determine whether the passenger's line of sight 50 has moved toward the landmark. The corresponding position of the passenger's current line of sight 50 in the display section is the intersection position between the passenger's current line of sight 50 and the display surface of the display section. The position corresponding to the landmark in the display section is the intersection position between the line of sight 50 when the passenger's line of sight 50 is directed toward the landmark and the display surface of the display section.

[0065] The line-of-sight determination unit 13 can obtain the required time until the passenger directs the line of sight 50 toward the landmark in the route guidance where the object detection unit 11 has detected the position. The line-of-sight determination unit 13 can store the obtained required time in the travel / visual recognition data storage unit 19 described later.

[0066] The corresponding position calculation unit 14 calculates the position of the marker to be displayed on the display section. The corresponding position calculation unit 14 calculates the position of the marker that makes the passenger notice the landmark for which the position information acquisition unit 21 of the object detection unit 11 has acquired the position information in the marker display section. The corresponding position calculation unit 14 can calculate, for example, the intersection position between the line of sight 50 when the passenger directs the line of sight 50 toward the landmark and the display surface of the display section as the position of the marker in the display section.

[0067] The driving information acquisition unit 18 acquires information regarding route guidance to the destination of the vehicle guided to the vehicle occupants. The driving information acquisition unit 18 can acquire, as the route guidance information, for example, route guidance information by a car navigation system or the like mounted on the vehicle.

[0068] The driving / visual recognition data storage unit 19 can store the required time acquired by the visual recognition determination unit 13 as log information regarding the reaction of the occupant. Further, the driving / visual recognition data storage unit 19 can store the route guidance information acquired by the driving information acquisition unit 18 as log information regarding the past performance of route guidance.

[0069] The driving / visual recognition data storage unit 19 can be configured by an accessible storage device. The storage device can be configured by, for example, an SSD (Solid State Drive) or an HDD (Hard Disk Drive).

[0070] The notification unit 20 can generate an audio signal for guiding the occupant output from a support device 80 (see FIGS. 7A to 7I) described later. The support device 80 can be assumed to have, for example, an audio output function including a speaker.

[0071] With reference to the flowcharts of FIGS. 5 and 6, an example of an information notification method respectively executed by the information notification device 10 of the second and third embodiments will be described. The information notification device 10 can execute the processing of the flowchart of FIG. 5 or FIG. 6, for example, when the vehicle is in a drivable state.

[0072] As shown in the flowchart of FIG. 5, the information notification device 10 of the second embodiment detects the line-of-sight direction of the occupant by the line-of-sight direction detection unit 12 (step S21). The information notification device 10 detects, based on the route guidance information acquired by the driving information acquisition unit 18, a landmark to guide the occupant in the route guidance to the occupant by the object detection unit 11 (step S23).

[0073] The information notification device 10 conducts guidance for guiding the detected landmarks (step S25). This guidance can be, for example, by means of a voice message output from the built-in speaker of the support device 80 using the voice signal generated by the notification unit 20. This guidance can be performed, for example, until the vehicle arrives at the destination to be guided in route guidance. Therefore, in the guidance, a plurality of landmarks existing on the route to the destination can be guided as needed.

[0074] The information notification device 10 checks whether the passenger has visually recognized all the landmarks guided in the conducted guidance (step S27). Whether the passenger has visually recognized a landmark can be detected by the attention detection unit 22 of the gaze determination unit 13. If the passenger has visually recognized all the landmarks (YES in step S27), the series of procedures is terminated. If the passenger has not yet visually recognized some or all of the landmarks (NO in step S27), the information notification device 10 checks whether the passenger has visually recognized the first (the first) landmark (step S29).

[0075] If the passenger has not visually recognized the first landmark (NO in step S29), the information notification device 10 displays an avatar image at the position corresponding to the first landmark on the marker display unit (step S31), and returns to step S27. When step S29 has been looped a certain number of times since the first visual confirmation, or when a certain period of time has elapsed since the first visual confirmation, step S27 may be forcibly skipped assuming that the passenger has visually recognized the first landmark.

[0076] When the passenger has visually recognized the first landmark (YES in step S29), etc., the information notification device 10 similarly checks in sequence whether the passenger has visually recognized the landmarks from the second one onwards.

[0077] When the occupant does not visually recognize the landmark at the last nth position (the nth landmark) guided in the implemented guidance (NO in step S33), the information notification device 10 displays an avatar image at the position corresponding to the nth landmark on the display unit of the marker (step S35), and returns to step S27.

[0078] In the information notification method executed by the information notification device 10 according to the third embodiment shown in the flowchart of FIG. 6, the content of the procedure when the occupant does not visually recognize the landmark (NO in step S29 and step S33) is slightly different from that of the information notification device 10 according to the second embodiment.

[0079] In the information notification device 10 according to the second embodiment, in steps S31 and S35 of FIG. 5, an avatar image was displayed at the position corresponding to the landmark that was not visually recognized. In the information notification device 10 according to the third embodiment, as shown in steps S32 and S36 of FIG. 6, in addition to displaying the avatar image, guidance indicating the position of the landmark where the avatar image is to be displayed is implemented. This guidance can be by a voice message output from the built-in speaker of the support device 80 using the voice signal generated by the notification unit 20, similar to the guidance implemented in step S25.

[0080] In the vehicle equipped with the information notification device 10 according to the second embodiment, as shown in FIG. 7A, in route guidance, when the vehicle approaches the next intersection 70 where a right turn is to be made, a voice message 81 guiding the blue building 71 at the corner of the intersection 70 is output from the support device 80. As shown in FIG. 7B, when the occupant directs the line of sight 50 to the building 71 at the intersection 70, no avatar image is displayed at the position of the building 71.

[0081] Thereafter, as shown in FIG. 7C, when the occupant directs the line of sight 50 in the direction of the road after turning right at the intersection 70, a voice message 81 for guidance regarding the subsequent route guidance is output by the support device 80 according to the movement of the vehicle.

[0082] As shown in FIG. 7D, when the occupant directs the line of sight 50 not to the intersection 70 but to the intersection 72 straight ahead of the intersection 70, as shown in FIG. 7E, an avatar image 82 indicating a landmark is displayed as a marker at the position of the road after turning right at the intersection 70. The avatar image 82 can be, for example, an image of a character simulating the assistance device 80. As shown in FIG. 7F, when the occupant directs the line of sight 50 to the avatar image 82 on the road after turning right at the intersection 70, the display of the avatar image 82 ends.

[0083] When the avatar image 82 is not displayed at the position of the landmark, the avatar image 82 may be displayed at the reference position of the windshield 40, or the display of the avatar image 82 may be ended. The reference position can be, for example, the right corner of the windshield 40 that is difficult to enter the occupant's field of view, as shown in FIGS. 7A to 7D, so as to ensure a large field of view for driving.

[0084] When guiding the building 71 at the intersection 70 to turn right as a landmark, if the occupant directs the line of sight 50 to the building 73 at the corner of the next intersection 72 as shown in FIG. 7G, as shown in FIG. 7H, the avatar image 82 is displayed at the position of the building 71 at the intersection 70. When the occupant directs the line of sight 50 to the building 71, the display of the avatar image 82 at the position of the building 71 ends.

[0085] Also in the information notification device 10 of the third embodiment, similar to the information notification device 10 of the second embodiment, the avatar image 82 is displayed in the patterns shown in FIGS. 7A to 7H according to the content of the guidance of the route guidance.

[0086] Also, in the information notification device 10 of the third embodiment, for example, when the occupant does not direct the line of sight 50 to the avatar image 82 on the road at the destination of turning right at the intersection 70 as shown in FIG. 7E and directs the line of sight 50 to the next intersection 72, guidance is provided for the position of the avatar image 82. This guidance can be provided, for example, by displaying a speech bubble screen 83 of "This way" near the avatar image 82 being displayed on the projection surface as shown in FIG. 7I. Guidance may also be provided by outputting an audio message by the support device 80. Thereby, it is possible to make it easier for the occupant to visually recognize the position of the landmark.

[0087] In the second and third embodiments, the required time acquired by the visual recognition determination unit 13 may be accumulated in the travel / visual recognition data storage unit 19 as log information regarding the reaction of the occupant as the reaction time from the display of the avatar image 82 until the occupant directs the line of sight 50 to the avatar image 82. Then, when the occupant does not direct the line of sight 50 to the avatar image 82 exceeding, for example, the average value of the reaction times of the accumulated log information when the avatar image 82 is displayed, guidance for the position of the avatar image 82 may be provided by displaying the speech bubble screen 83 shown in FIG. 7I or the like. Alternatively, the display mode such as the color, shape, and display orientation of the avatar image 82 as a marker may be changed. Thereby, guidance can be appropriately executed according to the reaction of the occupant, and the line of sight 50 of the occupant with a slow reaction to the past reaction history can be quickly directed to the landmark.

[0088] This guidance may be canceled, for example, when the landmark on which the avatar image 82 is displayed is a landmark on the same route as the route guidance performed in the past at a frequency equal to or higher than a predetermined frequency. Execution or cancellation of the guidance according to the frequency of the past route guidance can be determined, for example, based on the log information regarding the past performance of the route guidance accumulated in the travel / visual recognition data storage unit 19.

[0089] In the information notification device 10 of the second and third embodiments, all of the above operations can be performed as operations of the first control for guiding the passenger's line of sight 50 to the landmark. While not guiding the landmark, for example, the second control performed by the information notification device 10 of the first embodiment may be performed.

[0090] In the information notification device 10 of the second and third embodiments, the line of sight 50 of the passenger can be guided to a landmark that the passenger does not notice and is not visually recognized by displaying the avatar image 82 as a marker. When performing the second control performed by the information notification device 10 of the first embodiment while not guiding the landmark, for example, the marker (avatar image 82) on the projection surface may be moved from the display position in the second control to the position of the landmark.

[0091] Note that the above-described embodiments are examples of the present invention. Therefore, the present invention is not limited to the above-described embodiments, and various modifications can be made according to the design and the like without departing from the technical idea of the present invention as long as they are within the scope not deviating from the technical idea of the present invention.

Explanation of Reference Numerals

[0092] 10 Information notification device 11 Object detection unit 12 Line-of-sight direction detection unit 13 Visual recognition determination unit 14 Corresponding position calculation unit 15 Marker display unit 16 Switch detection unit 17 Switch notification unit 18 Attention detection unit 41, 43 Marker 82 Avatar image (marker)

Claims

1. Detect the line-of-sight direction of the vehicle occupant, detect the object of attention that the occupant should pay attention to, When, in a state where the object of attention is detected, the detected line-of-sight direction does not move toward the detected object of attention, perform first control to display a marker at a target position on the line of sight with respect to the detected object of attention on a display unit arranged within the field of view of the occupant, In the first control, the marker moves from a position different from the target position to the target position, In a state where the object of attention is not detected, perform second control to display the marker with a predetermined delay amount at a corresponding position in the detected line-of-sight direction of the display unit, When the object of attention is detected, switch from the second control to the first control, and the marker moves from the display position in the second control to the target position, An information notification method.

2. The information notification method according to claim 1, wherein during the movement of the marker from the display position in the second control to the target position, the moving direction of the marker changes continuously.

3. The information notification method according to claim 1 or 2, wherein in the first control, the marker moves from the display position in the second control to the target position through a path of a curved trajectory on a two-dimensional display surface of the display unit.

4. The information notification method according to any one of claims 1 to 3, wherein in the first control, the marker moves from the display position in the second control to the target position through a path passing through a preset area of the display unit.

5. The information notification method according to any one of claims 1 to 4, wherein the display mode of the marker is different between the first control and the second control.

6. The information notification method according to any one of claims 1 to 5, wherein the occupant is notified of the switch from the second control to the first control.

7. The information notification method according to claim 6, wherein when the detected line-of-sight direction does not move toward the detected object of attention until a preset time has elapsed since the detection of the object of attention started, switch from the second control to the first control. The information notification method according to claim 1, wherein when detecting a landmark in the route guidance of the vehicle as the target of attention, in the first control, as the marker to be displayed at the target position, an avatar image simulating a device for notifying the occupant during the route guidance is displayed at a position on the line of sight with respect to the detected landmark on the display unit.

9. Detect the line-of-sight direction of the occupant of the vehicle, detect the target of attention that the occupant should pay attention to, when, in the state where the target of attention is detected, the detected line-of-sight direction does not move toward the detected target of attention, perform first control to display a marker at the target position on the line of sight with respect to the detected target of attention on the display unit arranged within the field of view of the occupant, when detecting a landmark in the route guidance of the vehicle as the target of attention, in the first control, as the marker to be displayed at the target position, an avatar image simulating a device for notifying the occupant during the route guidance is displayed at a position on the line of sight with respect to the detected landmark on the display unit, when the detected line-of-sight direction does not move toward the detected landmark regarding the notification performed by the device by voice, perform the first control, information notification method.

10. The information notification method according to claim 9, wherein in the first control, the marker moves from a position different from the target position to the target position.

11. The information notification method according to claim 9 or 10, wherein after performing the first control, when the detected line-of-sight direction becomes the same as the direction of the detected landmark, the avatar image is displayed at the reference position on the display unit.

12. The information notification method according to any one of claims 9 to 11, wherein in the first control, when the reaction of the occupant in which the detected line-of-sight direction changes to the direction of the detected landmark due to the display of the avatar image is slow with respect to the history of the past reactions, the mode of the avatar image is changed.

13. The information notification method according to any one of claims 9 to 12, wherein during the display of the avatar image, guidance is provided for the position of the avatar image on the display unit.

14. The information notification method according to claim 13, wherein when the landmark corresponding to the position of the avatar image on the display unit is a landmark on the same route as the route guidance performed in the past at a frequency equal to or higher than a predetermined frequency, the execution of the guidance is canceled.

15. A line-of-sight direction detection unit that detects the line-of-sight direction of a vehicle occupant, a target detection unit that detects a target to be focused on by the occupant, and a marker display unit that performs first control to display a marker at a target position on the line of sight with respect to the detected target of a display unit arranged within the field of view of the occupant when it is determined based on the detected line-of-sight direction that the detected line-of-sight direction is not moving toward the detected target. The marker display unit in the first control, moves the marker from a position different from the target position to the target position. In a state where the target is not detected, performs second control to display the marker with a predetermined delay amount at a corresponding position in the detected line-of-sight direction of the display unit. When the target is detected, switches from the second control to the first control, and moves the marker from the display position in the second control to the target position. An information notification device.

16. A line-of-sight direction detection unit that detects the line-of-sight direction of a vehicle occupant, a target detection unit that detects a target to be focused on by the occupant, and a marker display unit that performs first control to display a marker at a target position on the line of sight with respect to the detected target of a display unit arranged within the field of view of the occupant when it is determined based on the detected line-of-sight direction that the detected line-of-sight direction is not moving toward the detected target. The marker display unit when detecting a landmark in the vehicle route guidance as the target, in the first control, as the marker to be displayed at the target position, displays an avatar image simulating a device that notifies the occupant during the route guidance at a position on the line of sight with respect to the detected landmark of the display unit. Performs the first control when the detected line-of-sight direction is not moving toward the detected landmark regarding the notification performed by the device by voice. An information notification device.

Images