Information display device, information display method, and information display program

Abstract

To provide an information display device that can provide appropriate driving assistance to the driver of a mobile vehicle. [Solution] The information display device 1 includes an information acquisition unit that acquires attention point information including information about a point of interest P where the driver's gaze is concentrated and the degree to which the gaze is concentrated on the point of interest P; a display unit 16 provided on the mobile body 3 that highlights the point of interest P; and a display control unit provided on the mobile body 3 that controls the display unit 16. The display control unit causes the display unit 16 to highlight the point of interest P on the condition that the degree to which the gaze is concentrated on the point of interest P is above a predetermined reference value, and changes the manner of highlighting the point of interest P according to the degree to which the gaze is concentrated on the point of interest P.

Description

Technical Field

[0001] The present invention relates to an information display device, an information display method, and an information display program.

Background Art

[0002] In recent years, efforts have been actively made to provide a sustainable transportation system that takes into account people in vulnerable positions among traffic participants. In order to further improve traffic safety and convenience towards this realization, research and development on driving support technologies have been carried out.

[0003] For example, Patent Document 1 discloses a technology for displaying driving support information (information indicating a section where the detection accuracy of biometric information decreases) on a display unit of a moving body as a driving support technology for a driver of the moving body.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] When driving support information is displayed on the display unit of a moving body in this way, if a large amount of driving support information including unnecessary driving support information is randomly displayed, there is a risk that the driver of the moving body will feel annoyed.

[0006] In view of the above background, an aspect of the present invention aims to provide an information display device, an information display method, and an information display program that can appropriately provide driving support to the driver of a moving body while suppressing the driver of the moving body from feeling annoyed. Thereby, an aspect of the present invention aims to contribute to the development of a sustainable transportation system.

Means for Solving the Problems

[0007] To solve the above problems, one aspect of the present invention is an information display device comprising: an information acquisition unit that acquires point of interest information including information relating to a point of interest where the driver's gaze is concentrated and the degree to which the gaze is concentrated on the point of interest; a display unit provided on a moving body that highlights the point of interest; and a display control unit provided on the moving body that controls the display unit, wherein the display control unit causes the display unit to highlight the point of interest on the display unit on the condition that the degree to which the gaze is concentrated on the point of interest is above a predetermined reference value, and changes the manner of highlighting the point of interest according to the degree to which the gaze is concentrated on the point of interest.

[0008] Another aspect of the present invention is an information display method in which a computer acquires point of interest information including information on points of interest where the driver's gaze is concentrated and the degree to which the gaze is concentrated on the points of interest, and on the condition that the degree to which the gaze is concentrated on the points of interest is above a predetermined standard value, the computer highlights the points of interest and changes the manner of highlighting the points of interest according to the degree to which the gaze is concentrated on the points of interest.

[0009] Another aspect of the present invention is an information display program that causes a computer to acquire point of interest information including information about points of interest where the driver's gaze is concentrated and the degree to which the gaze is concentrated on the points of interest, and on the condition that the degree to which the gaze is concentrated on the points of interest is above a predetermined standard value, the program highlights the points of interest, and changes the manner of highlighting the points of interest according to the degree to which the gaze is concentrated on the points of interest. [Effects of the Invention]

[0010] According to the above embodiments, it is possible to provide an information display device, an information display method, and an information display program that can provide appropriate driving assistance to the driver of a mobile vehicle while suppressing any inconvenience the driver of the mobile vehicle may feel. [Brief explanation of the drawing]

[0011] [Figure 1] Block diagram showing an information display device according to an embodiment. [Figure 2] Sequence diagram showing the processing flow of the information display device according to the embodiment. [Figure 3] Flowchart of the data generation process executed by the data generation unit according to the embodiment. [Figure 4] An explanatory diagram showing a method for estimating the position of the driver's line of sight according to an embodiment. [Figure 5] Flowchart of the information generation process executed by the information generation unit according to the embodiment [Figure 6] Explanatory diagram illustrating the movement path data in the point of interest data according to the embodiment: (A) When both the departure point and the arrival point are within the reference range; (B) When either the departure point or the arrival point is within the reference range; (C) When both the departure point and the arrival point are outside the reference range. [Figure 7] Table showing points of interest according to the embodiment [Figure 8] Diagram illustrating the front window during the highlighting process according to the embodiment. [Figure 9] An explanatory diagram illustrating a situation that highlights the points of interest according to the embodiment. [Figure 10] An explanatory diagram illustrating a situation that highlights the points of interest according to the embodiment. [Figure 11] An explanatory diagram illustrating a situation that highlights the points of interest according to the embodiment. [Modes for carrying out the invention]

[0012] The following describes embodiments of the information display device, information display method, and information display program with reference to the drawings.

[0013] <Information display device 1> Referring to FIG. 1, the information display device 1 includes a first vehicle 3, a plurality of second vehicles 4 (only one is shown in FIG. 1), and an external server 5 provided outside the first vehicle 3 and the plurality of second vehicles 4. The first vehicle 3 is an example of a moving body. The plurality of second vehicles 4 are examples of a plurality of second moving bodies. The first vehicle 3, the plurality of second vehicles 4, and the external server 5 are connected to each other via a wireless communication network N. Hereinafter, the components of the information display device 1 will be described in order.

[0014] <The first vehicle 3> The first vehicle 3 is, for example, a four-wheel automobile. The first vehicle 3 includes a first driving operator 11, a first navigation device 12, a first external sensor 13, a first driver monitor camera 14 (hereinafter abbreviated as "first DMC 14"), a first human machine interface 15 (hereinafter abbreviated as "first HMI 15"), a first head-up display 16 (hereinafter abbreviated as "first HUD 16"), and a first control device 17. The first external sensor 13 is an example of an external acquisition unit. The first HMI 15 is an example of a notification unit. The first HUD 16 is an example of a display unit.

[0015] The first driving operator 11 is an operator that receives a driving operation by the driver of the first vehicle 3. The first driving operator 11 includes a steering operator (e.g., a steering wheel) that receives a steering operation by the driver of the first vehicle 3, an acceleration operator (e.g., an accelerator pedal) that receives an acceleration operation by the driver of the first vehicle 3, a deceleration operator (e.g., a brake pedal) that receives a deceleration operation by the driver of the first vehicle 3, and a shift operator (e.g., a shift lever or a shift button) that receives a shift range switching operation by the driver of the first vehicle 3.

[0016] The first navigation device 12 is a device that provides route guidance to the destination of the first vehicle 3 and the like. The first navigation device 12 stores map information. This map information is, for example, three-dimensional map information including information regarding latitude, longitude, and altitude. In other embodiments, the above map information may be two-dimensional map information including only information regarding latitude and longitude. The above map information includes the surrounding map of the first vehicle 3. The first navigation device 12 identifies the current position of the first vehicle 3 based on GNSS signals (e.g., GPS signals) received from artificial satellites.

[0017] The first external sensor 13 is a sensor that acquires external information of the first vehicle 3. For example, the first external sensor 13 includes a plurality of external cameras, a plurality of radars, and a plurality of lidars (LiDARs). The plurality of external cameras acquire peripheral images of the first vehicle 3 by photographing the surroundings of the first vehicle 3. The plurality of radars emit radio waves such as millimeter waves around the first vehicle 3 and detect the positions of targets existing around the first vehicle 3 by capturing the reflected waves. The plurality of lidars irradiate light such as infrared rays around the first vehicle 3 and detect the positions of targets existing around the first vehicle 3 by capturing the reflected light.

[0018] The first DMC 14 is a device that acquires an image of the face of the driver of the first vehicle 3 by photographing the face of the driver of the first vehicle 3. The first DMC 14 is provided inside the first vehicle 3 (e.g., in front of the driver's seat).

[0019] The first HMI 15 is a device that performs various notifications to the driver of the first vehicle 3 and accepts input of various information by the driver of the first vehicle 3. The first HMI 15 includes a touch panel 19 having a display screen and a speaker 20 that generates sound. The touch panel 19 is provided so as to be able to display the peripheral image of the first vehicle 3 acquired by the plurality of external cameras and the peripheral map of the first vehicle 3 included in the map information of the first navigation device 12.

[0020] The first HUD 16 is a device that displays target information superimposed on the view in front of the driver of the first vehicle 3. The first HUD 16 includes a front window 21 that separates the interior space of the vehicle from the exterior space in front of the first vehicle 3, and a projection device 22 that projects target information onto a predetermined portion of the front window 21. The front window 21 is made of a transparent material. Therefore, the driver of the first vehicle 3 can see the exterior space in front of the first vehicle 3 through the front window 21.

[0021] The first control unit 17 is a computer having a first processor 24 and a first memory 25 that is communicatively connected to the first processor 24. The first processor 24 may include, for example, at least one of a CPU, GPU, or RISC as a core. The first memory 25 stores programs executed by the first processor 24 and various data. The first memory 25 may include at least one of volatile memory and non-volatile memory. The volatile memory may be, for example, DRAM or SRAM. The non-volatile memory may be an SSD, flash memory, magnetic disk storage device, or optical disk storage device. At least a part of the first control unit 17 may be implemented by hardware such as an LSI, ASIC, FPGA, or by a combination of software and hardware. The first control unit 17 may be composed of a single piece of hardware or by multiple pieces of hardware that can communicate with each other.

[0022] The first control device 17 includes, as a functional unit, an information acquisition unit 27, a display control unit 28, and a notification control unit 29. The first processor 24 functions as the information acquisition unit 27, the display control unit 28, and the notification control unit 29 by executing a program stored in the first memory 25. By the first processor 24 executing the program, the first control device 17 performs a part of the information display method.

[0023] <Multiple second vehicles 4> Each second vehicle 4 (hereinafter simply referred to as "second vehicle 4") is, for example, a four-wheeled automobile. The second vehicle 4 includes a second driver control unit 31, a second navigation device 32, a second external sensor 33, a second driver monitor camera 34 (hereinafter abbreviated as "second DMC 34"), a second human-machine interface 35 (hereinafter abbreviated as "second HMI 35"), a second head-up display 36 (hereinafter abbreviated as "second HUD 36"), and a second control device 37. The second DMC 34 is an example of an image acquisition unit. The configurations of the second driver control unit 31, the second navigation device 32, the second external sensor 33, the second DMC 34, the second HMI 35, and the second HUD 36 are the same as those of the first driver control unit 11, the first navigation device 12, the first external sensor 13, the first DMC 14, the first HMI 15, and the first HUD 16 of the first vehicle 3, respectively, so their explanation is omitted.

[0024] The second control unit 37 is a computer having a second processor 39 and a second memory 40 that is communicatively connected to the second processor 39. The physical configuration of the second processor 39 and the second memory 40 is the same as that of the first processor 24 and the first memory 25 of the first control unit 17, so a description is omitted.

[0025] The second control device 37 includes a data generation unit 42 and a data transmission unit 43 as functional units. The second processor 39 functions as the data generation unit 42 and the data transmission unit 43 by executing a program stored in the second memory 40. By the second processor 39 executing the program, the second control device 37 performs a part of the information display method.

[0026] <External Server 5> The external server 5 is a computer having a processor 45 and a memory 46 that is communicatively connected to the processor 45. The physical configuration of the processor 45 and the memory 46 is the same as the physical configuration of the first processor 24 and the first memory 25 of the first control unit 17, so a description is omitted.

[0027] The external server 5 has, as functional units, a data acquisition unit 48, a data storage unit 49, an information generation unit 50, and an information transmission unit 51. The processor 45 functions as the data acquisition unit 48, the information generation unit 50, and the information transmission unit 51 by executing a program stored in the memory 46. The memory 46 functions as the data storage unit 49 when the processor 45 executes a program. In addition, the external server 5 executes a part of the information display method when the processor 45 executes a program.

[0028] <Processing flow of information display device 1> Next, we will explain the processing flow of the information display device 1 configured as described above.

[0029] Referring to Figures 1 and 2, the data generation unit 42 of the second vehicle 4 generates point of interest data by executing a data generation process (details of which will be described later). The point of interest data is data relating to point P (hereinafter referred to as "point of interest P") where the driver's gaze of the second vehicle 4 is concentrated. The data transmission unit 43 of the second vehicle 4 transmits the point of interest data generated by the data generation unit 42 to the data acquisition unit 48 of the external server 5.

[0030] The data acquisition unit 48 of the external server 5 receives point-of-interest data from the data transmission unit 43 of the second vehicle 4. The data storage unit 49 of the external server 5 stores the point-of-interest data received by the data acquisition unit 48. As a result, point-of-interest data is accumulated in the data storage unit 49.

[0031] The information generation unit 50 of the external server 5 generates point of interest information by performing information generation processing (details described later) based on the point of interest data stored in the data storage unit 49. The point of interest information includes information about the point of interest P and the degree to which the driver of the second vehicle 4 focuses their gaze on the point of interest P (hereinafter referred to as "gaze concentration"). The information transmission unit 51 of the external server 5 transmits the point of interest information generated by the information generation unit 50 to the information acquisition unit 27 of the first vehicle 3.

[0032] The information acquisition unit 27 of the first vehicle 3 receives point of interest information from the information transmission unit 51 of the external server 5. The frequency of receiving point of interest information may be set based on the section of track the first vehicle 3 travels, the distance the first vehicle 3 travels, etc. The information acquisition unit 27 may receive point of interest information periodically while the first vehicle 3 is running, or it may receive point of interest information when the first vehicle 3 is started up. Whether or not to receive point of interest information may be set by the occupants of the first vehicle 3.

[0033] The display control unit 28 of the first vehicle 3 controls the first HUD 16 of the first vehicle 3 based on the point of interest information received by the information acquisition unit 27. More specifically, the display control unit 28 performs highlighting processing (details described later) based on the point of interest information received by the information acquisition unit 27, thereby highlighting the point of interest P on the front window 21 of the first HUD 16.

[0034] The notification control unit 29 of the first vehicle 3 controls the first HMI 15 of the first vehicle 3 based on the point of interest information received by the information acquisition unit 27. More specifically, the notification control unit 29 performs notification processing (details described later) based on the point of interest information received by the information acquisition unit 27, causing the first HMI 15 to provide notifications regarding driving operations and specific targets.

[0035] <Data generation process> Next, the data generation process performed by the data generation unit 42 of the second vehicle 4 will be described. Hereafter, when the term "driver" is used without specifying a vehicle, it refers to the driver of the second vehicle 4.

[0036] Referring to Figure 3, the data generation unit 42 estimates the position of the driver's line of sight based on the image of the driver's face acquired by the second DMC 34 while the second vehicle 4 is in motion (step ST1). The image of the driver's face acquired by the second DMC 34 is an example of data related to the driver's line of sight. Referring to Figure 4, for example, the data generation unit 42 detects the positions of both of the driver's eyes E from the image of the driver's face. Next, the data generation unit 42 generates two straight lines L for both of the driver's eyes E that pass through the center of the pupil E1 and are perpendicular to the tangent plane S of the eyeball E2. The data generation unit 42 estimates the intersection point M of these two straight lines L as the position of the driver's line of sight.

[0037] Referring to Figure 3, the data generation unit 42 then identifies the locations of landmarks (signs, roads, pedestrian crossings, road markings, etc.) present around the second vehicle 4 based on the detection results of the second external sensor 33 (step ST2). In other embodiments, step ST2 may be performed before step ST1, or steps ST1 and ST2 may be performed simultaneously.

[0038] Next, the data generation unit 42 sets a point of interest P based on the driver's line of sight position estimated in step ST1 and the target position identified in step ST2 (step ST3). For example, the data generation unit 42 sets a target as the point of interest P if the driver's line of sight position overlaps with the same target position for a predetermined time or longer. Alternatively, the data generation unit 42 sets a target as the point of interest P if the driver's line of sight position overlaps with the same target position for a predetermined time or longer in total.

[0039] Next, the data generation unit 42 generates point of interest data (step ST4) based on the point of interest P set in step ST3, the current position of the second vehicle 4 identified by the second navigation device 32, and the map information stored by the second navigation device 32. The point of interest data includes location data, acquisition time data, travel route data, and target data. The location data is data relating to the position of point of interest P on the map in the map information. The acquisition time data is data relating to the time when the second DMC 34 acquired the image of the driver's face. The travel route data is data relating to the travel route R of the second vehicle 4, and includes data relating to the departure point R1 and arrival point R2 of the second vehicle 4. The target data is data relating to the target that overlaps with point of interest P. If the second driver control 31 receives a driving operation within a predetermined range from point of interest P, the point of interest data includes driving operation data. The driving operation data is data relating to the driving operation received by the second driver control 31 within a predetermined range from point of interest P.

[0040] <Information generation process> Next, we will describe the information generation process performed by the information generation unit 50 of the external server 5.

[0041] Referring to Figure 5, the information generation unit 50 generates basic data based on the point of interest data stored in the data storage unit 49 (step ST11). The basic data is the data that forms the basis of the point of interest information. The method for generating the basic data will be described below.

[0042] The information generation unit 50 refers to the acquisition time data of the point of interest data and determines whether the time at which the second DMC 34 acquired the driver's face image (hereinafter referred to as "image acquisition time") is after a predetermined reference time. If the image acquisition time is after the reference time, the information generation unit 50 includes the point of interest data in the basic data. On the other hand, if the image acquisition time is before the reference time, the information generation unit 50 does not include the point of interest data in the basic data. In other words, the information generation unit 50 generates point of interest information based on the driver's face image acquired after the reference time (an example of data related to the driver's gaze), and does not generate point of interest information based on the driver's face image acquired before the reference time. The touch panel 19 of the first HMI 15 may be provided to display a setting screen for the occupant of the first vehicle 3 to set a reference time (for example, one week before, one month before, three months before, etc., the current time). In other embodiments, a reference time does not need to be set.

[0043] Referring to Figure 6, the information generation unit 50 sets a reference range B centered on the point of interest P (for example, a range with a radius of 5 km from the point of interest P). The information generation unit 50 also refers to the movement path data of the point of interest data and determines whether at least one of the starting point R1 and the arrival point R2 of the movement path R of the second vehicle 4 is within the reference range B. If at least one of the starting point R1 and the arrival point R2 of the movement path R of the second vehicle 4 is within the reference range B (see Figures 6(A) and (B)), the information generation unit 50 includes the point of interest data in the basic data. On the other hand, if both the starting point R1 and the arrival point R2 of the movement path R of the second vehicle 4 are outside the reference range B (see Figure 6(C)), the information generation unit 50 does not include the point of interest data in the basic data. In other words, the information generation unit 50 generates point of interest information based on point of interest data from the second vehicle 4 where at least one of the departure point R1 and arrival point R2 is within the reference range B, but does not generate point of interest information based on point of interest data from the second vehicle 4 where both the departure point R1 and arrival point R2 are outside the reference range B.

[0044] Referring to Figure 5, the information generation unit 50 sets the degree of gaze concentration on point of interest P based on the basic data generated in step ST11 (step ST12). The method for setting the degree of gaze concentration on point of interest P will be described below.

[0045] The information generation unit 50 calculates the standard score (deviation from the mean) of the number of points of interest data for each point of interest P in the basic data. If the standard score of the number of points of interest data for a certain point of interest P is greater than or equal to the first threshold, the information generation unit 50 sets the degree of gaze concentration for that point of interest P to "high". If the standard score of the number of points of interest data for a certain point of interest P is greater than or equal to the second threshold but less than the first threshold (however, the second threshold < the first threshold), the information generation unit 50 sets the degree of gaze concentration for that point of interest P to "medium". If the standard score of the number of points of interest data for a certain point of interest P is less than the second threshold, the information generation unit 50 sets the degree of gaze concentration for that point of interest P to "low". In this way, the information generation unit 50 sets the degree of gaze concentration for a point of interest P by analyzing the point of interest data. In other embodiments, the information generation unit 50 may set the degree of gaze concentration for a point of interest P to two or four or more levels. In other embodiments, the information generation unit 50 may set the degree of gaze concentration on point of interest P based on the number of point of interest data itself, rather than the standard score of the number of point of interest data.

[0046] The information generation unit 50 generates point of interest information (step ST13) based on the basic data generated in step ST11 and the degree of gaze concentration set in step ST12. Referring to Figure 7, the point of interest information includes, for each point of interest P, the location of point of interest P on the map, the object that overlaps with point of interest P, the driving operation received by the second driving control 31 within a predetermined range from point of interest P, and the degree of gaze concentration towards point of interest P. The location of point of interest P on the map is, for example, a three-dimensional location including latitude Xn, longitude Yn, and altitude Zn (n=1, 2, 3...). In other embodiments, the location of point of interest P on the map may be a two-dimensional location including only latitude Xn and longitude Yn.

[0047] <Highlighting> Next, the highlighting process performed by the display control unit 28 of the first vehicle 3 will be described. Hereinafter, the roads surrounding the first vehicle 3 will be referred to as "surrounding road C". Hereinafter, when simply referred to as "front window 21", it refers to the front window 21 of the first HUD 16.

[0048] Referring to Figure 8, the display control unit 28 recognizes the shape of the surrounding road C on the front windshield 21 based on the external information of the first vehicle 3 acquired by the first external sensor 13. In this embodiment, the surrounding road C includes both the roadway and the sidewalk. In other embodiments, the surrounding road C may include only the roadway.

[0049] The display control unit 28 identifies the left and right edges C1 and the upper edge C2 of the surrounding road C in the front window 21 based on the shape of the surrounding road C in the front window 21. If the left and right edges C1 or the upper edge C2 of the surrounding road C in the front window 21 cannot be identified (for example, if there are many obstacles that obscure the left and right edges C1 or the upper edge C2 of the surrounding road C in the front window 21), the display control unit 28 may estimate the left and right edges C1 or the upper edge C2 of the surrounding road C in the front window 21 based on the external information of the first vehicle 3 acquired by the first external sensor 13.

[0050] The display control unit 28 sets the left and right boundary lines D1 (hereinafter simply referred to as "left and right boundary lines D1") on the front window 21 based on the left and right edges C1 of the surrounding road C on the front window 21. For example, the display control unit 28 sets the left and right boundary lines D1 at a position offset by a predetermined width outward in the left and right direction from the left and right edges C1 of the surrounding road C on the front window 21. The display control unit 28 sets the upper boundary line D2 (hereinafter simply referred to as "upper boundary line D2") on the front window 21 based on the upper edge C2 of the surrounding road C on the front window 21. For example, the display control unit 28 sets the upper boundary line D2 at a position offset by a predetermined height upward from the upper edge C2 of the surrounding road C on the front window 21.

[0051] The display control unit 28 identifies the position of the point of interest P on the front windshield 21 based on the map information stored in the first navigation device 12, the current position of the first vehicle 3 identified by the first navigation device 12, and the map position of the point of interest P included in the point of interest information. If a point of interest P (see P1 and P2 in Figure 8) is located inward in the left-right direction from the left-right boundary line D1 and below the upper boundary line D2, the display control unit 28 displays an enhanced image Q (see Q1 and Q2 in Figure 8) that emphasizes that point of interest P on the front windshield 21. The enhanced image Q is, for example, an image that overlaps with the corresponding point of interest P. On the other hand, if a point of interest P (see P3 in Figure 8) is located outward in the left-right direction from the left-right boundary line D1, or if a point of interest P (see P4 in Figure 8) is located above the upper boundary line D2, the display control unit 28 does not display an enhanced image Q that emphasizes that point of interest P on the front windshield 21. In other words, the display control unit 28 highlights a point of interest P on the front window 21, provided that the point of interest P is located inward in the left-right boundary line D1 and below the upper boundary line D2.

[0052] The display control unit 28 acquires the degree of gaze concentration for each point of interest P based on the point of interest information. If the degree of gaze concentration for a certain point of interest P is equal to or greater than a first reference value (the degree of gaze concentration is "high" or "medium"), the display control unit 28 displays an enhanced image Q that highlights that point of interest P on the front window 21. On the other hand, if the degree of gaze concentration for a certain point of interest P is less than the first reference value (the degree of gaze concentration is "low"), the display control unit 28 does not display an enhanced image Q that highlights that point of interest P on the front window 21. In other words, the display control unit 28 highlights a point of interest P on the front window 21 only if the degree of gaze concentration for that point of interest P is equal to or greater than a first reference value.

[0053] The display control unit 28 changes the manner in which the point of interest P is highlighted according to the degree of gaze concentration on the point of interest P. For example, the display control unit 28 makes the density of the highlighting image Q (see Q1 in Figure 8) that highlights a point of interest P whose degree of gaze concentration is greater than or equal to a second reference value (however, second reference value > first reference value) (point of interest P with a "high" degree of gaze concentration: see P1 in Figure 8) darker than the density of the highlighting image Q (see Q2 in Figure 8) that highlights a point of interest P whose degree of gaze concentration is less than the second reference value (point of interest P with a "medium" degree of gaze concentration: see P2 in Figure 8). Alternatively, the display control unit 28 displays the highlighting image Q that highlights a point of interest P whose degree of gaze concentration is greater than or equal to the second reference value in a more conspicuous color than the highlighting image Q that highlights a point of interest P whose degree of gaze concentration is less than the second reference value.

[0054] The display control unit 28 calculates the distance from the first vehicle 3 to a point of interest P based on the current position of the first vehicle 3 identified by the first navigation device 12 and the map position of the point of interest P included in the point of interest information. When the distance from the first vehicle 3 to a point of interest P falls below a predetermined reference distance, the display control unit 28 displays a highlight image Q emphasizing that point of interest P on the front windshield 21. On the other hand, while the distance from the first vehicle 3 to a point of interest P exceeds the reference distance, the display control unit 28 does not display the highlight image Q emphasizing that point of interest P on the front windshield 21. In other words, the display control unit 28 starts highlighting the point of interest P on the front windshield 21 when the distance from the first vehicle 3 to a point of interest P falls below the reference distance. The touch panel 19 of the first HMI 15 may be provided to display a setting screen for the occupant of the first vehicle 3 to set a reference distance (for example, 50m, 100m, 200m, etc.). In other embodiments, a reference distance does not need to be set.

[0055] In the highlighting process described above, the display control unit 28 highlights a point of interest P on the front window 21 regardless of the type of object it overlaps with. In other embodiments, the display control unit 28 may highlight only points of interest P that overlap with specific objects (for example, signs, road markings, curved roads, tunnels, branches, etc.) on the front window 21.

[0056] In the highlighting process described above, the display control unit 28 highlights a point of interest P on the front window 21 regardless of whether the second operating control 31 has received an operation within a predetermined range from that point of interest P. In another embodiment, the display control unit 28 may highlight a point of interest P on the front window 21 only if the second operating control 31 has received an operation within a predetermined range from that point of interest P.

[0057] <Notification Processing> Next, the notification processing performed by the notification control unit 29 of the first vehicle 3 will be described. Hereinafter, driving operations received by the second driving control unit 31 within a predetermined range from point of interest P will be referred to as "driving operations at point of interest P".

[0058] The notification control unit 29 determines whether or not information regarding the operation of a particular point of interest P is included in the point of interest information. If information regarding the operation of a particular point of interest P is included in the point of interest information (i.e., if the second driver control unit 31 receives an operation within a predetermined range from a particular point of interest P), the notification control unit 29 causes the first HMI 15 to notify the driver of the operation of that point of interest P in conjunction with the highlighting of that point of interest P by the front window 21.

[0059] For example, if the information on a point of interest includes information on steering and deceleration operations as part of the information on driving operations at a point of interest P, the notification control unit 29 will either display a message such as "This is a curve, please slow down" on the touch panel 19 of the first HMI 15, or generate an audio message such as "This is a curve, please slow down" from the speaker 20 of the first HMI 15. If the speed of the first vehicle 3 is faster than a predetermined reference speed, the first control device 17 may automatically decelerate the first vehicle 3. For example, if the information on a point of interest includes information on turning on the headlights as part of the information on driving operations at a point of interest P, the notification control unit 29 will either display a message such as "Please turn on the headlights" on the touch panel 19 of the first HMI 15, or generate an audio message such as "Please turn on the headlights" from the speaker 20 of the first HMI 15.

[0060] The notification control unit 29 determines, based on the point of interest information, whether a certain point of interest P overlaps with a specific target. If a point of interest P overlaps with a specific target, the notification control unit 29 causes the first HMI 15 to issue a notification regarding the specific target, in conjunction with the highlighting of that point of interest P by the front window 21.

[0061] For example, if a point of interest P coincides with a pedestrian crossing, the notification control unit 29 will either display a message such as "This is a pedestrian crossing, please be careful of pedestrians" on the touch panel 19 of the first HMI 15, or emit an audio message such as "This is a pedestrian crossing, please be careful of pedestrians" from the speaker 20 of the first HMI 15. For example, if a point of interest P coincides with a sharp curve in the road, the notification control unit 29 will either display a message such as "This is a curve, please slow down" on the touch panel 19 of the first HMI 15, or emit an audio message such as "This is a curve, please slow down" from the speaker 20 of the first HMI 15.

[0062] <Effects> Figures 9 to 11 illustrate situations in which the front window 21 highlights point of interest P.

[0063] In Figure 9, part of the sign and traffic light in front of the first vehicle 3 is hidden by a tree, so the driver of the first vehicle 3 cannot see the entire sign and traffic light. Even in this situation, if the sign and traffic light are set as point of interest P, an enhanced image Q highlighting the sign and traffic light is displayed on the front windshield 21. In other words, the sign and traffic light are highlighted on the front windshield 21. Therefore, the driver of the first vehicle 3 can easily recognize the presence of the sign and traffic light.

[0064] In Figure 10, the structure of the surrounding road C is complex, making it difficult for the driver of the first vehicle 3 to grasp its structure. Even in this situation, if a point indicating the structure of the surrounding road C (for example, the boundary point between the surrounding road C and other roads) is set as point of interest P, an enhanced image Q highlighting the point indicating the structure of the surrounding road C will be displayed on the front windshield 21. In other words, the point indicating the structure of the surrounding road C will be highlighted on the front windshield 21. Therefore, the driver of the first vehicle 3 can easily grasp the structure of the surrounding road C.

[0065] In Figure 11, there is a small alleyway connecting to the surrounding road C, from which cars frequently emerge. However, because this alleyway is hidden behind a building, it is difficult for the driver of the first vehicle 3 to spot it. Even in this situation, if the entrance to this alleyway is set as point of interest P, an enhanced image Q highlighting the entrance to this alleyway will be displayed on the front windshield 21. In other words, the entrance to this alleyway will be highlighted on the front windshield 21. Therefore, the driver of the first vehicle 3 can easily recognize the existence of this alleyway.

[0066] In this way, by highlighting points of interest P on the front windshield 21 based on point of interest information collected from multiple second vehicles 4, the driving know-how of the drivers of multiple second vehicles 4 can be provided to the driver of the first vehicle 3. Therefore, when the driver of the first vehicle 3 is driving on unfamiliar roads or roads with poor visibility, this can give the driver of the first vehicle 3 a sense of security.

[0067] Incidentally, the display control unit 28 can also highlight all points of interest P that fit within the frame of the front windshield 21 on the front windshield 21. However, if the display control unit 28 indiscriminately highlights points of interest P on the front windshield 21, the driver of the first vehicle 3 may find it annoying. Therefore, the display control unit 28 highlights a point of interest P on the front windshield 21 only if the degree of gaze concentration on that point of interest P is equal to or greater than a first reference value. By narrowing the target of highlighting to points of interest P whose degree of gaze concentration is equal to or greater than the first reference value in this way, it is possible to provide appropriate driving assistance to the driver of the first vehicle 3 while suppressing the possibility of the driver of the first vehicle 3 feeling annoyed by the highlighting of points of interest P.

[0068] Furthermore, among the drivers of the multiple second vehicles 4, some drivers have driven the travel route R many times, while others are driving the travel route R for the first time. Also, among the drivers of the multiple second vehicles 4, some are veteran drivers who have held their driver's licenses for many years, while others are novice drivers who have only recently obtained their licenses. Moreover, among the drivers of the multiple second vehicles 4, some drive the second vehicle 4 on a daily basis, while others rarely drive the second vehicle 4 (so-called paper drivers). Thus, the drivers of the multiple second vehicles 4 include drivers with a variety of characteristics. As a result, the point of interest data transmitted from the multiple second vehicles 4 to the external server 5 contains a considerable amount of noise corresponding to the characteristics of the drivers. Therefore, the information generation unit 50 generates point of interest information based only on some of the point of interest data (point of interest data transmitted from second vehicles 4 where at least one of the departure point R1 and arrival point R2 is within the reference range B, and generated after the reference time). By narrowing down the data of points of interest that form the basis for generating point-of-interest information in this way, the influence of noise contained in the data can be reduced, thus enabling the generation of highly accurate point-of-interest information.

[0069] <Other variations> In the above embodiment, the first HUD 16 is an example of a display unit. In other embodiments, the touch panel 19 of the first HMI 15 may be an example of a display unit. In this case, the display control unit 28 may highlight points of interest P on the surrounding image or surrounding map of the first vehicle 3 displayed on the touch panel 19 of the first HMI 15.

[0070] In the above embodiment, the emphasis image Q that highlights the point of interest P is an image that overlaps with the point of interest P (see P1 and P2 in Figure 8). In other embodiments, the emphasis image Q may be an image other than an image that overlaps with the point of interest P (for example, an image that surrounds the point of interest P or an arrow image that indicates the point of interest P).

[0071] In the above embodiment, the point of interest P is highlighted by the front window 21 displaying the highlighting image Q. In other embodiments, the point of interest P may be highlighted by methods other than displaying the highlighting image Q. For example, if the touch panel 19 of the first HMI 15 is an example of a display unit, the point of interest P may be highlighted by changing the density, color, etc., of the point of interest P itself in the surrounding image or surrounding map of the first vehicle 3 displayed on the touch panel 19 of the first HMI 15. In other words, the point of interest P may be highlighted by any method.

[0072] In the above embodiment, the data generation unit 42 generates two straight lines L for both of the driver's eyes E that pass through the center of the pupil E1 and are perpendicular to the tangent plane S of the eyeball E2, and estimates the intersection M of these two straight lines L as the position of the driver's line of sight (see Figure 4). In other embodiments, the data generation unit 42 may estimate the position of the driver's line of sight by a method other than that described above. For example, the data generation unit 42 may estimate the position of the driver's line of sight based on the driver's binocular parallax, or it may estimate the position of the driver's line of sight based on the position of the pupils of both of the driver's eyes E. In other words, the data generation unit 42 may estimate the position of the driver's line of sight by any method.

[0073] In the above embodiment, the attention point information transmitted from the external server 5 to the first vehicle 3 includes information about attention points P whose gaze concentration level is less than a first reference value (i.e., attention points P that are not highlighted). In other embodiments, information about attention points P whose gaze concentration level is less than a first reference value may be excluded from the attention point information before the attention point information is transmitted from the external server 5 to the first vehicle 3.

[0074] In the above embodiment, a data generation unit 42 is provided in each second vehicle 4. In other embodiments, the data generation unit 42 may be provided in an external server 5. In this case, the data generation unit 42 of the external server 5 may receive images of the drivers' faces from each second vehicle 4 and generate point-of-interest data by performing the above-described data generation process based on the received images of the drivers' faces.

[0075] In the above embodiment, a second DMC34 (an example of an image acquisition unit) and a data generation unit 42 are provided in each second vehicle 4, an information generation unit 50 is provided in an external server 5, and an information acquisition unit 27, a display control unit 28, and a first HUD16 (an example of a display unit) are provided in the first vehicle 3. In other embodiments, the image acquisition unit, data generation unit, information generation unit, information acquisition unit, display control unit, and display unit may all be provided in the first vehicle 3. In this case, the first DMC14 (an example of an image acquisition unit) acquires an image of the driver's face in the first vehicle 3, the first processor 24 (an example of a data generation unit) generates point of interest data based on the image of the driver's face in the first vehicle 3, the first processor 24 (an example of an information generation unit and an information acquisition unit) generates and acquires point of interest information based on the point of interest data, and the first processor 24 (an example of a display control unit) highlights the point of interest P on the first HUD16 or first HMI15 (an example of a display unit) based on the point of interest information. By adopting this configuration, the entire process, from acquiring the driver's facial image to highlighting the point of interest P, can be performed entirely within the first vehicle 3.

[0076] In the above embodiment, the mobile body 3 and the plurality of second mobile bodies 4 are four-wheeled automobiles. In other embodiments, the mobile body 3 and / or the plurality of second mobile bodies 4 may be vehicles other than four-wheeled automobiles (such as motorcycles), or vehicles other than automobiles (such as ships and aircraft).

[0077] <Summary of Embodiments> One embodiment is an information display device 1 comprising: an information acquisition unit 27 that acquires attention point information including information about a point of interest P where the driver's gaze is concentrated and the degree to which the gaze is concentrated on the point of interest P; a display unit 16 provided on a mobile body 3 that highlights the point of interest P; and a display control unit 28 provided on the mobile body 3 that controls the display unit 16, wherein the display control unit 28 causes the display unit 16 to highlight the point of interest P on the condition that the degree to which the gaze is concentrated on the point of interest P is above a predetermined reference value, and changes the manner of highlighting the point of interest P according to the degree to which the gaze is concentrated on the point of interest P.

[0078] According to this embodiment, by narrowing the focus of the highlighting to points of interest P where the degree of gaze concentration exceeds a standard value, it is possible to provide appropriate driving assistance to the driver of the mobile vehicle 3 while suppressing the driver from feeling annoyed by the highlighting of points of interest P. Furthermore, by changing the manner of highlighting points of interest P according to the degree of gaze concentration on points of interest P, it is possible to encourage the driver of the mobile vehicle 3 to take appropriate action according to the degree of gaze concentration on points of interest P.

[0079] The information display device 1 further comprises an information generation unit 50 that generates the point of interest information, and the information generation unit 50 may generate the point of interest information based on data relating to the driver's gaze acquired after a predetermined reference time.

[0080] The point of focus P, where the driver's gaze is concentrated, may change over time. Therefore, if point of focus information is generated based on outdated data regarding the driver's gaze, the information generated may correspond to a past point of focus P rather than the current point of focus P, potentially leading to the past point of focus P being highlighted. According to the above embodiment, it is possible to suppress the generation of point of focus information based on outdated data regarding the driver's gaze. As a result, point of focus information corresponding to the current point of focus P can be generated, and the current point of focus P can be highlighted.

[0081] The information display device 1 further comprises a plurality of second mobile bodies 4 and an external server 5 provided outside the mobile body 3 and the plurality of second mobile bodies 4. Each second mobile body 4 generates point of interest data relating to the point of interest P and transmits it to the external server 5. The external server 5 analyzes the point of interest data to set the degree of gaze concentration on the point of interest P, generates point of interest information based on the point of interest data and the degree of gaze concentration on the point of interest P, and transmits the point of interest information to the mobile body 3. The display control unit 28 may highlight the point of interest P on the display unit 16 based on the point of interest information.

[0082] In this embodiment, each second mobile unit 4 generates point of interest data, the external server 5 generates point of interest information based on the point of interest data, and the display control unit 28 of the mobile unit 3 causes the display unit 16 of the mobile unit 3 to highlight the point of interest P based on the point of interest information. Therefore, the driving know-how of the drivers of each second mobile unit 4 can be provided to the driver of the mobile unit 3.

[0083] The aforementioned point of interest data includes data relating to the departure point R1 and arrival point R2 of each of the second mobile bodies 4. The external server 5 may set a reference range B centered on the point of interest P and generate the point of interest information based on the point of interest data from the second mobile bodies 4, where at least one of the departure point R1 and arrival point R2 is within the reference range B.

[0084] If both the departure point R1 and arrival point R2 of the second mobile vehicle 4 are not within the reference range B, the driver of that second mobile vehicle 4 is likely to be a driver who does not frequently drive near point of interest P (for example, a driver visiting the area near point of interest P for the first time for a trip, or a driver who does not drive often). According to the above embodiment, point of interest data from the second mobile vehicle 4 driven by such a driver can be excluded, and point of interest information can be generated based on highly reliable point of interest data from the second mobile vehicle 4 driven by a driver who is familiar with driving near point of interest P. Therefore, the reliability of the highlighted point of interest P can be improved.

[0085] The mobile body 3 further includes a notification unit 15 for notifying the occupants of the mobile body 3, and each of the second mobile bodies 4 includes a driving control unit 31 for receiving driving operations from the driver, and when the driving control unit 31 receives a driving operation within a predetermined range from the point of interest P, the notification unit 15 may provide a notification regarding the driving operation in conjunction with the highlighting of the point of interest P by the display unit 16.

[0086] According to this embodiment, the driver of the mobile vehicle 3 can recognize not only the location of the point of interest P, but also the driving operations that should be performed near the point of interest P.

[0087] Each of the second mobile units 4 may include an image acquisition unit 34 that acquires an image of the driver's face, and a data generation unit 42 that estimates the position of the driver's line of sight based on the image of the driver's face and generates the point of interest data based on the position of the driver's line of sight.

[0088] According to this embodiment, the position of the driver's gaze can be accurately estimated based on an image of the driver's face, thereby improving the accuracy of the position of point P in the point of interest data.

[0089] The display control unit 28 may start highlighting the point of interest P on the display unit 16 when the distance from the moving body 3 to the point of interest P becomes less than or equal to a predetermined reference distance.

[0090] According to this embodiment, the highlighting of point of interest P can be started before the moving body 3 passes near the point of interest P, making it easier for the driver of the moving body 3 to notice the highlighting of point of interest P.

[0091] The mobile body 3 further includes an external environment acquisition unit 13 that acquires external environment information of the mobile body 3, and the display control unit 28 identifies the left and right edges C1 of the road surrounding the mobile body 3 based on the external environment information, sets left and right boundary lines D1 based on the left and right edges C1 of the road surrounding the mobile body 3, and, on the condition that the point of interest P is located inward in the left and right direction from the left and right boundary lines D1, the display unit 16 is made to highlight the point of interest P.

[0092] According to this embodiment, it is possible to suppress the highlighting of points of interest P that are significantly shifted outward in the lateral direction from the road surrounding the mobile body 3 (i.e., points of interest P that have little relevance to the movement of the mobile body 3). Therefore, it is possible to more effectively suppress the driver of the mobile body 3 from feeling annoyed.

[0093] The mobile body 3 further includes an external environment acquisition unit 13 that acquires external environment information of the mobile body 3, and the display control unit 28 identifies the upper edge C2 of the road surrounding the mobile body 3 based on the external environment information, sets an upper boundary line D2 based on the upper edge C2 of the road surrounding the mobile body 3, and, on the condition that the point of interest P is below the upper boundary line D2, causes the display unit 16 to highlight the point of interest P.

[0094] According to this embodiment, it is possible to suppress the highlighting of points of interest P that are significantly shifted upward from the road surrounding the mobile body 3 (i.e., points of interest P that have little relevance to the movement of the mobile body 3). Therefore, it is possible to more effectively suppress the driver of the mobile body 3 from feeling annoyed.

[0095] The mobile body 3 further includes a notification unit 15 for notifying the occupants of the mobile body 3, and when the point of interest P overlaps with a specific target, the notification unit 15 may provide notification regarding the specific target in conjunction with the highlighting of the point of interest P by the display unit 16.

[0096] According to this embodiment, the driver of the mobile vehicle 3 can recognize not only the position of the point of interest P, but also specific targets that overlap with the point of interest P.

[0097] Another embodiment is an information display method in which a computer acquires attention point information including information about a point of interest P where the driver's gaze is concentrated and the degree to which the gaze is concentrated on the point of interest P, and, on the condition that the degree to which the gaze is concentrated on the point of interest P is equal to or greater than a predetermined standard value, highlights the point of interest P, and changes the manner of highlighting the point of interest P according to the degree to which the gaze is concentrated on the point of interest P.

[0098] According to this embodiment, it is possible to provide an information display method that can provide appropriate driving assistance to the driver of the mobile unit 3 while suppressing any inconvenience the driver of the mobile unit 3 may feel.

[0099] Another embodiment is an information display program that causes a computer to acquire attention point information including information about a point of interest P where the driver's gaze is concentrated and the degree to which the gaze is concentrated on the point of interest P, and on the condition that the degree to which the gaze is concentrated on the point of interest P is above a predetermined standard value, the program highlights the point of interest P, and changes the manner of highlighting the point of interest P according to the degree to which the gaze is concentrated on the point of interest P.

[0100] According to this embodiment, it is possible to provide an information display program that can provide appropriate driving assistance to the driver of the mobile unit 3 while suppressing any inconvenience the driver of the mobile unit 3 may feel.

[0101] This concludes the description of specific embodiments, but the present invention is not limited to the above embodiments or modifications and can be broadly modified and implemented. [Explanation of symbols]

[0102] 1: Information display device 3: First vehicle (an example of a mobile vehicle) 4: Second vehicle (an example of a second mobile unit) 5: External Server 13: First external sensor (an example of an external environment acquisition unit) 15: First HMI (Example of a notification section) 16: First HUD (Example of a display unit) 27: Information acquisition department 28: Display Control Unit 31: Second operating control unit 34: Second DMC (an example of an image acquisition unit) 42: Data Generation Unit 50: Information generation section B: Reference range R1: Starting point R2: Arrival point C1: Left and right ends C2:Top edge D1: Left and right border D2: Upper boundary line P: Points to note

Claims

1. An information display device, An information acquisition unit that acquires attention point information including information about points of focus where the driver's gaze is concentrated and the degree to which the gaze is concentrated on said points of focus, A display unit is provided on the moving body to highlight the points of interest, The mobile body comprises a display control unit that controls the display unit, The display control unit, Conditional on the degree of focus of the gaze on the aforementioned point of interest being equal to or greater than a predetermined standard value, the display unit will highlight the aforementioned point of interest. An information display device that changes the manner in which a point of interest is highlighted according to the degree of focus of the gaze on that point of interest.

2. The system further includes an information generation unit that generates the aforementioned point of interest information, The information display device according to claim 1, wherein the information generation unit generates the point of interest information based on data relating to the driver's gaze acquired after a predetermined reference time.

3. Multiple second mobile units, The system further comprises the aforementioned mobile body and external servers provided outside the plurality of second mobile bodies, Each of the second mobile units generates point data relating to the point of interest and transmits it to the external server. The aforementioned external server is By analyzing the aforementioned data on points of interest, the degree of focus of gaze on those points of interest is determined. Based on the aforementioned point of interest data and the degree of gaze concentration on the point of interest, the point of interest information is generated. The aforementioned point of interest information is transmitted to the mobile body. The information display device according to claim 1, wherein the display control unit causes the display unit to highlight the points of interest based on the points of interest information.

4. The aforementioned data of points of interest includes data relating to the departure and arrival points of each of the second mobile units. The aforementioned external server is A reference range is set centered on the aforementioned points of interest, The information display device according to claim 3, which generates the point of interest information based on the point of interest data from the second moving body, where at least one of the departure point and arrival point is within the reference range.

5. The mobile body further comprises a notification unit for providing notifications to the occupants of the mobile body, Each of the second mobile units is equipped with a control unit that receives driving operations from the driver, The information display device according to claim 3, wherein when the operating control unit receives the operating operation within a predetermined range from the point of interest, the notification unit provides notification regarding the operating operation in conjunction with the display unit highlighting the point of interest.

6. Each of the aforementioned second mobile bodies is An image acquisition unit that acquires an image of the driver's face, The information display device according to claim 3, further comprising: a data generation unit that estimates the position of the driver's gaze based on an image of the driver's face and generates the point of interest data based on the position of the driver's gaze.

7. The information display device according to any one of claims 1 to 6, wherein the display control unit causes the display unit to start highlighting the point of interest when the distance from the moving body to the point of interest becomes less than or equal to a predetermined reference distance.

8. The mobile body further comprises an external information acquisition unit that acquires external information about the mobile body, The display control unit, Based on the external information, the left and right edges of the road surrounding the moving object are identified. The left and right boundary lines are set based on the left and right edges of the road surrounding the moving body. The information display device according to any one of claims 1 to 6, wherein the point of interest is located inward in the left-right direction from the left-right boundary line, and the display unit is configured to highlight the point of interest.

9. The mobile body further comprises an external information acquisition unit that acquires external information about the mobile body, The display control unit, Based on the external information, the upper edge of the road surrounding the moving object is identified. An upper boundary line is set based on the upper edge of the road surrounding the moving body. An information display device according to any one of claims 1 to 6, wherein the point of interest is located below the upper boundary line, and the display unit is configured to highlight the point of interest.

10. The mobile body further comprises a notification unit for providing notifications to the occupants of the mobile body, The information display device according to any one of claims 1 to 6, wherein if the point of interest coincides with a specific target, the notification unit provides notification regarding the specific target in conjunction with the highlighting of the point of interest by the display unit.

11. Computers We obtain point of focus information that includes information about the points of focus where the driver's gaze is concentrated and the degree to which the gaze is concentrated on those points of focus. Conditional on the degree of focus of gaze on the aforementioned point of interest being above a predetermined threshold, the aforementioned point of interest is highlighted. An information display method that changes the manner in which a point of interest is highlighted according to the degree to which the gaze is focused on that point of interest.

12. On the computer, The system acquires point of focus information, which includes information about the points of focus where the driver's gaze is concentrated and the degree to which the gaze is concentrated on those points of focus. Conditional on the degree of focus of the gaze on the aforementioned point of interest being above a predetermined threshold, the aforementioned point of interest is highlighted. An information display program that changes the manner in which a point of interest is highlighted according to the degree of focus of the gaze on that point of interest.

Images