Information provision device, information provision system, information provision method, and information provision program

Abstract

An information providing device comprising: an intention estimation unit (23) that estimates the intention of a person based on a plurality of images of the person and display information being displayed on a display device at the time the plurality of images are captured; a selection unit (24) that selects the content or method of information to be provided in accordance with the estimated intention; and an output control unit (25) that controls an output device to provide information in accordance with the result of the selection.

Description

[Technical field]

[0001] The present disclosure relates to an information providing technique that is a technique for providing information. [Background technology]

[0002] In places where public transportation services are provided, such as railway stations, bus stops, and airport terminals, or facilities such as department stores and shopping malls (hereinafter, these places or facilities may be collectively referred to simply as "service providing spots"), information providing devices may be installed to provide various information related to the use of the service providing spots. For example, Patent Document 1 discloses a technology relating to "a stationary guidance information display device, characterized in that it comprises a code reading means for reading code information from a visible code, a storage means for storing a group of destination information in which the code information and destination information are associated, a route guidance generating means for searching the group of destination information for a destination associated with the read code information and generating route guidance information to the destination, and a display means for displaying the generated route guidance information in the form of a three-dimensional map" (claim 1 of Patent Document 1), and it is assumed that such a guidance information display device will be installed and used in a station (paragraph 0022 of Patent Document 1). According to Patent Document 1, a user holds a paper on which a visible code such as an AR marker is printed or a mobile terminal displaying the code over a code reader (code reading means), and the code reader reads the visible code and the specified information is displayed (paragraphs 0024-0025 of Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-3444 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, according to the technology of Patent Document 1, the user of the guidance information display device must possess a piece of paper on which the visible code is printed or a mobile terminal on which it is displayed, and there is a problem that if the user does not possess such a piece of paper or a mobile terminal, the user cannot operate the guidance information display device.

[0005] The present disclosure has been made to solve such problems, and aims to provide an information provision technology that can provide information without requiring special tools such as paper or a mobile terminal. [Means for solving the problem]

[0006] One aspect of an information providing device according to an embodiment of the present disclosure includes an intention estimation unit that estimates the intention of a person based on multiple images of the person and display information being displayed on a display device at the time the multiple images are captured, a selection unit that selects the content or method of information to be provided in accordance with the estimated intention, and an output control unit that controls an output device to provide information in accordance with the result of the selection. Effect of the Invention

[0007] According to the information providing device according to the embodiment of the present disclosure, it is possible to provide information without requiring special tools such as paper or a mobile terminal. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating a configuration example of an information providing device and an information providing system. [Figure 2A] FIG. 2 illustrates an example of a hardware configuration of an information providing device. [Figure 2B] FIG. 2 illustrates an example of a hardware configuration of an information providing device. [Diagram 3] FIG. 11 is a flowchart illustrating an information providing method by the information providing device. [Figure 4A] FIG. 11 is a diagram showing an example of information displayed by the information providing device. [Figure 4B] FIG. 13 is a diagram illustrating an example of gaze detection. [Figure 5A] FIG. 11 is a diagram showing an example of information displayed by the information providing device. [Figure 5B] FIG. 13 is a diagram illustrating an example of gaze detection. [Figure 5C] 13A and 13B are diagrams illustrating an example of display of information after gaze detection. [Figure 6A] FIG. 13 is a diagram illustrating an example of gaze detection. [Figure 6B] 13A and 13B are diagrams illustrating an example of display of information after gaze detection. [Figure 7A] FIG. 13 is a diagram illustrating an example of gaze detection. [Figure 7B] 13A and 13B are diagrams illustrating an example of display of information after gaze detection. [Figure 8] FIG. 13 is a diagram showing an example of a corresponding pattern table (corresponding pattern table 1). [Figure 9] FIG. 13 is a diagram showing an example of a corresponding pattern table (corresponding pattern table 2). DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Various embodiments of the present disclosure will be described in detail below with reference to the drawings. In the drawings, identical or similar parts are denoted by identical or similar reference numerals, and duplicated descriptions of such parts will be omitted. In the present disclosure, the term "or" is used to mean an inclusive logical OR unless otherwise specified.

[0010] Embodiment 1 <Configuration> With reference to FIG. 1, the information providing system Sys. according to the first embodiment of the present disclosure and the information providing device 20 included in the information providing system Sys. will be described. As an example, as shown in FIG. 1, the information providing system Sys. includes a camera 11, a microphone 12, a storage device 13, a storage device 14, a storage device 15, an information providing device 20, a display device 31, and a speaker 32. The display device 31 and the speaker 32 are both examples of output devices in the present disclosure. The output devices including the display device 31 and the speaker 32 provide various information under the control of the information providing device 20. As an example, the display device 31 displays train operation information on a screen included in the display device 31 under the control of the information providing device 20. As another example, the speaker 32 provides voice guidance of the train operation information under the control of the information providing device 20.

[0011] The information provision system Sys. is installed at a service provision spot such as a train station and is used as a digital signage system. The information provision system Sys. does not need to be installed in its entirety at the same location. For example, the camera 11, the microphone 12, the display device 31, and the speaker 32 may be installed on a station platform as an integrated device or not, and the storage device 13, the storage device 14, the storage device 15, and the information provision device 20 may be installed in a driver's room of the station.

[0012] (camera) Camera 11 typically captures multiple time-series images of a person (hereinafter sometimes referred to as a "user") standing in front of display device 31, and outputs the captured multiple images to information providing device 20. Camera 11 is installed so as to be able to capture images of a person standing in front of display device 31. Camera 11 may be provided integrally with display device 31. Camera 11, display device 31, and information providing device 20 may also be provided integrally. Camera 11 captures images at a frame rate of, for example, 20 fps (frames per second) or 30 fps (frames per second).

[0013] (Microphone; Environmental information acquisition device) The microphone 12 is an example of an environmental information acquisition device that acquires information about the user's environment. When the environmental information acquisition device is the microphone 12, the microphone 12 is installed near the camera 11 to collect sounds around the user. The microphone 12 collects sounds and outputs the collected sounds to the information providing device 20. The environmental information acquisition device is not limited to the microphone 12, and may be, for example, a processor that acquires weather information for the area in which the information providing device 20 is installed. When the environmental information acquisition device is a processor that acquires weather information, the environmental information acquisition device acquires the weather information via a communication network (not shown).

[0014] (Storage device) The storage device 13 is a storage device that holds a personal information database that stores personal information. Examples of personal information include a plurality of ID numbers that identify a user, and a facial image and attributes of the user that are associated with each ID number. The facial image of the user may include data that represents the feature amount of the facial image in addition to the data of the facial image.

[0015] The storage device 14 is a storage device that holds the correspondence patterns. As an example, the correspondence patterns are held in the form of a table as shown in FIG. 8, and correspond to "movement of the user's gaze", "characteristics of the displayed information", "user's intention", and "content of the information to be provided or method of providing the information". The correspondence patterns may correspond to "user's attributes", "environment in which the user is currently located", and "content of the information to be provided or method of providing the information" as shown in FIG. 9. For convenience of explanation, the table in FIG. 8 is referred to as "correspondence pattern table 1", and the table in FIG. 9 is referred to as "correspondence pattern table 2". In FIG. 8, nine correspondence patterns are exemplarily defined in the correspondence pattern table 1. The correspondence patterns defined in the correspondence pattern table 1 may be collectively referred to as "correspondence pattern 1". In FIG. 9, seven correspondence patterns are exemplarily defined in the correspondence pattern table 2. The correspondence patterns defined in the correspondence pattern table 2 may be collectively referred to as "correspondence pattern 2".

[0016] Corresponding pattern 2 in Fig. 9 may be used in combination with corresponding pattern 1 in Fig. 8. For example, if the content defined as "content of information to be provided" in corresponding pattern 1 in Fig. 8 can be provided by both display and audio, it may be defined that the information is provided by either display or audio depending on "user attributes" or "the environment in which the user is currently located" as in corresponding pattern 2 in Fig. 9.

[0017] The storage device 15 is a storage device that holds output device information related to an output device used for providing information by the information providing device 20. Examples of the output device information include information identifying the display device 31, the speaker 32, or a lighting device (not shown).

[0018] (Information providing device) 1, the information providing device 20 includes an eye gaze tracking unit 21, a display information acquisition unit 22, an intention estimation unit 23, a selection unit 24, and an output control unit 25. The information providing device 20 may also include an attribute determination unit 26 and an environment determination unit 27 as optional additional functional units. These functional units included in the information providing device 20 will be described below.

[0019] (Eye tracking unit) The gaze tracking unit 21 tracks the gaze of the user captured by the camera 11 from multiple images captured by the camera 11 using a known gaze tracking technique. The gaze tracking unit 21 outputs information on the tracked gaze as gaze information to the display information acquisition unit 22. The gaze information includes position information indicating which position on the screen of the display device 31 the tracked gaze is looking at, and characteristics of the movement of the tracked gaze. The characteristics of the movement of the gaze are, for example, a characteristic that the gaze moves back and forth between two or more multiple areas, or a characteristic that the gaze does not move from the same area. Note that, when multiple people are captured by the camera 11, the gaze tracking unit 21 may track the gaze of each person, or may include people whose gaze is tracked and people whose gaze is not tracked. Hereinafter, the user whose gaze is tracked may be referred to as a "target person".

[0020] (Display information acquisition section) The display information acquisition unit 22 acquires the display information displayed at the line of sight indicated by the line of sight information. That is, the display information acquisition unit 22 acquires the display information displayed at the position indicated by the position information included in the line of sight information. The display information acquisition unit 22 accesses a storage device (not shown) that stores the display information (display content) to acquire the display content displayed on the display device 31, and acquires the display information displayed at the line of sight indicated by the line of sight information. From the relative positional relationship between the user's line of sight and the display position at which the display content is displayed on the display device 31, the display information displayed at the line of sight indicated by the line of sight information can be identified.

[0021] The display information acquisition unit 22 analyzes the characteristics of the display information from the acquired display information. For example, the display of "A" is compared with the display of "B" and the analysis is that "different displays are made." As another example, the display of "A" is compared with the display of a text including "A" and the analysis is that "the same display of A is made." The display information acquisition unit 22 outputs the result of the analysis to the intention estimation unit 23.

[0022] (Intention Estimation Part) The intention estimation unit 23 estimates the intention of the user based on a plurality of images of the user and display information displayed on the display device 31 at the time the plurality of images are captured. In order to perform such estimation, the intention estimation unit 23 estimates the intention of the user from the characteristics of the gaze movement tracked by the gaze tracking unit 21 and the characteristics of the information displayed at the gaze point acquired by the display information acquisition unit 22. As an example, the estimation of the user's intention may be performed on a rule basis. For example, a correspondence pattern table 1 as shown in FIG. 8 may be created in advance, and the estimation may be performed by referring to the correspondence pattern table 1. The correspondence pattern 1 is stored in advance in the storage device 14, and the intention estimation unit 23 accesses the storage device 14 to refer to the correspondence pattern 1.

[0023] The intention estimation unit 23 may estimate the user's intention using a machine learning model that has learned the user's intention. Such a machine learning model may be generated by supervised learning or reinforcement learning, for example, using an image obtained by tracking the user's gaze and the user's intention as teacher data for each piece of display information displayed on the display device 31 as input. The machine learning model may also be generated by utilizing other methods. Details of the operation of the intention estimation unit 23 will be described later.

[0024] (Selection Department) The selection unit 24 selects the content of the information to be provided or the method of providing the information according to the result of estimation by the intention estimation unit 23. This selection is performed by referring to, for example, a correspondence pattern table 1 as shown in Fig. 8 or a correspondence pattern table 2 as shown in Fig. 9.

[0025] The selection unit 24 may use the judgment result by the attribute judgment unit 26 or the judgment result by the environment judgment unit 27 instead of the estimation result estimated by the intention estimation unit 23. Also, in addition to the estimation result estimated by the intention estimation unit 23, the selection unit 24 may use the judgment result by the attribute judgment unit 26 or the judgment result by the environment judgment unit 27. The selection unit 24 outputs the selection result to the output control unit 25. Details of the operation of the selection unit 24 will be described later.

[0026] (Output control section) The output control unit 25 controls the provision of information by an output device associated with the selected content or method of information to be provided, in accordance with the selection result output from the selection unit 24. Information on the associated output device is stored in the storage device 15, and the output control unit 25 accesses the storage device 15 to control the output device.

[0027] (Attribute determination section) The attribute determination unit 26 determines the attributes of the user captured by the camera 11. In the present disclosure, the attributes refer to a profile of a person including at least one of the person's name, age, sex, whether or not the person is able-bodied, whether or not the person has a visual or hearing impairment, whether or not the person is a wheelchair user, assistance history, whether or not the person has a commuter pass, or hobbies. The assistance history is a term that refers to the history of being assisted. The attribute determination unit 26 analyzes the face image of the person captured by the camera 11 using a known technique, and compares information representing the face image of the person captured by the camera 11 with personal information acquired by accessing the personal information database stored in the storage device 13 to determine the attributes of the person captured by the camera 11. The attribute determination unit 26 outputs the determined attributes to the selection unit 24.

[0028] (Environmental Judgment Department) The environment determination unit 27 determines the environment of the target person. In order to determine the environment of the target person, the environment determination unit 27 determines the environment of the target person, such as whether the sound around the target person is loud or quiet, whether the weather at the location where the target person is present is sunny or rainy, and whether the target person is alone or not, from information acquired by an environmental information acquisition device such as the microphone 12.

[0029] As an example, when the display device 31 is installed on a station platform, the environment determination unit 27 determines that the surrounding sound is loud if it is 70 dB or more, and determines that the surrounding sound is quiet if it is less than 70 dB. As another example, when the display device 31 is installed in a station concourse, the environment determination unit 27 determines that the surrounding sound is loud if it is 60 dB or more, and determines that the surrounding sound is quiet if it is less than 60 dB.

[0030] As another example, the environment determination unit 27 determines that the weather is rainy when the amount of rain in the area where the display device 31 is installed is 1 mm or more, and determines that the weather is sunny when the amount of rain in the area is less than 1 mm. When the environment determination unit 27 can obtain information indicating whether the area where the display device 31 is installed is sunny or rainy from a server (not shown), the environment determination unit 27 may use the obtained information as it is.

[0031] As another example, when an image including a target person includes an image of a person whose gaze cannot be detected in addition to the target person, the environment determination section 27 may determine that there is one target person.

[0032] Next, a hardware configuration example of the information providing device 20 will be described with reference to Fig. 2A and Fig. 2B. Each function of the information providing device 20 is realized by a processing circuit. The processing circuitry may be a dedicated processing circuit 100a as shown in Fig. 2A, or a processor 100b as a computer that executes a program stored in a memory 100c as shown in Fig. 2B.

[0033] When the processing circuitry is a dedicated processing circuit 100a, the dedicated processing circuit 100a may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a combination of these. The functions of the information providing device 20 may be realized by a plurality of separate processing circuits, or the functions of the information providing device 20 may be realized collectively by a single processing circuit.

[0034] When the processing circuitry is the processor 100b, the functions of the information providing device 20 are realized by software, firmware, or a combination of software and firmware. The software and firmware are written as programs and stored in the memory 100c. The processor 100b realizes the functions of the information providing device 20 by reading and executing the programs stored in the memory 100c. Here, examples of the memory 100c include non-volatile or volatile semiconductor memories such as random access memory (RAM), read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM), and electrically erasable programmable read-only memory (EEPROM), as well as magnetic disks, flexible disks, optical disks, compact disks, mini disks, and DVDs.

[0035] It should be noted that some of the functions of the information providing device 20 may be realized by dedicated hardware, and the other functions may be realized by software or firmware. In this way, the processing circuit can realize the functions of the information providing device 20 by hardware, software, firmware, or a combination of these.

[0036] <Operation; Overview> Next, an outline of an information providing method by the information providing device 20 will be described with reference to FIG.

[0037] (Step ST1) In step ST1, the gaze tracking unit 21 detects the gaze of the user from a plurality of images captured of the user standing in front of the display device 31, tracks the trajectory of the gaze, and analyzes the characteristics of the gaze movement. The gaze tracking unit 21 outputs the tracked gaze information to the display information acquisition unit 22 as gaze information.

[0038] (Step ST2) In step ST2, the display information acquisition unit 22 acquires display information displayed ahead of the line of sight being tracked, and analyzes the characteristics of the acquired display information.

[0039] (Step ST3) In step ST3, the intention estimation unit 23 estimates the intention of the user based on, for example, characteristics of eye movement and characteristics of the display information. The intention estimation unit 23 estimates the intention of the user by referring to a pattern created in advance that associates the eye movement, the display information, and the intention of the user. As described above, the intention estimation unit 23 may estimate the intention of the user by using a machine learning model.

[0040] (Step ST4) In step ST4, the selection unit 24 selects the content or method of the information to be provided in accordance with the estimated intention.

[0041] (Step ST5) In step ST5, the output control unit 25 controls the output device to provide information according to the result of the selection.

[0042] (Step ST6) In the case where the information providing device 20 additionally includes the attribute determining unit 26, in step ST6, the attribute determining unit 26 determines the attribute of the user.

[0043] In this case, in step ST4, the selection unit 24 selects the content or method of the information to be provided based on the determined attribute in addition to the estimated intention.

[0044] (Step ST7) In the case where the information providing device 20 additionally includes the environment determining unit 27, in step ST7, the environment determining unit 27 determines the environment of the user.

[0045] In this case, in step ST4, the selection unit 24 selects the content or method of the information to be provided based on the estimated intention as well as the determined environment. In step ST4, the selection unit 24 may select the content or method of the information to be provided based on the estimated intention, the determined attribute, and the determined environment.

[0046] <Behavior;Details> <Example 1; Figure 4A and Figure 4B> A detailed example of the operation will be described below with reference to Figures 4A and 4B. Figure 4A is a diagram showing an example of information displayed on the display device 31. More specifically, Figure 4A shows a display example assuming that the display device 31 is installed on the platform of the "Citizen Hall" station. The display device 31 shows train operation information and stop stations, as well as the exits and neighborhood information for the Civic Hall station.

[0047] The train operation information shows that a train bound for "AB Street" will arrive in two minutes. The stops of the train bound for "AB Street" are "AB Street" station and "CD Square" station, which are shown by solid lines in Figure 4A. "EF Street" station and "GH Square" station are stations located in the opposite direction to the direction of travel of the train bound for "AB Street", so the train bound for "AB Street" does not stop at these stations. The dashed lines in Figure 4A show that the train bound for "AB Street" does not stop at "EF Street" station or "GH Square" station.

[0048] In addition, in the example of display in Fig. 4A, it is shown that if you go in the direction of arrow A1, you will reach exit E1, and that the "Citizens' Hall" and "LM Tower" are located near exit E1. In addition, in the example of display in Fig. 4A, it is shown that if you go in the direction of arrow A2, you will reach exit E2, and that the "riverside" and "bus stop" are located near exit E2.

[0049] Fig. 4B is a diagram showing the movement of the user's gaze tracked by the gaze tracking unit 21 in the display example of Fig. 4A. In Fig. 4B, the trajectory of the gaze gazing at the area where "EF Street" or "GH Square" is displayed is represented by LOS1, the trajectory of the gaze gazing at the area where the destination "AB Street" is displayed is represented by LOS2, and the trajectory of the gaze from the destination "AB Street" to the area where "EF Street" or "GH Square" is displayed is represented by LOS3. Note that the trajectory of LOS3 indicates with an arrow that the gaze is moving from the area where the destination "AB Street" is displayed to the area where "EF Street" or "GH Square" is displayed.

[0050] The intention estimation unit 23 acquires the trajectories LOS1 to LOS3 of these lines of sight and the display information displayed at the ends of these lines of sight, i.e., at the positions of the trajectories LOS1 to LOS3. Based on the acquired information, the intention estimation unit 23 determines that the movement of the line of sight of the user goes back and forth between the area where "EF Street" or "GH Square" is displayed and the area where the destination "AB Street" is displayed, if the movement of the line of sight is detected to go back and forth between the area where "EF Street" or "GH Square" is displayed, the movement of the line of sight goes back and forth between the area where the destination "AB Street" is displayed. Note that the "movement of going back and forth" may be a movement that is about to go back and forth, and may not be a movement that has already gone back and forth. For example, it may be determined that there is a "movement of going back and forth" when LOS2 is detected following LOS1, and when LOS3 is detected, the line of sight does not have to completely reach the area indicated by LOS1.

[0051] When such a determination is made, the intention estimation unit 23 analyzes the characteristics of the information displayed at the sight line based on the information displayed at the sight line acquired by the display information acquisition unit 22. In the example of FIG. 4B, the information displayed at the position of the trajectory LOS1 is "EF Street" or "GH Square", the information displayed at the position of the trajectory LOS2 is "AB Street", and the information at the sight line relates to a destination in the opposite direction. In this way, when the movement of the gaze goes back and forth between a plurality of the same areas, when the information at the sight line relates to a destination in the opposite direction, the intention estimation unit 23 refers to the correspondence pattern table 1 in FIG. 8, determines that it corresponds to the correspondence pattern defined in the first row of the correspondence pattern table 1, and estimates that the station the user wants to go to is on the opposite line. The intention estimation unit 23 outputs the result of the estimation to the selection unit 24.

[0052] When the selection unit 24 receives the estimation result estimated by the intention estimation unit 23, the selection unit 24 selects “track number information for the direction to the station the user wishes to visit” as information to be provided to the user by referring to the correspondence pattern table 1 in FIG. 8. The selection unit 24 outputs the selection result to the output control unit 25.

[0053] When the output control unit 25 receives the selection result selected by the selection unit 24, the output control unit 25 controls an output device such as the display device 31 or the speaker 32 to provide information according to the selection result. When "track number information for the direction of the station you want to go to" is selected, the output control unit 25 may perform display control so that the display device 31 displays text information such as "The train bound for GH Square is on the opposite line," or may perform notification control so that the speaker 32 issues audio information such as "The train bound for GH Square is on the opposite line."

[0054] <Example 2; Figures 5A to 5C> A detailed example of the operation will be described below with reference to Figs. 5A to 5C. Fig. 5A shows a display example in which a display area DA2 is added to the display area DA1 in which the display example described with reference to Fig. 4A is displayed. An information button 41 and a guidance message 42 are displayed in the display area DA2. A touch sensor such as a touch panel is provided in the display area DA2 so that the information button 41 can be operated by touch. As the guidance message 42, a message saying "Press this button if you have an inquiry" is displayed.

[0055] Fig. 5B is a diagram showing the movement of the user's gaze tracked by the gaze tracking unit 21 in the display example of Fig. 5A. In Fig. 5B, the trajectory of the gaze gazing at the area where the information button 41 is displayed is represented by LOS4, the trajectory of the gaze gazing at the area where the guidance message 42 is displayed is represented by LOS5, and the trajectory of the gaze from the area where the guidance message 42 is displayed to the area where the information button 41 is displayed is represented by LOS6. Note that the trajectory of LOS6 is indicated by an arrow, indicating that the gaze is moving from the area where the guidance message 42 is displayed to the area where the information button 41 is displayed.

[0056] The intention estimation unit 23 acquires these line of sight trajectories LOS4 to LOS6 and display information displayed at the ends of these lines of sight, i.e., at the positions of the trajectories LOS4 to LOS6. When the intention estimation unit 23 detects, based on the acquired information, that the user's line of sight moves back and forth between the area where the information button 41 is displayed and the area where the guidance message 42 is displayed, it determines that the line of sight moves back and forth between the same multiple areas.

[0057] When such a determination is made, the intention estimation unit 23 analyzes the characteristics of the information displayed at the sight line based on the information displayed at the sight line acquired by the display information acquisition unit 22. In the example of FIG. 5B, the information displayed at the position of the locus LOS4 is the information button 41, the guidance message 42 displayed at the position of the locus LOS5 is the guidance message of the information button 41, and all the information at the sight line relates to the information button 41. In this way, when the movement of the gaze goes back and forth between the same areas, when the information at the sight line relates to the information button, which is the same object, the intention estimation unit 23 refers to the correspondence pattern table 1 in FIG. 8, determines that it corresponds to the correspondence pattern defined in the second row of the correspondence pattern table 1, and estimates that "I don't know what the information button is." The intention estimation unit 23 outputs the result of the estimation to the selection unit 24.

[0058] As can be seen from the examples of Figures 4B and 5B, even if the gaze movement is the same, that is, a back and forth movement through the same multiple areas, the intention estimation unit 23 estimates the user's intention differently by taking into consideration what is displayed in front of the gaze.

[0059] When the selection unit 24 receives the estimation result estimated by the intention estimation unit 23, the selection unit 24 selects “detailed explanation regarding the information button” as information to be provided to the user by referring to the correspondence pattern table 1 in Fig. 8. The selection unit 24 outputs the selection result to the output control unit 25.

[0060] When the output control unit 25 receives the selection result selected by the selection unit 24, it controls an output device such as the display device 31 or the speaker 32 to provide information according to the selection result. When "Detailed explanation about the information button" is selected, for example, as shown in Fig. 5C, the output control unit 25 may control the display device 31 to display a text guide saying "Touch the i button to talk to a station staff member. Please contact us if you need assistance or if there is an emergency." The output control unit 25 may control the speaker 32 to provide a similar guide by voice.

[0061] <Example 3; Figures 6A and 6B> A detailed example of the operation will be described below with reference to Figures 6A and 6B. Figure 6A is a diagram showing the movement of the user's gaze tracked by the gaze tracking unit 21 in the display example of Figure 4A. In Figure 6A, the trajectory of the gaze gazing at the area where "2" is displayed is represented by LOS7, the trajectory of the gaze gazing at the area where "min" is displayed is represented by LOS8, and the trajectory of the gaze from the area where "min" is displayed to the area where "2" is displayed is represented by LOS9. Note that the trajectory of LOS9 is indicated by an arrow, indicating that the gaze is moving from the area where "min" is displayed to the area where "2" is displayed.

[0062] The intention estimation unit 23 acquires these gaze trajectories LOS7 to LOS9 and display information displayed at the ends of these gazes, i.e., at the positions of the trajectories LOS7 to LOS9. When the intention estimation unit 23 detects, based on the acquired information, that the user's gaze moves back and forth between the area where "2" is displayed and the area where "min" is displayed, it determines that the gaze movement is moving back and forth between the same multiple areas.

[0063] When such a determination is made, the intention estimation unit 23 analyzes the characteristics of the information displayed at the sight line based on the information displayed at the sight line acquired by the display information acquisition unit 22. In the example of FIG. 6A, the information displayed at the position of the trajectory LOS7 is "2", the information displayed at the position of the trajectory LOS8 is "min", and all of the information at the sight line relates to the arrival time. In this way, when the movement of the gaze goes back and forth between the same areas, and the information at the sight line relates to the arrival time, which is the same object, the intention estimation unit 23 refers to the correspondence pattern table 1 in FIG. 8, determines that it corresponds to the correspondence pattern defined in the third row of the correspondence pattern table 1, and estimates that the user "wants to see the remaining time in seconds, not in minutes." The intention estimation unit 23 outputs the result of the estimation to the selection unit 24.

[0064] Upon receiving the estimation result estimated by the intention estimation unit 23, the selection unit 24 selects “display of countdown ring” as information to be provided to the user by referring to the correspondence pattern table 1 in Fig. 8. The selection unit 24 outputs the selection result to the output control unit 25.

[0065] When the selection result selected by the selection unit 24 is "display of countdown rings," the output control unit 25 controls the display device 31 to provide information according to the selection result. For example, as shown in Fig. 6B, the output control unit 25 controls the display device 31 to display countdown rings. The display of the rings may change, for example, every second.

[0066] <Example 4; Figures 7A and 7B> A detailed example of the operation will be described below with reference to Fig. 7A and Fig. 7B. Fig. 7A is a diagram showing the movement of the user's gaze tracked by the gaze tracking unit 21 in the display example of Fig. 4A. In Fig. 7A, the trajectory of the gaze gazing at the area where "Citizen Hall" and "LM Tower" are displayed is represented by LOS10, the trajectory of the gaze gazing at the area where the arrow A1 is displayed is represented by LOS11, and the trajectory of the gaze from the area where the arrow A1 is displayed to the area where the "Citizen Hall" and "LM Tower" are displayed is represented by LOS12. Note that the trajectory of LOS12 indicates with an arrow that the gaze is moving from the area where the arrow A1 is displayed to the area where the "Citizen Hall" and "LM Tower" are displayed.

[0067] The intention estimation unit 23 acquires the trajectories LOS10 to LOS12 of these lines of sight and the display information displayed at the ends of these lines of sight, i.e., at the positions of the trajectories LOS10 to LOS12. When the intention estimation unit 23 detects, based on the acquired information, that the user's line of sight moves back and forth between the area where "Citizens Hall" and "LM Tower" are displayed and the area where the arrow A1 is displayed, the intention estimation unit 23 determines that the line of sight moves back and forth between the same multiple areas.

[0068] When such a determination is made, the intention estimation unit 23 analyzes the characteristics of the information displayed at the sight line based on the information displayed at the sight line acquired by the display information acquisition unit 22. In the example of FIG. 7A, the information displayed at the position of the trajectory LOS10 is "Citizen Hall" and "LM Tower", the information displayed at the position of the trajectory LOS11 is an arrow A1, and all of the information at the sight line relates to an exit. In this way, when the movement of the gaze goes back and forth between a plurality of the same areas, when the information at the sight line relates to the same target, that is, an exit, the intention estimation unit 23 refers to the correspondence pattern table 1 in FIG. 8 and determines that it corresponds to the correspondence pattern defined in the fourth row of the correspondence pattern table 1, and estimates that "the destination is not displayed, so I don't know which exit is the destination." The intention estimation unit 23 outputs the result of the estimation to the selection unit 24.

[0069] Upon receiving the estimation result estimated by the intention estimation unit 23, the selection unit 24 selects “additional display related to destination” as information to be provided to the user by referring to the table in Fig. 8. The selection unit 24 outputs the selection result to the output control unit 25.

[0070] When the output control unit 25 receives the selection result selected by the selection unit 24, it controls an output device such as the display device 31 or the speaker 32 to provide information according to the selection result. When "additional display related to destination" is selected, the output control unit 25 controls the display device 31 to display "MN Hotel" and "Taxi stand", for example, as shown in Fig. 7B. The specific content of the additional display may be determined based on information on "Citizens Hall", "LM Tower", or the position where the arrow A1 is displayed.

[0071] <Consideration of attributes or environment; Figure 9> The selection unit 24 may determine the content of the information to be provided or the method of providing the information based on the attributes or environment of the user. This point will be described with reference to Fig. 9. As described above, Fig. 9 is a diagram showing an example of the correspondence pattern table 2.

[0072] The selection unit 24 may determine a method of providing information by referring to the corresponding pattern table 2 based on the attributes of the user (target person) determined by the attribute determination unit 26 or the environment of the target person determined by the environment determination unit 27.

[0073] As an example, if the determination result by the attribute determination unit 26 indicates that the user is “hearing impaired,” the selection unit 24 may refer to the corresponding pattern table 2 based on the determination result and determine that the information should be “displayed in text on the screen.”

[0074] As an example, if the judgment result by the environment judgment unit 27 indicates that “surrounding sounds are low,” the selection unit 24 may refer to the corresponding pattern table 2 based on the judgment result and determine that the information is to be “transmitted as audio information from broadcasting equipment near the person.”

[0075] As an example, if the judgment result by the environment judgment unit 27 indicates that “ambient noise is loud,” the selection unit 24 may refer to the corresponding pattern table 2 based on the judgment result and judge to “display information in text on the screen.”

[0076] As an example, when the determination result by the attribute determination unit 26 indicates "wheelchair user" and the determination result by the environment determination unit 27 indicates that the user is "one person", the selection unit 24 may determine, based on these determination results, to "display a wheelchair route button" by referring to the correspondence pattern table 2. Note that the "wheelchair route button" refers to a button that indicates a route that can be traveled by wheelchair.

[0077] As an example, when the determination result by the attribute determination unit 26 indicates a “wheelchair user” and the determination result by the environment determination unit 27 indicates that the user is “one person” and the weather is “rainy”, the selection unit 24 may refer to the corresponding pattern table 2 based on these determination results and determine to “display the boarding location and contact information for secondary transportation (taxi, bus).”

[0078] As an example, when the result of the determination by the attribute determination unit 26 indicates that the user is "visually impaired," the selection unit 24 may determine, based on the result of the determination, to "broadcast using the left and right directional speakers what is on the right or left side." In this case, the selection unit 24 may determine, together with or instead of this determination, to "broadcast an announcement informing the user of the presence of signage."

[0079] The information providing device 20 or the information providing system Sys. described above detects the user's line of sight, estimates the user's intention based on the detected line of sight, and provides information according to the estimated intention. Therefore, it is possible to provide information without requiring special tools such as paper or a mobile terminal as in the conventional technology.

[0080] In addition, the information providing device 20 or the information providing system Sys. estimates the user's intention based on what information is displayed in the detected line of sight of the user, and therefore can accurately estimate the user's intention. When making the estimation, the display information displayed on the display device 31 at the time the image of the user is captured is acquired and analyzed, thereby improving the accuracy of the estimation of the intention.

[0081] In addition, the information providing device 20 or the information providing system Sys. may select the content of the information to be provided or the method of providing the information, taking into consideration the attributes or environment of the user. By taking into consideration the attributes or environment, it becomes possible to provide appropriate information.

[0082] It is possible to combine the embodiments, and to modify or omit each embodiment as appropriate. [Industrial Applicability]

[0083] The information providing device or information providing system of the present disclosure can be used as a digital signage system that presents information to station users and the like. [Explanation of symbols]

[0084] 11 camera, 12 microphone, 13 storage device, 14 storage device, 15 storage device, 20 information providing device, 21 gaze tracking unit, 22 display information acquisition unit, 23 intention estimation unit, 24 selection unit, 25 output control unit, 26 attribute determination unit, 27 environment determination unit, 31 display device, 32 speaker, 41 information button, 42 guidance message, 100a processing circuit, 100b processor, 100c memory.

Claims

1. an intention estimation unit that estimates an intention of a person based on a plurality of images of the person and display information that is displayed in a plurality of areas of a display device at the time when the plurality of images are captured; A selection unit that selects the content or method of information to be provided according to the estimated intention; an output control unit that controls an output device to provide information according to the result of the selection; a gaze tracking unit that detects the gaze of the person across the plurality of regions, tracks a trajectory of the detected gaze, and analyzes characteristics of the gaze movement; a display information acquisition unit that acquires the display information and analyzes characteristics of the acquired display information; Equipped with the intention estimation unit estimates the intention of the person by referring to a correspondence pattern created in advance that associates the gaze movement, the display information, and the user's intention based on the analyzed gaze movement characteristics and the analyzed display information characteristics; Information provision device.

2. The characteristic of the eye movement is a movement back and forth between the multiple areas on the display device.

2. An information providing device according to claim 1.

3. an attribute determination unit that determines an attribute of the person; Further comprising: The selection unit performs the selection further based on the determined attribute.

2. An information providing device according to claim 1.

4. an environment determination unit that determines an environment of the person; Further comprising: The selection unit performs the selection further based on the determined environment.

2. An information providing device according to claim 1.

5. An information providing device according to any one of claims 1 to 4; A camera that captures the plurality of images; The display device; An information provision system comprising:

6. 6. The information providing system according to claim 5, wherein the output device is the display device.

7. An information providing method performed by an information providing device including an eye gaze tracking unit, a display information acquisition unit, an intention estimation unit, a selection unit, and an output control unit, the method comprising: The gaze tracking unit detects the gaze of the person across multiple regions of the display device from multiple images of the person, tracks a trajectory of the detected gaze, and analyzes characteristics of the gaze movement; a step of the display information acquisition unit acquiring display information of the plurality of areas displayed ahead of the line of sight being tracked at a time point when the plurality of images are captured, and analyzing a feature of the acquired display information; the intention estimation unit estimating the intention of the person by referring to a correspondence pattern that is created in advance to associate the gaze movement, the display information, and the user's intention based on the analyzed gaze movement characteristics and the analyzed display information characteristics; a step of the selection unit selecting information to be provided in accordance with the estimated intention; the output control unit controlling an output device to provide the selected information according to a result of the selection; An information providing method comprising:

8. a function of detecting a line of sight of a person across multiple regions of a display device from multiple images of the person, tracking a trajectory of the detected line of sight, and analyzing characteristics of the line of sight movement; a function of acquiring display information of the plurality of areas displayed ahead of the line of sight being tracked at a time point when the plurality of images are captured, and analyzing characteristics of the acquired display information; A function of estimating the intention of the person by referring to a correspondence pattern that is created in advance to associate the gaze movement, the display information, and the user's intention based on the analyzed gaze movement characteristics and the analyzed display information characteristics; and A function to select the information to be provided according to the presumed intent; controlling an output device to provide the selected information in accordance with the selection; An information providing program that causes a computer to execute the above.

Images