Information presentation system and information presentation method

Abstract

In one aspect of the present invention, when an information presentation control device performs a process for presenting information to a user, the information to be presented is stored in a storage unit in advance, the location and position of a display device are measured on the basis of first sensing data output from a first sensor, and the location and line of sight of the user are estimated on the basis of second sensing data output from a second sensor. Then, based on the measured location and position of the display device and the estimated location and line of sight of the user, a moving medium is controlled so as to move and location the display device within a field of view corresponding to the line of sight of the user and to set the position of the display device, and said information is displayed on the display device with the location and position of the display device being set within the field of view.

Description

Information presentation system and information presentation method

[0001] One aspect of the present invention relates to an information presentation system and an information presentation method that present augmented information at a position that is visible to a user, for example.

[0002] In recent years, various efforts have been made to improve the convenience and efficiency of desktop environments using IT technology. For example, systems have been proposed that use augmented reality (AR) technology to additionally display annotation information on documents on a desk or to display today's task information in the space surrounding a personal computer. These systems can present various types of augmented information that is useful to users in a location that the users can see, which is expected to improve the convenience and work efficiency of users' work.

[0003] Conventionally, information presentation systems using such AR technology have been realized by using display devices such as smart glasses or head-mounted displays (HMDs), or by displaying information using projectors. However, systems using display devices require users to constantly wear special smart glasses, HMDs, or other display devices, which can be quite stressful for users. Furthermore, projectors require a wall or screen onto which to project information. For this reason, it is not possible to display augmented information in the space next to a personal computer that a user is using for work, for example.

[0004] On the other hand, as another information presentation method, a method has been proposed in which particles are emitted from a plurality of particle emitters, respectively, to form a screen consisting of a particle retention field in space, and an image is displayed on this screen using, for example, a projector, as shown in Patent Document 1. Using this method, it becomes possible to present augmented information in the form of an image at any location in space.

[0005] Japanese Patent Application Publication No. 2019-144464

[0006] However, the method described in Patent Document 1 uses fluids such as particles, and therefore has low robustness against air currents in space, making it difficult to present augmented information at a specific location for a long period of time.

[0007] The present invention has been made in light of the above circumstances, and aims to provide a technique that enables information to be permanently presented at a spatial position that is visible to the user.

[0008] In order to solve the above problems, one aspect of an information presentation system or information presentation method according to the present invention is to, when performing a process of presenting information to a user, store information to be presented in a storage unit, measure the position and orientation of a display device based on first sensing data output from a first sensor, and estimate the user's position and line of sight based on second sensing data output from a second sensor, then control a moving medium to move the position and orientation of the display device within a field of view corresponding to the user's line of sight based on the measured position and orientation of the display device and the estimated position and line of sight of the user, and display the information on the display device with the position and orientation of the display device set within the field of view.

[0009] According to one aspect of the present invention, the display device is supported by a moving medium and its position and orientation are variably controlled so that it is within the user's field of view in accordance with the user's position and line of sight. This makes it possible to constantly present information within the user's field of view in response to changes in the user's line of sight. Furthermore, since the display device is used as a display medium, it is possible to continue presenting information as needed.

[0010] That is, according to one aspect of the present invention, it is possible to provide a technology that enables information to be permanently presented at a spatial position that is visible to a user.

[0011] FIG. 1 is a diagram illustrating an example of a configuration of an information presentation system according to an embodiment of the present invention. FIG. 2 is a block diagram illustrating an example of a hardware configuration of an information presentation control device used in the information presentation system illustrated in FIG. 1. FIG. 3 is a block diagram illustrating an example of a software configuration of the information presentation control device used in the information presentation system illustrated in FIG. 1. FIG. 4 is a flowchart illustrating an example of a processing procedure and processing content of an information presentation control process executed by a control unit of the information presentation control device illustrated in FIG. 3. FIG. 5 is a flowchart illustrating an example of a processing procedure and processing content of an information presentation control process executed by a control unit of the information presentation control device illustrated in FIG. 3. FIG. 6 is a diagram illustrating an example of display device information stored in a display device information storage unit included in the information presentation control device illustrated in FIG. 3. FIG. 7 is a diagram illustrating an example of gaze dwell information stored in a gaze dwell information storage unit included in the information presentation control device illustrated in FIG. 3. FIG. 8 is a diagram illustrating an example of presentation information stored in a presentation information storage unit included in the information presentation control device illustrated in FIG. 3.

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0013] [One Embodiment] (Configuration Example) (1) System FIG. 1 is a diagram showing an example of the configuration of an information presentation system according to one embodiment of the present invention.

[0014] In one embodiment, the information presentation system presents, for example, extended information related to the work to a user US working on a personal computer in an office, within the field of view of the user US by variably controlling the position and attitude of a display device DP based on the estimated position and line of sight of the user US.

[0015] In order to change the position and orientation of the display device DP, a dolly-type moving device MS on which a robot arm RB is mounted is provided. The robot arm RB and the moving device MS constitute a moving medium. Specifically, the display device DP is attached to the tip of the robot arm RB, and its position changes as the moving device MS moves, and its orientation changes as the robot arm RB moves. The position is represented, for example, by three-dimensional coordinates, and the orientation is represented, for example, by three elements: roll, tilt, and pan.

[0016] For simplicity of illustration, Figure 1 shows only one moving medium consisting of a display device DP and a moving device MS and robot arm RB that support it, but multiple display devices DP and moving media are provided to enable multiple pieces of information to be presented simultaneously.

[0017] A camera CM1 is placed in the movement area of ​​the display device DP to detect the position and orientation of the display device DP, and a camera CM2 is also placed in the work area of ​​the user US to detect the position and line of sight of the user US.

[0018] The information presentation system according to one embodiment includes an information presentation control device CS that functions as the core of the system. The information presentation control device CS is capable of transmitting presentation information, image data, and control signals between the display device DP, the robot arm RB, the mobile device MS, and the cameras CM1 and CM2 via a network NW.

[0019] The network NW may be, for example, a local area network (LAN) or a wireless LAN established in an office. Note that the network NW may also be a network that employs a low-power wireless data communication standard such as Bluetooth (registered trademark) or a public wireless network that employs the 4G or 5G standard.

[0020] (2) Devices (2-1) Display Device DP The display device DP is, for example, a transmissive display, and receives and displays presentation information transmitted from the information presentation control device CS. The screen size of the display device DP can be selected arbitrarily depending on the amount of presentation information to be displayed at one time or the display size and the size of the space around the user US.

[0021] (2-2) Moving Device MS and Robot Arm RB The moving device MS is a cart equipped with an electric moving mechanism, and moves the display device DP to a specified position in accordance with a movement control signal sent from the information presentation control device CS. The robot arm RB includes, for example, an articulated arm and a servo mechanism for operating this arm, and changes the attitude of the display device DP in accordance with an attitude control signal sent from the information presentation control device CS.

[0022] (2-2) Cameras CM1 and CM2 The camera CM1 is, for example, an infrared camera, and captures an image of an area including an infrared marker attached to the display device DP, and outputs the obtained infrared image data to the information presentation control device CS.

[0023] The camera CM2 is, for example, a web camera, captures an image of an area including at least the face of the user US, and outputs the resulting image data to the information presentation control device CS. While one embodiment describes an example in which a single camera CM2 detects both the position and gaze of the user US, the position and gaze of the user US may be detected by separate cameras. For example, the gaze detection camera may be a camera installed for video calls on the personal computer used by the user US.

[0024] (2-3) Information Presentation Control Device CS The information presentation control device CS is, for example, a general-purpose personal computer. Note that the information presentation control device CS may also be a server computer installed on the web or in the cloud.

[0025] 2 and 3 are block diagrams showing an example of the hardware configuration and software configuration of the information presentation control device CS, respectively.

[0026] The information presentation control device CS has a control unit 1 that uses a hardware processor such as a central processing unit (CPU), and this control unit 1 is connected via a bus 5 to a storage unit having a program storage unit 2 and a data storage unit 3, and an input / output interface (hereinafter, interface will be abbreviated as I / F) unit 4.

[0027] The input / output I / F unit 4 has a communication interface function and transmits and receives presentation information, image data, and control signals between the display device DP, the moving device MS, the robot arm RB, and the cameras CM1 and CM2 via the network NW. Input / output devices 40 such as a keyboard, a display, and an external memory are connected to the input / output I / F unit 4, and input information is taken in from these input / output devices 40.

[0028] The program storage unit 2 is configured by combining, for example, a nonvolatile memory such as a solid-state drive (SSD) as a storage medium that can be written to and read from at any time, and a nonvolatile memory such as a read-only memory (ROM), and stores middleware such as an operating system (OS), as well as application programs required to execute various controls according to an embodiment. Hereinafter, the OS and each application program will be collectively referred to as the program.

[0029] The data storage unit 3 is, for example, a combination of a non-volatile memory such as an SSD that can be written to and read from at any time as a storage medium, and a volatile memory such as a RAM (Random Access Memory), and its storage area includes a presentation information storage unit 31, a display device information storage unit 32, and a gaze dwell information storage unit 33 as storage units according to one embodiment of the present invention.

[0030] The presentation information storage unit 31 is used to store multiple pieces of presentation information input from the input / output device 40. Each piece of presentation information is, for example, image data of a presentation target and information indicating its presentation position coordinates or a relationship with the user's line of sight, associated with identification information (hereinafter referred to as ID) of the presentation information. An example of the presentation information will be described in the operation example.

[0031] The display device information storage unit 32 is used to store information representing the latest state of each display device DP. The display device information includes, for example, information representing the latest position coordinates and attitude, and the ID of the currently presented information, which is associated with the device ID of the display device DP.

[0032] The gaze dwell information storage unit 33 is used to store information representing the gaze dwell state of the user US. The gaze dwell information includes gaze position coordinates and dwell times when the gaze of the user US dwelled during a predetermined period from the present to the past.

[0033] The control unit 1 includes, as processing functions necessary for implementing one embodiment, a presentation information acquisition processing unit 11, a display device position / orientation measurement processing unit 12, a gaze estimation processing unit 13, a user position estimation processing unit 14, a presentation information determination processing unit 15, a display device position / orientation determination processing unit 16, a movement control processing unit 17, an attitude control processing unit 18, and a presentation information output processing unit 19.

[0034] Each of the processing units 11 to 19 is realized by causing a hardware processor of the control unit 1 to execute an application program stored in the program storage unit 2. Note that some or all of the processing units 11 to 19 may be realized using hardware such as an LSI (Large Scale Integration) or an ASIC (Application Specific Integrated Circuit).

[0035] Prior to information presentation control, the presentation information acquisition processing unit 11 stores a plurality of pieces of presentation information input from the input / output device 40 in the presentation information storage unit 31 .

[0036] During the information presentation control period, the display device position and orientation measurement processing unit 12 captures, at regular time intervals, infrared image data output from the camera CM1 for each display device DP via the input / output I / F unit 4. Then, the display device position and orientation measurement processing unit 12 recognizes the positions of the infrared markers from the captured infrared image data by image processing, obtains the position and orientation of the display device DP, and stores the obtained information representing the position and orientation of the display device DP in the display device information storage unit 32 in association with the display device ID.

[0037] During the information presentation control period, the gaze estimation processing unit 13 captures image data output from the camera CM2 at regular time intervals via the input / output I / F unit 4. Then, the gaze estimation processing unit 13 estimates the gaze of the user US based on the captured image data, and stores information representing the gaze pause position coordinates and pause time of the gaze in the gaze pause information storage unit 33.

[0038] During the information presentation control period, the user position estimation processing unit 14 captures image data output from the camera CM2 at regular time intervals via the input / output I / F unit 4. Then, the user position estimation processing unit 14 estimates the user's position based on the captured image data and outputs the estimated position coordinates of the user US to the presentation information determination processing unit 15.

[0039] The presentation information determination processing unit 15 determines the presentation information to be presented and the display device DP to which the presentation information will be presented based on the current position and posture of the display device DP, the position and gaze dwell information of the user US, and the information stored in the presentation information storage unit 31, the display device information storage unit 32, and the gaze dwell information storage unit 33.

[0040] The display device position / posture determination processing unit 16 determines a new position and posture of the display device DP to be presented, based on the presentation information determined as the presentation target by the presentation information determination processing unit 15 and the display device DP to be presented thereto, and the information stored in the presentation information storage unit 31, the display device information storage unit 32, and the gaze dwell information storage unit 33.

[0041] The movement control processing unit 17 generates a movement control signal for moving the display device DP to the new position of the display device DP determined by the display device position / orientation determination processing unit 16, and outputs the generated movement control signal from the input / output I / F unit 4 to the moving device MS.

[0042] The posture control processing unit 18 generates a posture control signal for the display device DP so that the display device DP assumes the new posture determined by the display device position / posture determination processing unit 16, and outputs the generated posture control signal from the input / output I / F unit 4 to the robot arm RB.

[0043] The presentation information output processing unit 19 outputs the presentation information determined as the presentation target by the presentation information determination processing unit 15 from the input / output I / F unit 4 to the display device DP as the presentation destination.

[0044] (Example of Operation) Next, an example of operation of the information presentation system configured as above will be described.

[0045] (1) Pre-registration of Presentation Information Prior to information presentation control, a system administrator or a user inputs presentation information generated in advance to the information presentation control device CS from the input / output device 40. In response, under the control of the presentation information acquisition processing unit 11, the information presentation control device CS acquires the presentation information input by the input / output device 40 via the input / output I / F unit 4 and stores the acquired presentation information in the presentation information storage unit 31.

[0046] 8 shows an example of presentation information stored in the presentation information storage unit 31. That is, for each piece of presentation information, the presentation information storage unit 31 stores image data or character data and information indicating the relationship between the presentation position coordinates and the user's line of sight, in association with a presentation information ID. In the example shown in FIG. 8, presentation position coordinates are fixedly specified for image data with presentation information IDs of "1" to "N+1," and a presentation position is specified to follow the user's line of sight for character data with presentation information ID "N."

[0047] In addition to being registered from the input / output device 40, the presentation information may also be registered via the network NW from, for example, a management terminal used by a system administrator or a terminal used by the organizer of an online meeting.

[0048] (2) Information Presentation Control Information presentation control is performed under the control of the information presentation control device CS as follows: Figures 4 and 5 are flowcharts showing an example of the processing procedure and processing content of information presentation control performed by the control unit 1 of the information presentation control device CS.

[0049] (2-1) Measuring the position and orientation of the display device DP For example, when a request to start information presentation control is input from the input / output device 40 by operation of the user US and this start request is detected in step S10, the control unit 1 of the information presentation control device CS first measures the current position and orientation of each display device DP in step S11 under the control of the display device position / orientation measurement processing unit 12.

[0050] That is, an infrared camera CM1 is placed in the movement area of ​​each display device DP, and this infrared camera CM1 captures an image of an area including each display device DP. The infrared image data captured by the infrared camera CM1 is then transmitted to the information presentation control device CS via the network NW.

[0051] The control unit 1 of the information presentation control device CS, under the control of the display device position / orientation measurement processing unit 12, imports the infrared image data via the input / output I / F unit 4, recognizes images of infrared markers provided on each display device DP from the imported infrared data, and calculates the current position coordinates and orientation (pan, tilt, roll) of each display device DP based on the position within the image frame. The display device position / orientation measurement processing unit 12 then stores the calculated values ​​representing the current position coordinates and orientation of each display device DP in the display device information storage unit 32.

[0052] 6 shows an example of values ​​representing the current position coordinates and orientation of each display device DP stored in the display device information storage unit 32. As shown in this example, the display device information storage unit 32 stores values ​​representing the position coordinates and orientation of each display device DP in association with a device ID. The display device information storage unit 32 also stores an ID indicating the presentation information currently being displayed for each display device DP. The presentation information ID is stored by the presentation information determination processing unit 15, which will be described later.

[0053] Although the above description has been given taking an example in which one infrared camera CM1 captures images of all display devices DP, it is also possible to capture infrared images using a separate infrared camera for each display device DP. In this case, the display device position / orientation measurement processing unit 12 determines the position and orientation of each display device DP based on the infrared image data output from each infrared camera CM1.

[0054] (2-2) Estimation of the Gaze of the User US Subsequently, in step S12, the control unit 1 of the information presentation control device CS estimates the gaze of the user US under the control of the gaze estimation processing unit 13.

[0055] That is, the gaze estimation processing unit 13 acquires image data including the face of the user US captured by the camera CM2 at regular time intervals via the input / output I / F unit 4. Then, each time the gaze estimation processing unit 13 acquires image data, it estimates the gaze of the user US based on the image data. The gaze can be estimated, for example, by recognizing the eyes of the user US from the acquired image data, detecting the three-dimensional position of a bright spot due to the corneal reflection of the eyeball and the two-dimensional position of the pupil from an image representing the eye, and then based on the detected three-dimensional position of the bright spot of the eyeball and the two-dimensional position of the pupil.

[0056] The gaze estimation processing unit 13 calculates the position coordinates and dwell time of the gaze each time the gaze is estimated, and stores the calculated gaze position and dwell time in the gaze dwell information storage unit 33. In this case, if the same gaze position is stored in the gaze dwell information storage unit 33, the value of the dwell time corresponding to this gaze position is updated. Figure 7 shows an example of gaze dwell information stored in the gaze dwell information storage unit 33.

[0057] (2-3) Estimation of the Position of the User US Subsequently, in step S13, the control unit 1 of the information presentation control device CS estimates the position of the user US under the control of the user position estimation processing unit 14.

[0058] That is, the user position estimation processing unit 14 captures image data including the face of the user US captured by the camera CM2 at regular time intervals via the input / output I / F unit 4. Then, each time the user position estimation processing unit 14 captures image data, it estimates the position of the user US based on the image data. The position of the user US can be determined, for example, by recognizing an image of the user US's eyes from the image data and calculating the three-dimensional position coordinates of the image of the eyes.

[0059] (2-4) Determining the presentation information to be presented and the display device to which it will be presented Next, under the control of the presentation information determination processing unit 15, the control unit 1 of the information presentation control device CS performs the process of determining the presentation information to be presented and the process of determining the display device to which it will be presented, as follows.

[0060] That is, the presentation information determination processing unit 15 first determines the field of view of the user US based on the current line of sight of the user US estimated by the line of sight estimation processing unit 13. The field of view is estimated as, for example, a range of a predetermined angle centered on the line of sight.

[0061] Then, in step S14, the presentation information determination processing unit 15 determines whether or not there is any presentation information to be presented within the current field of view of the user US based on the information stored in the presentation information storage unit 31. If there is no presentation information to be presented as a result of this determination, the control unit 1 of the information presentation control device CS determines whether to end presentation control in step S15.

[0062] (2-4-1) Cases in which the information to be presented and a display device are present in the field of view of the user US It is assumed that the presentation information to be presented in the current field of view of the user US is stored in the presentation information storage unit 31. In this case, in step S20, the presentation information determination processing unit 15 next determines whether the display device DP is present at a position corresponding to the current field of view of the user US, based on the display device information stored in the display device information storage unit 32.

[0063] As a result of this determination, it is assumed that the display device DP is present within the current field of view of the user US. In this case, in step S21, the control unit 1 of the information presentation control device CS, under the control of the presentation information output processing unit 19, reads the information to be presented, such as image data, from the presentation information storage unit 1. Then, the presentation information output processing unit 19 outputs the read image data from the input / output I / F unit 4 to the display device DP present within the current field of view of the user US.

[0064] As a result, the image data to be presented is displayed on the display device DP, thereby enabling the user US to check the image data displayed on the display device DP within his or her field of vision.

[0065] (2-4-2) When Presentation Information of the Gaze Tracking Target is Registered: Assuming that the result of the determination in step S20 above is that no display device DP is present in the current field of view of the user US, in step S22, the presentation information determination processing unit 15 refers to the display device information storage unit 32 and determines whether or not there is a display device DP currently displaying presentation information of the gaze tracking target.

[0066] As a result of this determination, it is assumed that there is a display device DP that is displaying presentation information that is the target of gaze tracking. In this case, in step S23, under the control of the display device position / orientation determination processing unit 16, the control unit 1 of the information presentation control device CS calculates new position coordinates and orientation of the display device DP that is displaying the presentation information that is the target of gaze tracking, in order to move the display device DP that is displaying the presentation information that is the target of gaze tracking into the current field of view of the user US.

[0067] Next, in step S24, the control unit 1 of the information presentation control device CS generates a movement control signal and a posture control signal based on the calculated new position coordinates and posture, respectively, under the control of the movement control processing unit 17 and the posture control processing unit 18. Then, the generated movement control signal and posture control signal are output from the input / output I / F unit 4 to the movement device MS and the robot arm RB, respectively.

[0068] Therefore, the display device DP displaying the presentation information to be tracked by the gaze of the user US moves within the user's field of view in accordance with the gaze of the user US, and its posture, i.e., display orientation, is controlled. Therefore, even if the user US moves their gaze, they can continue to view the presentation information to be tracked by the gaze of the user US. In other words, information is presented with priority given to tracking the gaze of the user US.

[0069] (2-4-3) When Information on the Gaze Tracking Target is Not Registered On the other hand, if it is determined in step S22 that there is no display device currently presenting presentation information for the gaze tracking target, in step S25, the control unit 1 of the information presentation control device CS, under the control of the presentation information determination processing unit 15, refers to the display device information storage unit 32 to determine whether there is an empty display device DP that is not displaying presentation information.

[0070] As a result of this determination, it is assumed that an available display device DP is found. In this case, in step S23, the control unit 1 of the information presentation control device CS, under the control of the display device position / orientation determination processing unit 16, calculates new position coordinates and orientation to move the available display device DP into the current field of view of the user US.

[0071] Next, in step S24, the control unit 1 of the information presentation control device CS, under the control of the movement control processing unit 17 and the posture control processing unit 18, generates a movement control signal and a posture control signal based on the calculated position coordinates and posture, and outputs the generated movement control signal and posture control signal from the input / output I / F unit 4 to the movement device MS and the robot arm RB, respectively.

[0072] As a result, the display device DP moves into the field of view of the user US, and its orientation, i.e., its display direction, is controlled so that the information to be presented in the field of view is displayed. Therefore, the user US can visually recognize the information to be presented in the field of view each time the user changes his or her line of sight.

[0073] (2-4-4) When No Free Display Device is Found: Suppose that it is determined in step S25 that no free display device DP exists. In this case, under the control of the presentation information determination processing unit 15, in step S26, the control unit 1 of the information presentation control device CS refers to the gaze dwell information storage unit 33 and the display device information storage unit 32, and selects a display device that displays presentation information at a position corresponding to the line of sight with the shortest gaze dwell time.

[0074] Then, in step S23, the control unit 1 of the information presentation control device CS, under the control of the display device position / attitude determination processing unit 16, calculates new position coordinates and attitude to move the selected display device DP within the current field of view of the user US and control its attitude.

[0075] Next, in step S24, the control unit 1 of the information presentation control device CS, under the control of the movement control processing unit 17 and the posture control processing unit 18, generates a movement control signal and a posture control signal based on the calculated position coordinates and posture, and outputs the generated movement control signal and posture control signal from the input / output I / F unit 4 to the movement device MS and the robot arm RB, respectively.

[0076] As a result, the display device DP, which was located at a position corresponding to the line of sight with the shortest gaze dwell time, moves into the current field of view of the user US, and its posture, i.e., its display orientation, is controlled so that the presentation information to be presented in the current field of view of the user US is displayed. That is, presentation information displayed at a position with a longer gaze dwell time is displayed preferentially over presentation information displayed at a position with a shorter gaze dwell time.

[0077] (2-5) End of Information Presentation Control When the presentation information is displayed on the display device DP, the control unit 1 of the information presentation control device CS finally monitors input of a request to end the presentation control in step S15. If no request to end the presentation control is input, the process returns to step S11, and thereafter, as described above, a series of detection processes from the process of measuring the current position and orientation of the display device DP to the process of estimating the line of sight of the user US in steps S11 to S13 are executed, and if presentation information to be presented exists, a series of information presentation control processes in steps S20 to S26 are executed.

[0078] On the other hand, for example, when the user US inputs a request to end the presentation control via the input / output device 40, the control unit 1 of the information presentation control device CS detects the request in step S15 and ends the information presentation control.

[0079] (Effect) As described above, one embodiment includes a mobile device MS equipped with a robot arm RB on which a display device DP is mounted in a state in which its position and attitude can be changed, and an information presentation control device CS controls the mobile device MS and the robot arm RB based on the measurement results of the position and attitude of the display device DP and the estimation results of the position and line of sight of the user US, thereby moving the display device DP into the field of view of the user US and controlling the attitude, i.e., the display direction, to face the direction of the user US, and in this state displaying information on the display device DP.

[0080] Therefore, the display device DP is automatically positioned within the user US's field of view according to the user's line of sight, making it possible to present information to the user US corresponding to the position of the user's field of view for each line of sight. This makes it possible to display extended information in a space next to the personal computer the user is using for work, for example. Furthermore, because the display device DP is used as a display medium, it is possible to permanently present the presented information as needed.

[0081] In one embodiment, the display device is configured to prioritize display of information that requires immediate attention, such as an incoming email notification or emergency contact information, by setting the information that requires immediate attention as the gaze tracking target. This allows the user US to be notified of the information that requires immediate attention by tracking the user's gaze.

[0082] Furthermore, in one embodiment, a display device is preferentially allocated to information presented at a position corresponding to a longer gaze than a shorter gaze, thereby enabling presentation information that is of high interest to the user and that the user has been viewing for a longer period of time to be presented preferentially.

[0083] [Other Embodiments] (1) In the embodiment, the presentation position is set for the presentation information, but the presentation time or the type of display device may also be set. In this way, it is possible to control the presentation time for each piece of presentation information, or to assign a display device corresponding to the type or size of the presentation information.

[0084] (2) In the embodiment, a case where a cart-type moving device MS equipped with a robot arm RB is used as a moving medium for moving the display device DP has been described as an example. However, the present invention is not limited to this, and a small drone that can move in the space around the user US or a lift-type moving medium that is suspended from the ceiling may also be used.

[0085] (3) In addition, various modifications can be made to the method for detecting the position and orientation of the display device, the method for estimating the user's position and line of sight, the configuration and processing functions of the information presentation control device, the processing procedure and processing content, the type of presented information, etc., without departing from the spirit of this invention.

[0086] Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations according to the embodiments may be appropriately adopted.

[0087] In short, this invention is not limited to the above-described embodiments, and in the implementation stage, the components can be modified and embodied without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0088] US...User CS...Information presentation control device DP...Display device MS...Mobile device RB...Robot arm CM1, CM2...Camera NW...Network 1...Control unit 2...Program storage unit 3...Data storage unit 4...Input / output I / F unit 5...Bus 11...Presentation information acquisition processing unit 12...Display device position / orientation measurement processing unit 13...Gaze estimation processing unit 14...User position estimation processing unit 15...Presentation information determination processing unit 16...Display device position / orientation determination processing unit 17...Movement control processing unit 18...Orientation control processing unit 19...Presentation information output processing unit 31...Presentation information storage unit 32...Display device information storage unit 33...Gaze retention information storage unit

Claims

1. An information presentation system for presenting information to a user, comprising: at least one display device for displaying the information; a moving medium for supporting the display device so that its position and attitude can be variably changed; a first sensor for outputting first sensing data required to measure the position and attitude of the display device; a second sensor for outputting second sensing data required to estimate the position and line of sight of the user; and an information presentation control device connected to the display device, the moving medium, the first sensor, and the second sensor, wherein the information presentation control device comprises: a memory unit for storing the information; a first processing unit for measuring the position and attitude of the display device based on the first sensing data output from the first sensor; a second processing unit for estimating the position and line of sight of the user based on the second sensing data output from the second sensor; a third processing unit for controlling the moving medium to move the position of the display device within a field of view corresponding to the line of sight of the user and set its attitude based on the measured position and attitude of the display device and the estimated position and line of sight of the user; and a fourth processing unit that displays the information on the display device with the position and orientation of the display device set within the field of view.

2. The information presentation system of claim 1, wherein the memory unit stores a plurality of pieces of information including information of a gaze-tracking target specified to be presented by following the user's gaze, and the third processing unit controls the moving medium so that, each time the user's gaze changes, the position and attitude of the display device displaying the information of the gaze-tracking target is changed to within a field of view corresponding to the changed gaze.

3. The information presentation system described in claim 1, wherein the second processing unit manages the gaze dwell time for each of the user's gazes, and the third processing unit has: a processing unit that determines whether or not there is an available display device when the presentation target information to be presented is present within a field of view corresponding to the estimated user's gaze; a processing unit that selects, when there is no available display device, the display device whose position and orientation are set within the field of view corresponding to the gaze of the user having the shortest gaze dwell time among the respective gazes; and a processing unit that controls the moving medium so as to change the position and orientation of the selected display device to within the field of view corresponding to the user's gaze; and the fourth processing unit causes the display device whose position and orientation have been changed to within the field of view corresponding to the user's gaze to display the presentation target information in place of the information that was displayed before the change.

4. An information presentation method in which a process of presenting information to a user is executed by an information presentation control device, comprising the steps of: storing information to be presented in a memory unit; measuring the position and orientation of a display device based on first sensing data output from a first sensor; estimating the user's position and line of sight based on second sensing data output from a second sensor; controlling a moving medium to move the position and orientation of the display device within a field of view corresponding to the user's line of sight and set its orientation based on the measured position and orientation of the display device and the estimated position and line of sight of the user; and displaying the information on the display device with the position and orientation of the display device set within the field of view.

Images