Information processing system, information processing device, information processing method, and program

Abstract

Provided is an information processing system comprising: a line-of-sight detection unit that detects a line-of-sight position of a second user with respect to a second user-side terminal used by the second user, the second user-side terminal sharing a presentation screen displayed on a first user-side terminal used by a first user; a relative coordinate conversion unit that calculates the line-of-sight position of the second user as relative coordinates with respect to the presentation screen; and a display control unit that displays, on the first user-side terminal, second user information based on the relative coordinates of the line-of-sight position of the second user.

Description

Information processing system, information processing device, information processing method, and program

[0001] The present disclosure relates to an information processing system, an information processing device, an information processing method, and a program.

[0002] In recent years, remote conference systems that enable people in remote locations to share documents, audio, etc. have become widely used. By using such remote conference room systems, people in remote locations can easily hold conferences without having to travel long distances.

[0003] JP 2014-50018 A

[0004] However, even if documents can be shared between people in remote locations during a remote conference, it is difficult for the presenter to know whether the viewers in the remote locations are looking at the documents or even which parts of the documents they are looking at. As a result, the presenter cannot grasp the viewers' situation in real time, making it difficult to give an explanation that meets the viewers' interests and understanding, making it difficult to give an effective presentation (communicate information).

[0005] Therefore, this disclosure proposes an information processing system, an information processing device, an information processing method, and a program that can make the information transmission provided by a system for sharing materials, audio, etc. between remote locations more effective.

[0006] According to the present disclosure, an information processing system is provided that includes a gaze detection unit that detects the gaze position of a second user relative to a second user terminal used by a second user that shares a presentation screen displayed on a first user terminal used by the first user, a relative coordinate conversion unit that calculates the gaze position of the second user as a relative coordinate with respect to the presentation screen, and a display control unit that displays second user information based on the relative coordinate of the gaze position of the second user on the first user terminal.

[0007] Furthermore, according to the present disclosure, there is provided an information processing device comprising: a gaze detection unit that detects the gaze position of a second user with respect to a second user terminal used by a second user that shares a presentation screen displayed on a first user terminal used by the first user; a relative coordinate conversion unit that calculates the gaze position of the second user as relative coordinates with respect to the presentation screen; and a display control unit that displays second user information based on the relative coordinates of the gaze position of the second user on the first user terminal.

[0008] Furthermore, according to the present disclosure, an information processing method is provided, including an information processing device detecting a gaze position of a second user relative to a second user terminal used by a second user that shares a presentation screen displayed on a first user terminal used by the first user, calculating the gaze position of the second user as relative coordinates with respect to the presentation screen, and displaying second user information based on the relative coordinates of the gaze position of the second user on the first user terminal.

[0009] Furthermore, according to the present disclosure, a program is provided for causing a computer to function as: a gaze detection unit that detects the gaze position of a second user with respect to a second user terminal used by a second user that shares a presentation screen displayed on a first user terminal used by the first user; a relative coordinate conversion unit that calculates the gaze position of the second user as relative coordinates with respect to the presentation screen; and a display control unit that displays second user information based on the relative coordinates of the gaze position of the second user on the first user terminal.

[0010] FIG. 1 is a functional block diagram of an information processing system according to a first embodiment of the present disclosure. FIG. 2 is an explanatory diagram for describing the operation of a viewer relative coordinate acquisition unit according to the first embodiment of the present disclosure. FIG. 3 is an explanatory diagram for describing the operation of an ideal gaze position estimator according to the first embodiment of the present disclosure. FIG. 4 is an explanatory diagram (part 1) for describing the operation of a second embodiment of the present disclosure. FIG. 5 is a flowchart (part 1) of an information processing method according to a second embodiment of the present disclosure. FIG. 6 is an explanatory diagram (part 2) for describing the operation of a second embodiment of the present disclosure. FIG. 7 is a flowchart (part 2) of an information processing method according to a second embodiment of the present disclosure. FIG. 8 is an explanatory diagram for describing the operation of a third embodiment of the present disclosure. FIG. 9 is a flowchart of an information processing method according to a third embodiment of the present disclosure. FIG. 10 is an explanatory diagram for describing the operation of a fourth embodiment of the present disclosure. FIG. 11 is an explanatory diagram for describing the operation of a fifth embodiment of the present disclosure. FIG. 12 is a flowchart of an information processing method according to a fifth embodiment of the present disclosure. FIG. 13 is an explanatory diagram for describing the operation of a sixth embodiment of the present disclosure. FIG. 14 is a flowchart of an information processing method according to a sixth embodiment of the present disclosure. FIG. 15 is an explanatory diagram for describing the operation of a seventh embodiment of the present disclosure. FIG. 16 is a flowchart of an information processing method according to a seventh embodiment of the present disclosure. FIG. 17 is an explanatory diagram (part 1) for describing the operation of an eighth embodiment of the present disclosure. FIG. 1 is a flowchart (part 1) of an information processing method according to an eighth embodiment of the present disclosure. FIG. 2 is an explanatory diagram (part 2) for explaining the operation in the eighth embodiment of the present disclosure. FIG. 3 is an explanatory diagram (part 3) for explaining the operation in the eighth embodiment of the present disclosure. FIG. 4 is a flowchart (part 4) of an information processing method according to an eighth embodiment of the present disclosure. FIG. 5 is an explanatory diagram for explaining the operation in a ninth embodiment of the present disclosure. FIG. 6 is a flowchart of an information processing method according to a ninth embodiment of the present disclosure. FIG. 7 is an explanatory diagram for explaining the operation in a tenth embodiment of the present disclosure. FIG. 8 is a flowchart of an information processing method according to a tenth embodiment of the present disclosure. FIG. 9 is an explanatory diagram for explaining the operation in an eleventh embodiment of the present disclosure. FIG. 10 is a flowchart (part 1) of an information processing method according to an eleventh embodiment of the present disclosure.FIG. 16 is a flowchart (part 2) of an information processing method according to an eleventh embodiment of the present disclosure. FIG. 17 is an explanatory diagram for explaining operations in a twelfth embodiment of the present disclosure. FIG. 18 is a flowchart of an information processing method according to a twelfth embodiment of the present disclosure. FIG. 19 is an explanatory diagram for explaining operations in a thirteenth embodiment of the present disclosure. FIG. 20 is an explanatory diagram for explaining operations in a fourteenth embodiment of the present disclosure. FIG. 21 is a hardware configuration diagram showing an example of a computer that realizes the functions of each device according to an embodiment of the present disclosure.

[0011] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted. Furthermore, in this specification and the drawings, multiple components having substantially the same or similar functional configurations may be distinguished by adding different letters after the same reference numeral. However, when there is no particular need to distinguish between multiple components having substantially the same or similar functional configurations, only the same reference numerals will be used.

[0012] The explanation will be given in the following order: 1. Background 2. First Embodiment 2.1 Configuration of Information Processing System 2.2 Configuration of Viewer-Side Device 2.3 Configuration of Presenter-Side Device 3. Second Embodiment 4. Third Embodiment 5. Fourth Embodiment 6. Fifth Embodiment 7. Sixth Embodiment 8. Seventh Embodiment 9. Eighth Embodiment 10. Ninth Embodiment 11. Tenth Embodiment 12. Eleventh Embodiment 13. Twelfth Embodiment 14. Thirteenth Embodiment 15. Fourteenth Embodiment 16. Summary 17. Hardware Configuration Example 18. Supplementary Information

[0013] <<1. Background>> First, the background that led the inventors to create the embodiments of the present disclosure will be described.

[0014] As explained above, the use of remote conference room systems has made it possible for people in remote locations to easily hold conferences without having to travel long distances. However, even though remote conferences allow for the sharing of materials, audio, and the like, it is difficult to grasp the status of participants at each location. For example, when a presenter is using a remote conference system to give a presentation while displaying materials on a screen, it is difficult for the presenter to grasp whether the audience is looking at the materials, and even what part of the materials they are looking at.

[0015] Specifically, if the viewer's gaze wanders from the displayed materials, it is assumed that the viewer is not interested in the presentation. Furthermore, if the viewer's gaze remains within the materials but moves away from the part the presenter is explaining, it is assumed that the viewer is taking time to understand the content of the presentation. If this is the case, the presenter can effectively deliver the presentation by changing the way they explain or by taking actions to attract the viewer's attention. However, in a remote conference, the presenter cannot grasp the viewer's gaze, and therefore cannot deliver an explanation that is in line with the viewer's interest and understanding, making it difficult to deliver an effective presentation.

[0016] Furthermore, even if a remote conferencing system capable of detecting the viewer's gaze position is used, if it is not possible to recognize the viewer's position in relation to the material displayed on the screen, it is not possible to know whether the viewer is looking at the material, or even where on the material they are looking.

[0017] In light of this situation, the present inventors have devised the following embodiments of the present disclosure. In the embodiments of the present disclosure, the viewer's gaze position on the viewer's side is converted into relative coordinates with respect to the area of ​​the document screen displayed on the viewer screen. Then, in the embodiments of the present disclosure, the viewer's gaze position is displayed within the area of ​​the document displayed on the presenter screen or superimposed on a frame indicating the area of ​​the document displayed on the presenter screen based on the converted relative coordinates. In the present embodiment, such display allows the presenter to intuitively grasp the viewer's gaze position relative to the document screen. Therefore, according to the present embodiment, the presenter can easily grasp the viewer's gaze position, thereby changing the way or progress of the explanation or taking action to attract the viewer's attention. As a result, the presenter can deliver an effective presentation. The following describes the details of the embodiments of the present disclosure created by the present inventors.

[0018] In the following, an example in which an embodiment of the present disclosure is applied to a remote conference room system will be described. However, the application of the embodiment of the present disclosure is not limited to such a remote conference room system, and may be applied to, for example, remote classes, business negotiations, remote consulting, interviews, sales, etc., or video creation, video distribution, etc.

[0019] <<2. First Embodiment>> <2.1 Configuration of Information Processing System> First, the configuration of an information processing system according to a first embodiment of the present disclosure will be described with reference to Fig. 1. Fig. 1 is a functional block diagram of the information processing system according to this embodiment.

[0020] The information processing system according to this embodiment is a system for conducting a remote conference between, for example, a location where a presenter (first user) 80 is located and a location where a viewer (second user) 90 is located. Note that the information processing system is not limited to a remote conference between two locations, but may also be a system for conducting a remote conference between three or more locations, or may be a system for conducting a conference using different terminals at the same location.

[0021] In detail, as shown in FIG. 1 , the information processing system according to this embodiment includes, for example, a viewer-side device (second user-side terminal) 20 located at a location where a viewer 90 is located, and a presenter-side device (first user-side terminal) 40 located at a location where a presenter 80 is located. Furthermore, the information processing system according to this embodiment also includes, for example, a remote conference system 10 having a function for sharing video (images), audio, various data, etc. between the viewer-side device 20 and the presenter-side device 40. In this embodiment, the viewer-side device 20, the remote conference system 10, and the presenter-side device 40 are connected to each other via a communication network (not shown). In this embodiment, any communication method, whether wired or wireless, can be used in the communication network, but it is preferable to use a communication method that can maintain stable operation. In this embodiment, the viewer-side device 20, the remote conference system 10, and the presenter-side device 40 may be separate devices as shown in FIG. 1 , or each of these devices may be configured as an integrated device. Furthermore, in this embodiment, the viewer-side device 20, the remote conference system 10, and the presenter-side device 40 may each be a single device, or may each be composed of multiple devices. Each device included in the information processing system according to this embodiment will be described below in order.

[0022] (Remote Conference System 10) As described above, the remote conference system 10 has a function for sharing video, audio, various data, and the like between the viewer-side device 20 and the presenter-side device 40. The remote conference system 10 may be, for example, a server, a desktop PC, a tablet PC, a laptop PC, a notebook PC (Personal Computer), a smartphone, or other device. In this embodiment, the remote conference system 10 includes a presenter screen sharing mechanism 100. Specifically, the presenter screen sharing mechanism 100 can, for example, cause the viewer-side device 20 to share and display a screen displayed by the presenter 80 on the presenter-side device 40. Furthermore, the presenter screen sharing mechanism 100 can, for example, cause the presenter-side device 40 to share information indicating the status of the screen displayed on the viewer-side device 20 (e.g., whether multiple windows are open, etc.).

[0023] (Viewer-side device 20) The viewer-side device 20 can share with the viewer 90 a screen (presentation screen) that the presenter 80 has displayed on the presenter-side device 40, and can detect the gaze position of the viewer 90 and transmit the information to the presenter-side device 40. The viewer-side device 20 can be, for example, a desktop, tablet, laptop, or notebook PC, a smartphone, or an HMD (Head Mounted Display).

[0024] The viewer-side device 20 also has, as an input device, a camera (not shown) that can detect the line of sight of the viewer 90. In detail, the camera can be, for example, a web camera mounted on the viewer-side device 20 that captures the face of the viewer 90. The camera is not limited to being mounted on the viewer-side device 20, but can also be a device separate from the viewer-side device 20, such as a camera built into a smartphone that cooperates with the viewer-side device 20.

[0025] The viewer-side device 20 also has input devices such as a touch panel, buttons, levers, sticks, keyboards, controllers, and a mouse (not shown) that accept input operations from the viewer 90. The input devices are not limited to being installed in the viewer-side device 20, but may be separate devices from the viewer-side device 20, such as a touch panel of a tablet PC that cooperates with the viewer-side device 20. In this embodiment, the camera may detect the viewer 90's hand movements, thereby accepting the input operations from the viewer 90. Alternatively, an inertial measurement unit (IMU) (not shown) attached to a part of the viewer 90's body may detect the viewer 90's hand movements, thereby accepting the input operations from the viewer 90. The IMU may include, for example, an acceleration sensor, which is an inertial sensor that acquires acceleration, a gyro sensor (angular velocity sensor), which is an inertial sensor that acquires angular velocity, and the like.

[0026] The viewer-side device 20 may also have, as an input device, a microphone (hereinafter referred to as a microphone) for capturing the speech of the viewer 90 and surrounding environmental sounds.

[0027] Furthermore, the viewer-side device 20 has, as output devices, an audio output unit 220, a viewer-side display unit 200, etc. In this embodiment, the above-mentioned input device and output device may be separate devices from the viewer-side device 20, or may be integrated with the viewer-side device 20. The detailed configuration of the viewer-side device 20 will be described later.

[0028] (Presenter-Side Device 40) The presenter-side device 40 can display a screen of materials toward the presenter 80 and can display the gaze position of the audience 90. The presenter-side device 40 can be, for example, a desktop, tablet, laptop, or notebook PC, a smartphone, an HMD, or other device.

[0029] The presenter-side device 40 may also have, as an input device, a camera (not shown) capable of detecting the line of sight of the presenter 80. Specifically, the camera may be, for example, one or more cameras installed on the presenter 80 side. Furthermore, the presenter-side device 40 may also have, as an input device, a touch panel, a button, a lever, a stick, a keyboard, a controller, a mouse, or the like (not shown) that accepts input operations from the presenter 80. In this embodiment, the presenter-side device 40 may also accept input operations from the viewer 90 by detecting hand operations of the presenter 80 with the camera. Alternatively, in this embodiment, the presenter-side device 40 may also accept input operations from the presenter 80 by detecting hand operations of the presenter 80 with an IMU (not shown) attached to a part of the presenter 80's body.

[0030] The presenter-side device 40 also has, as an input device, a microphone (not shown) that captures the speech of the presenter 80. The presenter-side device 40 also has, as an output device, a presenter-side display unit 300, etc. In this embodiment, the above-mentioned input device and output device may be separate devices from the presenter-side device 40, or may be integrated with the presenter-side device 40. The detailed configuration of the presenter-side device 40 will be described later.

[0031] 1 is merely an example, and the configuration of the information processing system according to this embodiment is not limited to this example. In other words, the configuration of the information processing system according to this embodiment can be flexibly modified according to specifications and operations.

[0032] 2.2 Configuration of the Viewer-Side Device The detailed configuration of the viewer-side device 20 according to this embodiment will be described with reference to Fig. 1 and Fig. 2. Fig. 2 is an explanatory diagram for explaining the operation of the viewer relative coordinate acquisition unit (relative coordinate conversion unit) 204 according to this embodiment.

[0033] 1 , the viewer-side device 20 mainly includes a viewer-side display unit 200, a viewer line-of-sight estimator (line-of-sight detection unit) 202, a viewer relative coordinate acquisition unit 204, a viewer line-of-sight display control unit 210, a state estimator (cause estimator) 212, a viewer operation recognition unit 214, a calibration unit 216, and an audio output unit 220. Each functional unit of the viewer-side device 20 will be explained below in order.

[0034] (Viewer-Side Display Unit 200) The viewer-side display unit 200 is a display device that displays video (images) toward the viewer 90. The display device can be, for example, a liquid crystal display or an organic electroluminescence (EL) display. Alternatively, in this embodiment, the display device may be an HMD that is worn on the head of the viewer 90 and displays video, etc. Furthermore, in this embodiment, the display device may be a projector that projects video, etc. onto the surroundings of the viewer 90 (for example, a wall, a screen, etc.).

[0035] (Viewer Gaze Estimator 202) The viewer gaze estimator 202 detects the gaze position of the viewer 90 relative to the viewer-side device 20. In detail, the viewer gaze estimator 202 analyzes an image from a camera capturing an image of the viewer 90, and identifies the gaze direction of the viewer 90 based on the face position and orientation, pupil position, pupil diameter, eye movement, etc. of the viewer 90. The viewer gaze estimator 202 then estimates the direction of the gaze (gaze position) relative to the viewer-side display unit 200 of the viewer-side device 20. Note that the viewer gaze estimator 202 may detect the amount of movement of the gaze position and acquire information regarding the fixation of the gaze position, such as gaze duration.

[0036] (Viewer Relative Coordinate Acquisition Unit 204) The viewer relative coordinate acquisition unit 204 converts the gaze position of the viewer 90 obtained from the viewer line-of-sight estimator 202 into relative coordinates with respect to a screen (presentation screen) of materials or the like displayed on the viewer-side display unit 200. For example, as shown in FIG. 2 , the viewer relative coordinate acquisition unit 204 acquires coordinates (200, 800) within the viewer-side display unit 200 as the coordinates of the gaze position 92a of the viewer 90 relative to the viewer-side display unit 200 of the presenter-side device 40. Furthermore, the viewer relative coordinate acquisition unit 204 acquires coordinates ((0, 300), (1600, 1080)) within the viewer-side display unit 200 as coordinates indicating the area of ​​the material screen 400a displayed on the viewer-side display unit 200 of the viewer-side device 20. Then, the viewer relative coordinate acquisition unit 204 converts the gaze position of the viewer 90 into relative coordinates with respect to the material screen 400a using the acquired coordinates. In the above example, the relative coordinates of the gaze position of the viewer 90 are (0.13, 0.74).

[0037] In this embodiment, the relative coordinates are transmitted to the presenter-side device 40, and the presenter-side device 40 can display the gaze position of the viewer 90 as a cursor 500a based on the relative coordinates. Specifically, by using the relative coordinates for display, the presenter-side device 40 can display the gaze position 92a of the viewer 90 in the area of ​​the document screen 400b displayed on the presenter-side display unit 300, faithfully reproducing the situation on the viewer 90's side. Thus, even if the sizes of the display units on the viewer 90's side and the presenter 80's side are different, or the position and size of the document screen relative to the display unit are different, the use of relative coordinates can faithfully reproduce the gaze position of the viewer 90. Therefore, according to this embodiment, the presenter 80 can easily determine whether the viewer 90 is looking at the document and, more specifically, where on the document the viewer 90 is looking.

[0038] Furthermore, in this embodiment, even if the viewer 90 is not looking at the document screen 400a, the viewer 90's gaze position 92b can be converted into relative coordinates with respect to the document screen 400a as long as the gaze position 92b is within the detection range of the viewer gaze estimator 202. In this embodiment, the presenter-side device 40 displays the viewer 90's gaze position 600 obtained using the relative coordinates with a cursor 500b on the periphery of the document screen 400b adjacent to the position 600. In this embodiment, the cursor 500b is displayed as a different cursor from the cursor 500a. By displaying the cursor 500b in this manner, the presenter 80 can easily grasp that the viewer 90's gaze is outside the frame of the document screen and the general position of the viewer 90's gaze. In this embodiment, displaying the cursor 500b in this manner can inform the presenter 80 whether the viewer 90's gaze position has been detected, without interfering with the display of other information on the presenter-side device 40.

[0039] (Viewer Gaze Display Control Unit 210) The viewer gaze display control unit 210 displays the estimation result of the gaze position of the viewer 90 by the viewer gaze estimator 202 on the viewer-side display unit 200. Note that in this embodiment, the viewer gaze display control unit 210 may be provided as an option, or may not be provided.

[0040] (State Estimator 212) The state estimator 212 estimates the cause of the viewer 90's gaze position being outside the area (outside the frame) of the document screen. The state estimator 212 estimates the cause based on the viewer 90's gaze position, environmental sounds around the viewer 90, operation information from the viewer 90, and the like. In this embodiment, the state estimator 212 may estimate the viewer 90's psychological state based on the movement of the viewer's eyeballs and estimate the cause based on the estimated psychological state. Furthermore, the state estimator 212 may be provided in a wearable device worn on the viewer 90's body and estimate the viewer 90's psychological state based on sensing data from various sensors (not shown) that detect the viewer 90's state. For example, the sensors may be a blood flow sensor, a sweat sensor, an electroencephalogram sensor, a skin temperature sensor, or the like. In this embodiment, the state estimator 212 may be provided as an option, or may not be provided.

[0041] (Viewer Operation Recognition Unit 214) The viewer operation recognition unit 214 recognizes input operations by the viewer 90, such as input operations by the viewer 90 on input devices such as a touch panel, a button, a lever, a stick, a keyboard, a controller, a mouse, etc. In this embodiment, the viewer operation recognition unit 214 may be provided as an option, or may not be provided.

[0042] (Calibration Unit 216) The calibration unit 216 can calibrate the viewer's line of sight estimator 202 based on the coordinates of the line of sight of the viewer 90 and input operation information from the viewer 90. Details of the operation of the calibration unit 216 will be described later. In this embodiment, the calibration unit 216 may be provided as an option, or may not be provided.

[0043] (Audio Output Unit 220) The audio output unit 220 can be, for example, a speaker that outputs audio to the viewer 90. In this embodiment, the audio output unit 220 may be provided as an option, or may not be provided.

[0044] In this embodiment, the viewer-side device 20 is also provided with a communication unit and a storage unit (not shown). The communication unit can transmit and receive information to and from external devices. In other words, the communication unit is a communication interface having the function of transmitting and receiving data, and is realized by a communication device such as a communication antenna, a transmission / reception circuit, or a port. The storage unit stores programs, information, etc. for the above-mentioned functional units to execute various processes, as well as information obtained by the processes. The storage unit is realized by, for example, a nonvolatile memory such as a flash memory.

[0045] 1 is merely an example, and the configuration of the viewer device 20 according to this embodiment is not limited to this example. In other words, the configuration of the viewer device 20 according to this embodiment can be flexibly modified according to specifications and operations.

[0046] 2.3 Configuration of the Presenter-Side Device The detailed configuration of the presenter-side device 40 according to this embodiment will be described with reference to Fig. 1 to Fig. 3. Fig. 3 is an explanatory diagram for explaining the operation of the ideal gaze position estimator 304 according to this embodiment.

[0047] 1 , the presenter-side device 40 mainly includes a presenter-side display unit 300, a viewer line-of-sight visualization unit (display control unit) 302, an ideal line-of-sight position estimator 304, a drawing unit (display control unit) 306, a presenter shared material recognition unit 308, a presenter voice recognition unit 310, a presenter operation recognition unit 312, and a presenter line-of-sight estimator 314. Each functional unit of the presenter-side device 40 will be described below in order.

[0048] (Presenter-side display unit 300) The presenter-side display unit 300 is a display device that displays video (images) toward the presenter 80. The display device can be, for example, a liquid crystal display or an organic EL display. Alternatively, in this embodiment, the display device may be an HMD worn on the head of the presenter 80. Furthermore, in this embodiment, the display device may be a projector that projects video and the like onto the periphery of the presenter 80 (for example, onto a wall, a screen, etc.).

[0049] (Viewer gaze visualization unit 302) The viewer gaze visualization unit 302 can display information (second user information) regarding the gaze position of the viewer 90 on the presenter side display unit 300 based on relative coordinates indicating the gaze position of the viewer 90 transmitted from the viewer side device 20.

[0050] As described above, in this embodiment, the viewer gaze visualization unit 302 can display the gaze position of the viewer 90 as a cursor 500a (first display mode) based on relative coordinates, as shown in FIG. 2 . In this embodiment, when the gaze position of the viewer 90 is located outside the frame of the document image, the viewer gaze visualization unit 302 displays the gaze position 600 of the viewer 90 obtained using the relative coordinates as a cursor 500b (second display mode) that is different in display mode from the cursor 500a, on the periphery of the area of ​​the document screen 400b adjacent to the gaze position 600. For example, the cursor 500b is different in size, shape, color, etc. from the cursor 500a. By displaying the cursor in this manner, according to this embodiment, the presenter 80 can easily grasp that the gaze of the viewer 90 is located outside the frame of the document screen and the general position of the gaze of the viewer 90.

[0051] (Ideal Gaze Position Estimator 304) The ideal gaze position estimator 304 estimates the ideal position of the gaze of the viewer 90 relative to the document screen. For example, the ideal gaze position estimator 304 estimates the ideal position of the gaze of the viewer 90 based on the gaze position of the presenter 80, the speech of the presenter 80, operation information of the presenter 80 (for example, the position of a cursor pointing to a part of the document), etc. In this embodiment, the ideal gaze position estimator 304 may or may not be provided as an option.

[0052] For example, as shown in FIG. 3 , the ideal gaze position estimator 304 performs processing such as vectorization on the words (words and sentences) extracted from the speech of the presenter 80 recognized by the presenter speech recognition unit 310 (described later), and generates data indicating the characteristics of the words. In the example of FIG. 3 , the presenter 80 utters, "About apples," so data associated with "apples" and words related to "apples" (e.g., "fruits") is generated. The ideal gaze position estimator 304 also recognizes the materials used by the presenter 80 output from the presenter shared material recognition unit 308 (described later), and extracts the content and position of the materials. In the example of FIG. 3 , the materials contain a picture of an "apple" and the words "tangerine" and "grape," so information about "apple," "tangerine," and "grape" is extracted. The ideal gaze position estimator 304 performs processing such as vectorization on the extracted information in the same manner as described above, and generates data indicating the characteristics of the words. The ideal gaze position estimator 304 then searches for information extracted from materials that matches the information from the spoken voice, or for information extracted from materials that have feature data similar to the feature data of the information from the spoken voice (string match search, vector search, etc.). If there is information extracted from materials that matches the information from the spoken voice, or if there is information extracted from materials that have feature data similar to the feature data of the information from the spoken voice, the ideal gaze position estimator 304 estimates the position of that information on the materials as the ideal gaze position of the viewer 90. Furthermore, as shown in FIG. 3 , the ideal gaze position estimator 304 may estimate the position of a cursor 500 pointed by the presenter 80 as the ideal gaze position of the viewer 90.

[0053] In this way, in this embodiment, the ideal gaze position of the viewer 90 can be estimated in real time, and by combining this with information on the gaze position of the viewer 90, it is possible to provide the presenter 80 with useful information for making an effective presentation. Details of this operation will be described later.

[0054] (Drawing unit 306) The drawing unit 306 causes the presenter side display unit 300 to display, for example, information about the state of the viewer 90 that can be estimated from the line of sight, information about the estimated ideal position of the viewer 90's line of sight, etc., superimposed on at least a portion of the displayed material screen (presentation screen).

[0055] (Presenter Shared Material Recognition Unit 308) The presenter shared material recognition unit 308 recognizes materials used by the presenter 80. In this embodiment, the presenter shared material recognition unit 308 may be provided as an option, or may not be provided at all.

[0056] (Presenter Speech Recognition Unit 310) The presenter speech recognition unit 310 recognizes the speech of the presenter 80 acquired by the microphone. Note that in this embodiment, the presenter speech recognition unit 310 may be provided as an option, or may not be provided at all.

[0057] (Presenter Operation Recognition Unit 312) The presenter operation recognition unit 312 recognizes input operations such as input operations performed by the presenter 80 on input devices such as a touch panel, a button, a lever, a stick, a keyboard, a controller, a mouse, etc. In this embodiment, the presenter operation recognition unit 312 may be provided as an option, or may not be provided.

[0058] (Presenter Gaze Estimator 314) The presenter gaze estimator 314 analyzes an image from a camera capturing an image of the presenter 80, and identifies the direction of the presenter's gaze based on the position and orientation of the presenter's face, the position and size of the pupil, the size of the pupil diameter, eye movement, etc., and estimates the direction in which the gaze is directed (gaze position) relative to the presenter-side display unit 300. Note that in this embodiment, the presenter gaze estimator 314 may or may not be provided as an option.

[0059] In this embodiment, the presenter-side device 40 is also provided with a communication unit and a storage unit (not shown). The communication unit can transmit and receive information to and from external devices. In other words, the communication unit is a communication interface having the function of transmitting and receiving data, and is realized by a communication device such as a communication antenna, a transmission / reception circuit, or a port. The storage unit stores programs, information, etc. for the above-mentioned functional units to execute various processes, as well as information obtained through the processes. The storage unit is realized by, for example, a non-volatile memory such as a flash memory. Furthermore, in this embodiment, the presenter-side device 40 may also be provided with an audio output unit (not shown). The audio output unit can be, for example, a speaker or a headphone speaker that outputs the audio of the audience 90 to the presenter 80.

[0060] 1 is merely an example, and the configuration of the presenter-side device 40 according to the present embodiment is not limited to this example. In other words, the configuration of the presenter-side device 40 according to the present embodiment can be flexibly modified according to specifications and operations.

[0061] As described above, in this embodiment, the gaze position of the viewer 90 is converted into relative coordinates with respect to the document screen displayed on the viewer-side display unit 200 and handled. Therefore, in this embodiment, by using relative coordinates, the presenter-side device 40 can faithfully display the gaze position of the viewer 90 relative to the area of ​​the document screen 400b displayed on the presenter-side display unit 300 in the same manner as the situation on the viewer 90's side. According to this embodiment, even if the display sizes of the viewer 90 and the presenter 80 are different, or even if the position and size of the document screen relative to the display are different, the gaze position of the viewer 90 can be faithfully reproduced by using relative coordinates. Therefore, according to this embodiment, the presenter 80 can easily determine whether the viewer 90 is looking at the document and, further, what part of the document the viewer 90 is looking at.

[0062] Furthermore, in this embodiment, when the gaze position of the viewer 90 is located outside the frame of the document image, the gaze position 600 of the viewer 90 obtained using relative coordinates is displayed on the periphery of the area of ​​the document screen 400b adjacent to the position 600 as a cursor 500b having a different shape from the cursor 500a when the gaze position of the viewer 90 is located within the frame of the document image. By displaying in this manner, according to this embodiment, the presenter 80 can easily grasp that the gaze of the viewer 90 is located outside the frame of the document screen and the general position of the gaze of the viewer 90.

[0063] Furthermore, in this embodiment, the ideal gaze position of the viewer 90 can be estimated in real time, and by combining this with information on the gaze position of the viewer 90, it is possible to provide the presenter 80 with useful information for making an effective presentation. Details of specific embodiments that utilize this function will be described below.

[0064] Note that the embodiments of the present disclosure described below are executed by an information processing system according to a first embodiment. However, in the first embodiment of the present disclosure and each embodiment described below, the configuration of the information processing system and the configuration of each device included in the information processing system are not limited to the configuration shown in FIG. 1 and can be modified into various forms. For example, the information processing system according to the first embodiment may be configured as a single information processing device. Furthermore, in the first embodiment of the present disclosure and each embodiment described below, each functional unit is not limited to being included in the device shown in FIG. 1 and may be included in another device, or even in multiple devices.

[0065] Furthermore, in the first embodiment of the present disclosure and each embodiment described below, the operation of each functional block included in the information processing system is not limited to the form shown in Figures 2 and 3, but can be modified into various forms.

[0066] <<3. Second Embodiment>> First, as one embodiment of the present disclosure, it is conceivable to directly display the gaze position of the viewer 90 on the presenter-side device 40. For example, it is conceivable to attempt to display a change over time in the gaze position of the viewer 90. In such a case, however, errors that may occur when estimating the gaze position or blinking (saccades) of the viewer 90 cause the line indicating the change over time to move slightly up, down, left, and right. As a result, even if the change over time in the gaze position of the viewer 90 is displayed to the presenter 80, the change over time is difficult to see, and therefore useful information is not provided to the presenter 80.

[0067] Furthermore, due to the data transmission rate between the viewer-side device 20 and the presenter-side device 40, information on the viewer's 90 gaze position may be temporarily lost, causing the movement of the points indicating changes over time to become discontinuous and difficult to see. Therefore, in a second embodiment of the present disclosure, when displaying changes over time in the viewer's 90 gaze position, a predetermined process is performed to make the changes over time easier to see and provide useful information to the presenter 80. Hereinafter, this second embodiment will be described with reference to FIGS. 4A to 5B. FIGS. 4A and 5A are explanatory diagrams for explaining the operation of this embodiment, and FIGS. 4B and 5B are flowcharts of an information processing method according to this embodiment.

[0068] First, as shown in FIG. 4A , if a time-dependent change 94 in the gaze position of a viewer 90 is directly displayed on the presenter-side device 40, the time-dependent change 94 will fluctuate slightly up, down, left, and right due to errors that may occur when estimating the gaze position and noise caused by the viewer 90's blinking (saccades). Therefore, in this embodiment, an FIR (Finite Impulse Response Filter) filter or an IIR (Infinite Impulse Response) filter is used to remove noise from the information on the gaze position of the viewer 90, and then the time-dependent change 96 is displayed. As shown in FIG. 4A , the time-dependent change 96 obtained by processing does not fluctuate slightly up, down, left, and right, compared to the time-dependent change 94. Because an IIR filter has an impulse response function that returns a non-zero value over an infinitely long period of time, all of the coordinates in the history are fed back to determine the information to be output. Since the FIR filter has a finite-time impulse response function, the coordinates of a predetermined time range in the history are fed back to determine the information to be output, thereby making it possible to obtain a smooth change over time 96, as shown in FIG.

[0069] Next, an example of information processing according to this embodiment will be described. First, as shown in FIG. 4B , the information processing system acquires information on the gaze positions of the viewers 90 (specifically, relative coordinates) (step S41). Next, the information processing system stores the acquired information on the gaze positions of the viewers 90 as a coordinate history in a storage unit (not shown) (step S42). Then, the information processing system determines whether a sufficient number of pieces of information on the gaze positions of the viewers 90 have been stored in order to display changes in the gaze positions of the viewers 90 over time (step S43). If the information processing system determines that a sufficient number of pieces of information on the gaze positions of the viewers 90 have been stored (step S43: Yes), it proceeds to processing in step S44. On the other hand, if the information processing system determines that a sufficient number of pieces of information on the gaze positions of the viewers 90 have not been stored (step S43: No), it proceeds to processing in step S45. Furthermore, the information processing system uses an FIR filter to remove noise and smooth the stored information on the gaze positions of the multiple viewers 90 (step S44). Next, the information processing system displays the change over time in the gaze position of the viewer 90 (step S45), and ends the process.

[0070] 5A , when the change over time 94 in the gaze position of the viewer 90 is directly displayed on the presenter-side device 40, the data transmission rate between the viewer-side device 20 and the presenter-side device 40 may temporarily drop out information about the gaze position of the viewer 90, causing the point cloud representing the change over time 94 to appear discontinuous, as shown by the large circle. Therefore, in this embodiment, a process such as drawing frame interpolation (linear interpolation, etc.) is performed to display a point cloud including small circles as the change over time 96. For example, as shown in FIG. 5A , the change over time 96 obtained by the above process is smoother and not discontinuous, compared to the point cloud representing the change over time 94 shown by the large circle.

[0071] Next, an example of information processing according to this embodiment will be described. As shown in FIG. 5B , the information processing system acquires information on the gaze positions of the viewers 90 (step S51). Next, the information processing system causes the viewer-side device 20 to display changes in the gaze positions of the viewers 90 over time based on the acquired gaze positions of the viewers 90 (step S52). Next, the information processing system performs throttling control based on the data transmission rate (line speed) between the viewer-side device 20 and the presenter-side device 40 to temporarily suppress transmission of the gaze position information of the viewers 90 from the viewer-side device 20 to the presenter-side device 40 (step S53). The information processing system then passes the gaze position information of some of the viewers 90 (step S53: Yes) and transmits it to the presenter-side device 40 (step S54). On the other hand, the information processing system does not pass the gaze position information of some of the viewers 90 (step S53: No). Furthermore, the information processing system performs interpolation processing etc. on the information of the multiple gaze positions (step S55), causes the presenter side device 40 to display the change in the gaze position of the viewer 90 over time (step S56), and ends the processing.

[0072] As described above, in this embodiment, when displaying the change over time in the gaze position of the viewer 90, a predetermined process is performed to obtain a smooth change over time 96, without small up / down and left / right fluctuations. As a result, this embodiment makes the change over time easier to see, and can provide useful information to the presenter 80.

[0073] It should be noted that this embodiment is not limited to the configurations shown in FIGS. 4A to 5B, and can be modified into various configurations.

[0074] <<4. Third Embodiment>> In a third embodiment of the present disclosure, the display format of the gaze position of the viewer 90 is changed using a fixation degree calculated from the amount of gaze movement of the viewer 90 per unit time. In detail, in this embodiment, by changing the display format of the gaze position of the viewer 90 according to the fixation degree, the presenter 80 can easily grasp information such as where the viewer 90 is paying attention for a long time or whether the viewer 90 is confused and unable to focus their gaze. Hereinafter, this third embodiment will be described with reference to FIGS. 6A and 6B . FIG. 6A is an explanatory diagram for describing the operation of this embodiment, and FIG. 6B is a flowchart of an information processing method according to this embodiment.

[0075] As shown in the upper part of FIG. 6A , in this embodiment, when displaying a change over time in the gaze position of the viewer 90, if the gaze movement amount per unit time is small and the fixation degree is high, for example, a line indicating the change over time 98a may be drawn thick and dark. Furthermore, in this embodiment, if the fixation degree is high, for example, a cursor indicating the gaze position may be drawn thick and dark. On the other hand, as shown in the lower part of FIG. 6A , when displaying a change over time in the gaze position of the viewer 90, if the gaze movement amount per unit time is small and the fixation degree is low, for example, a line indicating the change over time 98b may be drawn thin and light. Furthermore, in this embodiment, if the fixation degree is low, for example, a cursor indicating the gaze position may be drawn thin and light.

[0076] For example, in this embodiment, the amount of gaze movement per predetermined unit time is calculated from information on the gaze position of the viewer 90, and the reciprocal of the calculated amount of movement is then calculated to obtain a fixation degree indicating the degree of fixation of the gaze position. In this embodiment, the display form of the change in the gaze position of the viewer 90 over time is changed based on this fixation degree. Therefore, according to this embodiment, the presenter 80 can easily and intuitively grasp information such as where the viewer 90 is paying attention for a long time, or whether the viewer 90 is confused and unable to focus their gaze.

[0077] Next, an example of information processing according to this embodiment will be described. First, as shown in FIG. 6B , the information processing system acquires information on the gaze positions of the viewers 90 (specifically, relative coordinates) (step S61). Next, the information processing system stores the acquired information on the gaze positions of the viewers 90 as a coordinate history in a storage unit (not shown) (step S62). Then, the information processing system determines whether a sufficient number of pieces of information on the gaze positions of the viewers 90 have been stored in order to display changes in the gaze positions of the viewers 90 over time (step S63). If the information processing system determines that a sufficient number of pieces of information on the gaze positions of the viewers 90 have been stored (step S63: Yes), it proceeds to processing in step S64. On the other hand, if the information processing system determines that a sufficient number of pieces of information on the gaze positions of the viewers 90 have not been stored (step S63: No), it proceeds to processing in step S68.

[0078] Next, the information processing system calculates a fixation degree using the stored information on the gaze positions of the multiple viewers 90 (step S64). The information processing system then compares the calculated fixation degree with a predetermined threshold (step S65). If the information processing system determines that the calculated fixation degree is equal to or greater than the predetermined threshold (step S65: Yes), the system proceeds to processing in step S67. On the other hand, if the information processing system determines that the calculated fixation degree is less than the predetermined threshold (step S65: No), the system proceeds to processing in step S66. Furthermore, the information processing system draws a cursor or line indicating the gaze position of the viewer 90 in a thin, light manner (step S66). The information processing system draws a cursor or line indicating the gaze position of the viewer 90 in a thick, dark manner (step S67). Finally, the information processing system displays a cursor or line indicating the gaze position of the viewer 90 (step S68), and the process ends.

[0079] It should be noted that this embodiment is not limited to the configurations shown in FIGS. 6A and 6B, and can be modified into various other configurations.

[0080] <<5. Fourth Embodiment>> Furthermore, in each of the above-described embodiments of the present disclosure, the gaze position of the viewer 90 is displayed in real time, which may be distracting to the presenter 80. Furthermore, in each of the above-described embodiments of the present disclosure, the gaze position of the viewer 90 is displayed in real time, which may be uncomfortable for the viewer 90 from the perspective of privacy. Therefore, in a fourth embodiment of the present disclosure, the estimation and display of the gaze position can be stopped at the will of the presenter 80 or the viewer 90. Such a fourth embodiment will be described below with reference to FIG. 7. FIG. 7 is an explanatory diagram for describing the operation of this embodiment.

[0081] 7 , when stopping the display of the gaze positions of the viewers 90, the presenter 80 operates, for example, a button 702 displayed together with the names of the viewers in the video 950 of each viewer displayed on the presenter-side display unit 300. According to this embodiment, by operating such button 702, the presenter 80 can stop the display of the gaze positions of the viewers 90 or display them again.

[0082] 7, when the viewer 90 wishes to stop detection of the position of his / her gaze, the viewer 90 operates a button 700 displayed on the viewer-side display unit 200. According to this embodiment, by operating such button 700, the viewer 90 can stop detection of the position of his / her gaze or start detection again.

[0083] As described above, according to this embodiment, if the real-time display of the gaze position of the viewer 90 is disturbing, the presenter 80 can stop the display of the gaze position of the viewer 90. Furthermore, according to this embodiment, if the viewer 90 feels uncomfortable with the detection of his or her own gaze position from the viewpoint of privacy, the viewer 90 can stop the detection of his or her own gaze position.

[0084] It should be noted that this embodiment is not limited to the configuration shown in FIG. 7, and can be modified into various other configurations.

[0085] <<6. Fifth Embodiment>> In addition, in each of the above-described embodiments of the present disclosure, the gaze position of the viewer 90 is displayed using a cursor or the like. However, the embodiments of the present disclosure are not limited to displaying the gaze position of the viewer 90 using a cursor. Therefore, in a fifth embodiment of the present disclosure, the gaze position of the viewer 90 is displayed in a display form other than a cursor. Hereinafter, such a fifth embodiment will be described with reference to FIGS. 8A and 8B. FIG. 8A is an explanatory diagram for describing the operation in this embodiment, and FIG. 8B is a flowchart of an information processing method according to this embodiment.

[0086] For example, in this embodiment, a grid 350 is provided on the document screen (presentation screen) and the screen is divided into a plurality of cells 352. In this embodiment, when the gaze position of the viewer 90 is located within the document screen, the cell 352 (more specifically, the entire cell 352 or the frame (outer periphery) of the cell 352) containing the gaze position of the viewer 90 is displayed by blinking, as shown in the upper part of Fig. 8A. Furthermore, in this embodiment, when the gaze position of the viewer 90 is located outside the document screen, of the four line segments 354 constituting the periphery of the cell 352 adjacent to the gaze position of the viewer 90, the line segment 354 adjacent to the gaze position of the viewer 90 is displayed by blinking, as shown in the lower part of Fig. 8A.

[0087] In this embodiment, the gaze position of the viewer 90 is not displayed with a cursor, but is displayed with a cell 352 that includes the gaze position of the viewer 90 or a line segment 354 adjacent to the gaze position of the viewer 90. Therefore, in this embodiment, even if the estimation accuracy of the gaze position of the viewer 90 is not high, the gaze position of the viewer 90 can be presented to the presenter 80. Furthermore, in this embodiment, the cell 352 to be displayed is specified based on whether the gaze position is included in the cell 352 having a predetermined area. Therefore, even if the gaze position of the viewer 90 moves drastically, the displayed cell 352 does not change immediately, and therefore the presenter 80 does not have to bear a strain on its eyes.

[0088] Next, an example of information processing according to this embodiment will be described. As shown in FIG. 8B , first, the information processing system acquires the relative coordinates of the gaze position of the viewer 90 (step S81). Next, the information processing system determines whether the gaze position of the viewer 90 is within the frame of the document screen (step S82). If the information processing system determines that the gaze position of the viewer 90 is within the frame of the document screen (step S82: Yes), it proceeds to processing of step S83. On the other hand, if the information processing system determines that the gaze position of the viewer 90 is outside the frame of the document screen (step S82: No), it proceeds to processing of step S86.

[0089] Next, the information processing system creates a grid that divides the frame of the document screen, and identifies a cell 352 that includes the gaze position of the viewer 90 from among the multiple cells 352 generated by the grid (step S83). Then, the information processing system draws the identified cell 352 so that it glows (step S84). Next, the information processing system creates a grid that divides the frame of the document screen, and identifies a line segment 354 that is close to the gaze position of the viewer 90 from among the multiple cells 352 generated by the grid, of a cell 352 that is adjacent to the gaze position of the viewer 90 (step S85). Then, the information processing system draws the identified line segment 354 so that it glows (step S86). Finally, the information processing system displays the drawings made in steps S84 and S86 (step S87), and ends the process.

[0090] It should be noted that this embodiment is not limited to the configurations shown in FIGS. 8A and 8B, and can be modified into various other configurations.

[0091] 7. Sixth Embodiment Next, a sixth embodiment of the present disclosure will be described with reference to Fig. 9A and Fig. 9B. Fig. 9A is an explanatory diagram for describing the operation in this embodiment, and Fig. 9B is a flowchart of an information processing method according to this embodiment.

[0092] In the above-described embodiments of the present disclosure, as shown in the upper part of FIG. 9A , when the viewer 90's gaze position is within the frame of the document screen, the viewer 90's gaze position is displayed with a cursor 500a (inside frame cursor), and when the viewer 90's gaze position is outside the frame of the document screen, the viewer 90's gaze position is displayed with a cursor 500b (outside frame cursor). That is, in the above-described embodiments of the present disclosure, the display mode of the viewer 90's gaze position is changed depending on whether the viewer 90's gaze position is within the frame of the document screen (the shape and size of the cursor 500 are different in the upper part of FIG. 9A ). Therefore, in the above-described embodiments of the present disclosure, the display mode is switched the moment the viewer 90's gaze position moves from within the frame of the document screen to outside the frame, or when the viewer 90's gaze position moves back and forth between inside and outside the frame. If the display mode switches too frequently, the presenter 80 may be distracted and be unable to focus on the explanation, etc.

[0093] Therefore, in a sixth embodiment of the present disclosure, even if the gaze position of the viewer 90 is outside the frame of the document screen, if the distance from the document screen (frame) is within a predetermined threshold (predetermined range), the cursor is displayed in the same display form (inside-frame cursor 500a) as when the gaze position of the viewer 90 is within the frame of the document screen, as shown in the lower part of Fig. 9A . Furthermore, in this embodiment, if the gaze position of the viewer 90 is outside the frame of the document screen and the distance from the document screen (frame) is equal to or greater than a predetermined threshold, the cursor is displayed in the display form (outside-frame cursor 500b) as when the gaze position of the viewer 90 is outside the frame of the document screen.

[0094] In this embodiment, even if the gaze position of the viewer 90 is outside the frame of the document screen, if it is within a predetermined range from the frame of the document screen, the in-frame cursor 500a is displayed. In this embodiment, the out-of-frame cursor 500b is displayed only when the gaze position of the viewer 90 is significantly outside the frame of the document screen. That is, in this embodiment, a buffer is provided for the timing of switching the display mode of the gaze position of the viewer 90, thereby preventing the display mode from switching too rapidly. Therefore, according to this embodiment, the presenter 80 only needs to be aware of the out-of-frame cursor 500b when the gaze position of the viewer 90 is significantly outside the frame of the document screen, allowing him or her to focus on his or her explanation, etc. Note that in this embodiment, it is preferable to set a buffer (i.e., a predetermined threshold) depending on the estimation accuracy of the gaze position.

[0095] Next, an example of information processing according to this embodiment will be described. As shown in FIG. 9B , first, the information processing system acquires the relative coordinates of the gaze position of the viewer 90 (step S91). Next, the information processing system determines whether the gaze position of the viewer 90 is within the frame of the document screen (step S92). If the information processing system determines that the gaze position of the viewer 90 is within the frame of the document screen (step S92: Yes), it proceeds to processing of step S93. On the other hand, if the information processing system determines that the gaze position of the viewer 90 is outside the frame of the document screen (step S92: No), it proceeds to processing of step S94. Then, the information processing system displays the gaze position of the viewer 90 with an in-frame cursor 500a (step S93).

[0096] Next, the information processing system determines whether the distance from the viewer 90's gaze position to the frame of the document screen is within a predetermined threshold (step S94). If the information processing system determines that the distance from the viewer 90's gaze position to the frame of the document screen is within the predetermined threshold (step S94: Yes), it proceeds to processing of step S93. On the other hand, if the information processing system determines that the distance from the viewer 90's gaze position to the frame of the document screen is equal to or greater than the predetermined threshold (step S94: No), it proceeds to processing of step S95. Then, the information processing system displays the viewer 90's gaze position with an outside-frame cursor 500b (step S95). Finally, the information processing system ends the processing.

[0097] It should be noted that this embodiment is not limited to the configurations shown in FIGS. 9A and 9B, and can be modified into various other configurations.

[0098] 8. Seventh Embodiment Next, a seventh embodiment of the present disclosure will be described with reference to Fig. 10A and Fig. 10B. Fig. 10A is an explanatory diagram for describing the operation in this embodiment, and Fig. 10B is a flowchart of an information processing method according to this embodiment.

[0099] 10A , even if the viewer 90's gaze position 92 is located within a window 750a showing a document screen, the viewer 90 may be looking at another window 750b that overlaps with the window 750a because, for example, the viewer 90 is not interested in the explanation by the presenter 80. Therefore, in the seventh embodiment of the present disclosure, a warning is displayed indicating that the viewer is looking at another window 750b that overlaps with the document screen so that the presenter 80 can be made aware of such a situation. In this embodiment, the warning display can be performed, for example, by blinking the cursor within the frame.

[0100] According to this embodiment, the presenter 80 can be made aware that the viewer 90 is looking at another window 750b that overlaps the material window 750a. Furthermore, according to this embodiment, the presenter 80 can grasp the detailed status of the viewer 90, and therefore can give an effective presentation, such as by taking actions that will attract the interest of the viewer 90.

[0101] Next, an example of information processing according to this embodiment will be described. As shown in FIG. 10B , first, the information processing system acquires the relative coordinates of the viewer's 90 line of sight (step S101). Next, the information processing system determines whether the viewer's 90 line of sight is within the frame of the document screen (step S102). If the information processing system determines that the viewer's 90 line of sight is within the frame of the document screen (step S102: Yes), the information processing system proceeds to processing in step S103. On the other hand, if the information processing system determines that the viewer's 90 line of sight is outside the frame of the document screen (step S102: No), the information processing system proceeds to processing in step S105.

[0102] Next, the information processing system determines whether the gaze position of the viewer 90 is outside the coordinate range of another window 750b that overlaps with the window 750a showing the document screen (step S103). If the information processing system determines that the gaze position of the viewer 90 is outside the coordinate range of the other window 750b (step S103: Yes), the information processing system proceeds to processing of step S104. On the other hand, if the information processing system determines that the gaze position of the viewer 90 is within the coordinate range of the other window 750b (step S103: No), the information processing system proceeds to processing of step S106.

[0103] The information processing system then selects to display the gaze position of the viewer 90 with the in-frame cursor 500a (step S104). Next, the information processing system displays the gaze position of the viewer 90 with the out-frame cursor 500b (step S105). Furthermore, the information processing system displays the gaze position of the viewer 90, for example, by blinking the in-frame cursor 500a (displaying a warning cursor) (step S106). The information processing system then ends the process.

[0104] It should be noted that this embodiment is not limited to the configurations shown in FIGS. 10A and 10B, and can be modified into various other configurations.

[0105] 9. Eighth Embodiment Next, an eighth embodiment of the present disclosure will be described with reference to FIGS. 11A to 13B . FIGS. 11A , 12A , and 13A are explanatory diagrams for explaining the operation of this embodiment, and FIGS. 11B , 12B , and 13B are flowcharts of an information processing method according to this embodiment. As described above, in this embodiment of the present disclosure, the ideal gaze position estimator 304 can estimate the ideal gaze position of the viewer 90 in real time. Therefore, in the eighth embodiment of the present disclosure, information on the ideal gaze position of the viewer 90 and information on the actual gaze position of the viewer 90 are combined to provide useful information for the presenter 80 to effectively deliver a presentation. In particular, in this embodiment, the display mode of the gaze position of the viewer 90 is changed depending on the positional relationship between the gaze position of the viewer 90 and the ideal position.

[0106] In this embodiment, for example, as shown in FIG. 11A , when the presenter 80 explains a document while pointing at a portion of the document with a cursor (mouse cursor) 960, the ideal gaze position estimator 304 estimates the position pointed to by the cursor 960 as the ideal gaze position of the viewer 90. For example, as shown in FIG. 11A , when the viewer 90's actual gaze is at position 92, the viewer 90's gaze is further away from the ideal gaze position 960 than when the viewer 90's gaze position is at position 91. Therefore, in this embodiment, when the gaze position is at position 91, the viewer 90's gaze is away from the location the presenter 80 wants them to look at, and a highlight (warning display) is displayed to inform the presenter 80 of this. Therefore, according to this embodiment, the presenter 80 can recognize that the viewer 90's gaze is away from the location the presenter 80 wants them to look at, thanks to the highlight, and can immediately take action to deliver an effective presentation.

[0107] For example, if the presenter 80's explanation becomes faster than the viewer 90 can understand, it is expected that the viewer 90's gaze position 92 will gradually move away from the viewer 90's ideal gaze position 91. Therefore, according to this embodiment, since it is possible to immediately notify the presenter 80 that the viewer 90's gaze position 92 and the viewer 90's ideal gaze position 91 are different, the presenter 80 can take action such as slowing down the speed of the explanation or repeating the explanation.

[0108] Next, an example of information processing according to this embodiment will be described. As shown in FIG. 11B , first, the information processing system acquires the relative coordinates of the gaze position of the viewer 90 (step S111). Next, the information processing system acquires information on the ideal gaze position of the viewer 90 (step S112). Furthermore, the information processing system determines whether the gaze position of the viewer 90 is within the frame of the document screen (step S113). If the information processing system determines that the gaze position of the viewer 90 is within the frame of the document screen (step S113: Yes), it proceeds to processing of step S114. On the other hand, if the information processing system determines that the gaze position of the viewer 90 is outside the frame of the document screen (step S113: No), it proceeds to processing of step S116.

[0109] Next, the information processing system determines whether the distance from the ideal position to the gaze position of the viewer 90 is within a predetermined threshold (step S114). If the information processing system determines that the distance from the ideal position to the gaze position of the viewer 90 is within the predetermined threshold (step S114: Yes), the information processing system proceeds to processing in step S115. On the other hand, if the information processing system determines that the distance from the ideal position to the gaze position of the viewer 90 is equal to or greater than the predetermined threshold (step S114: No), the information processing system proceeds to processing in step S117.

[0110] The information processing system then selects to display the gaze position of the viewer 90 with the in-frame cursor 500a (step S115). Next, the information processing system displays the gaze position of the viewer 90 with the out-frame cursor 500b (step S116). Furthermore, the information processing system displays the gaze position of the viewer 90, for example, by blinking the in-frame cursor 500a (display of a warning cursor) (third display mode) (step S117). The information processing system then ends the process.

[0111] Furthermore, a modification of this embodiment will be described. Here, for example, as shown in FIG. 12A , a modification that can be applied when the material is composed of multiple columns 980 is described. In such a case, it is assumed that the ideal gaze position of the viewer 90 moves sequentially within the area of ​​each column 980 in accordance with the order of the presenter 80's explanation. However, the viewer 90 may never gaze at some of the columns 980 that include such ideal gaze positions. In such cases, it is highly likely that the viewer 90 does not recognize the content written in the corresponding column 980. Therefore, in this modification, the presenter 80 is continuously shown columns 980 that the viewer 90 never gazes at. Therefore, according to this modification, such a display allows the presenter 80 to notice the existence of a column 980 that the viewer 90 never gazes at, even after the series of explanations has ended. Therefore, according to this modification, the presenter 80 can take action to deliver an effective presentation, such as pointing to the column 980 again to return the viewer 90's gaze to that column 980. Furthermore, in this embodiment, the display indicating the existence of a column 980 that the viewer 90 never pays attention to continues, so that the viewer 90 can notice the relevant column 980 after the explanation is over and review the content in that column 980.

[0112] Next, an example of information processing according to this embodiment will be described. As shown in Fig. 12B, first, the information processing system acquires the relative coordinates of the gaze position of the viewer 90 (step S121). Next, the information processing system acquires information on the ideal gaze position of the viewer 90 (step S122). Furthermore, the information processing system acquires information on the relative coordinates on the document screen of the region of column 980 included in the document (step S123).

[0113] Next, the information processing system determines whether the gaze position of the viewer 90 is within the frame of the document screen (step S124). If the information processing system determines that the gaze position of the viewer 90 is within the frame of the document screen (step S124: Yes), the information processing system proceeds to the processing of step S126. On the other hand, if the information processing system determines that the gaze position of the viewer 90 is outside the frame of the document screen (step S124: No), the information processing system proceeds to the processing of step S127.

[0114] Next, the information processing system determines whether the ideal position of the viewer's 90's gaze is within column 980 (step S125). If the information processing system determines that the ideal position of the viewer's 90's gaze is within column 980 (step S125: Yes), it identifies the column 980 that includes the ideal position of the viewer's 90's gaze as the relevant column 980, and proceeds to the processing of step S126. On the other hand, if the information processing system determines that the ideal position of the viewer's 90's gaze is outside column 980 (step S125: No), it returns to the start.

[0115] Next, the information processing system determines whether the distance from the center of gravity of the column 980 to the position of the viewer's gaze is within a predetermined threshold (step S126). If the information processing system determines that the distance from the center of gravity of the column 980 to the position of the viewer's gaze is within the predetermined threshold (step S126: Yes), the information processing system proceeds to processing of step S128. On the other hand, if the information processing system determines that the distance from the center of gravity of the column 980 to the position of the viewer's gaze is equal to or greater than the predetermined threshold (step S126: No), the information processing system proceeds to processing of step S129.

[0116] The information processing system then displays the gaze position of the viewer 90 with the outside-frame cursor 500b (step S127). Next, the information processing system displays the gaze position of the viewer 90 with the inside-frame cursor 500a (step S128). Furthermore, the information processing system displays the gaze position of the viewer 90, for example, by blinking the inside-frame cursor 500a (displaying a warning cursor), and also displays the relevant column 980 identified in step S125 by blinking (step S129). In step S129, it is preferable that the display of the relevant column 980 continues until a predetermined process is performed. Then, the information processing system ends the process.

[0117] Next, another modification of this embodiment will be described. Incidentally, a cause of the deviation between the gaze position 92 of the viewer 90 and the ideal gaze position 91 of the viewer 90 is that the viewer 90 finds it difficult to understand an explanation written in a certain part of the material, and the viewer 90's gaze may remain fixed on that part even after the presenter 80 has moved on to explaining the next part. In such a case, in the embodiment and modification described above, as shown in FIG. 13A , if the gaze position 92 of the viewer 90 deviates from the ideal gaze position 91 of the viewer 90 with respect to the presenter 80, the gaze position 92 of the viewer 90 at that time is displayed using a cursor 560 or the like. The presenter 80 may then notice the display of the cursor 560 and explain the part again, but if the cursor 560 remains displayed, it may be cumbersome to repeat the explanation. Therefore, in this modified example, when the presenter 80, for example, moves the mouse cursor 550 to the position corresponding to the cursor 560 to explain again the content at the position corresponding to the cursor 560, or gives an explanation corresponding to the position of the cursor 560, and the position of the viewer's 90's gaze coincides with the ideal position 91 of the viewer's gaze, the display of the cursor 560 is erased.

[0118] Next, an example of information processing according to this embodiment will be described. As shown in Fig. 13B, first, the information processing system acquires information on the ideal gaze position from the mouse cursor 550 operated by the presenter 80 (step S131). Next, the information processing system acquires the relative coordinates of the gaze position of the viewer 90 (step S132).

[0119] Next, the information processing system determines whether the gaze position of the viewer 90 is within the frame of the document screen (step S133). If the information processing system determines that the gaze position of the viewer 90 is within the frame of the document screen (step S133: Yes), the information processing system proceeds to the processing of step S134. On the other hand, if the information processing system determines that the gaze position of the viewer 90 is outside the frame of the document screen (step S133: No), the information processing system proceeds to the processing of step S136.

[0120] Next, the information processing system determines whether the distance from the ideal position to the gaze position of the viewer 90 is within a predetermined threshold (step S134). If the information processing system determines that the distance from the ideal position to the gaze position of the viewer 90 is within the predetermined threshold (step S134: Yes), the information processing system proceeds to the processing of step S135. On the other hand, if the information processing system determines that the distance from the cursor 550 to the gaze position of the viewer 90 is equal to or greater than the predetermined threshold (step S134: No), the information processing system proceeds to the processing of step S137.

[0121] The information processing system then displays the gaze position of the viewer 90 using the in-frame cursor 500a (step S135). Next, the information processing system displays the gaze position of the viewer 90 using the out-frame cursor 500b (step S136). Furthermore, the information processing system displays the gaze position of the viewer 90, for example, by placing a predetermined marker in alignment with the in-frame cursor 500a (display of a warning marker) (step S137). In step S137, the display of the warning marker continues until the ideal gaze position approaches the warning marker. The information processing system then terminates processing.

[0122] It should be noted that the present embodiment and the modified examples are not limited to the forms shown in FIGS. 11A to 13B, and can be modified into various forms.

[0123] <<10. Ninth Embodiment>> Next, a ninth embodiment of the present disclosure will be described with reference to FIGS. 14A and 14B . FIG. 14A is an explanatory diagram for describing the operation of this embodiment, and FIG. 14B is a flowchart of an information processing method according to this embodiment. In the ninth embodiment of the present disclosure, when the gaze position of the viewer 90 is outside the frame of the document screen, the state estimator 212 estimates the cause of the gaze position of the viewer 90 being outside the frame of the document screen. In this embodiment, the state estimator 212 estimates the cause based on, for example, the gaze position of the viewer 90, environmental sounds around the viewer 90, operation information of the viewer 90, etc. Then, in this embodiment, the estimated cause is displayed to the presenter 80.

[0124] In this embodiment, for example, as shown in the upper part of FIG. 14A , information about the estimated cause is displayed using icons 562a to 562d. In this embodiment, for example, these icons 562a to 562d are displayed superimposed on the frame of the document screen so as not to obstruct the display of the document. In detail, icon 562a indicates a case where it is estimated that the viewer 90's gaze position is outside the frame of the document screen due to input operations on a keyboard (not shown). In this case, the state estimator 212 can estimate the cause based on, for example, information about the viewer 90's keyboard operation. Furthermore, icon 562b indicates a case where it is estimated that the viewer 90's gaze position is outside the frame of the document screen due to a visitor. In this case, the state estimator 212 can estimate the cause based on, for example, the sound of an intercom (not shown) detected as environmental sound around the viewer 90. Furthermore, icon 562c indicates a case where it is estimated that the viewer 90's gaze position is outside the frame of the document screen due to the viewer 90 being deep in thought. In this case, the state estimator 212 can estimate the cause based on, for example, the gaze position and eye movement of the viewer 90. Furthermore, icon 562d indicates a case where it is estimated that the gaze position of the viewer 90 is outside the frame of the document screen because the viewer 90 is looking at camera footage of another viewer 90. In this case, the state estimator 212 can estimate the cause based on, for example, the gaze position of the viewer 90. Note that in this embodiment, when displaying the cause, the gaze position of the viewer 90 may be displayed with an outside-frame cursor 500b, and further, the ideal gaze position of the viewer 90 may also be displayed.

[0125] Thus, according to this embodiment, the presenter 80 can grasp the detailed state of the viewer 90, and can take action according to the situation of the viewer 90 (for example, temporarily stopping the explanation), thereby enabling an effective presentation.

[0126] In addition, in this embodiment, as shown in the lower part of Figure 14A, an icon 562e may be displayed to indicate that the viewer 90 is watching the camera image of the presenter 80. With this display, the presenter 80 can respond immediately if the viewer 90 speaks to him / her.

[0127] Next, an example of information processing according to this embodiment will be described. As shown in FIG. 14B , first, the information processing system acquires the relative coordinates of the gaze position of the viewer 90 (step S141). Furthermore, the information processing system determines whether the gaze position of the viewer 90 is outside the frame of the document screen (step S142). If the information processing system determines that the gaze position of the viewer 90 is outside the frame of the document screen (step S142: Yes), it proceeds to processing of step S143. On the other hand, if the information processing system determines that the gaze position of the viewer 90 is within the frame of the document screen (step S142: No), it proceeds to processing of step S146.

[0128] Next, the information processing system determines whether the viewer 90's gaze position is in the camera images of the presenter 80 and the other viewers 90 (step S143). If the information processing system determines that the viewer 90's gaze position is in the camera images of the presenter 80 and the other viewers 90 (step S143: Yes), the information processing system proceeds to processing of step S144. On the other hand, if the information processing system determines that the viewer 90's gaze position is not in the camera images of the presenter 80 and the other viewers 90 (step S143: No), the information processing system proceeds to processing of step S145.

[0129] Next, the information processing system determines whether the viewer 90's gaze position is in the camera image of the presenter 80 (step S144). If the information processing system determines that the viewer 90's gaze position is in the camera image of the presenter 80 (step S144: Yes), the information processing system proceeds to processing of step S147. On the other hand, if the information processing system determines that the viewer 90's gaze position is not in the camera image of the presenter 80 (step S144: No), the information processing system proceeds to processing of step S148.

[0130] Next, the information processing system determines whether the cause estimated by the state estimator 212 is that the viewer 90 is thinking (step S145). If the information processing system determines that the estimated cause is that the viewer 90 is thinking (step S145: Yes), it proceeds to the processing of step S147. On the other hand, if the information processing system determines that the estimated cause is that the viewer 90 is not thinking (step S145: No), it proceeds to the processing of step S148.

[0131] Next, the information processing system selects to display the inside-frame cursor 500a (step S146). Then, the information processing system highlights the outside-frame cursor 500b (for example, blinks the outside-frame cursor 500b) and displays an icon 562 indicating the cause (step S147). For example, as shown in the lower part of FIG. 14A, an icon 562e indicating that the audience 90 is watching the camera video of the presenter 80 is displayed. Furthermore, the information processing system displays the outside-frame cursor 500b and displays the icon 562 indicating the cause (step S148). The information processing system then ends the processing.

[0132] It should be noted that this embodiment is not limited to the configurations shown in FIGS. 14A and 14B, and can be modified into various other configurations.

[0133] <<11. Tenth Embodiment>> Next, a tenth embodiment of the present disclosure will be described with reference to Fig. 15A and Fig. 15B. Fig. 15A is an explanatory diagram for describing the operation in this embodiment, and Fig. 15B is a flowchart of an information processing method according to this embodiment.

[0134] In the above-described embodiments of the present disclosure, a warning is displayed when the viewer 90's gaze position is outside the frame of the document screen. However, in such embodiments, for example, if the detection accuracy of the gaze position is not high, a warning may be erroneously displayed even when the viewer 90 is viewing the document. Therefore, the tenth embodiment of the present disclosure is an embodiment for avoiding such erroneous warning displays. In this embodiment, as shown in the upper part of FIG. 15A , when the presenter 80 is giving a presentation while pointing the cursor 550 at a location close to the frame of the document screen, even if the viewer 90's gaze position is detected to be outside the frame of the document screen, the gaze position is simply displayed by the cursor 500b, and no warning display such as blinking is performed. Furthermore, in this embodiment, as shown in the lower part of FIG. 15A , when the presenter 80 is giving a presentation while pointing the cursor 550 at a location far from the frame of the document screen, if the viewer 90's gaze position is detected to be outside the frame of the document screen, the position is displayed by blinking the cursor 500b. Note that in this embodiment, the ideal gaze position of the viewer 90 described above may be used instead of the position of the cursor 550.

[0135] Thus, according to this embodiment, even if the detection accuracy of the gaze position is not high, the presenter 80 will not be aware of a warning display that does not accurately reflect the situation of the viewer 90, and therefore can give an effective presentation.

[0136] Furthermore, while the viewer 90 is listening to the explanation by the presenter 80, the viewer 90 may sometimes position their gaze outside the frame of the document screen in order to take notes on the explanation. In such a case, even if the viewer 90's gaze is outside the frame of the document screen, it is preferable not to display a warning to the presenter 80 because the viewer 90's interest is in the document. Therefore, in this embodiment, for example, information on the viewer 90's input operation for taking notes (keyboard operation information) is acquired. Then, in this embodiment, from the information, it is determined whether the content of the note matches or is similar to the explanation based on whether it matches or is similar to the content at the position of the cursor 550 used by the presenter 80, whether it matches or is similar to the speech of the presenter 80, etc. Furthermore, in this embodiment, if it is determined that the content of the note matches or is similar to the explanation, no warning is displayed. Note that in this embodiment, instead of displaying a warning, an icon (not shown) may be displayed instructing the presenter 80 to temporarily pause the explanation to allow the viewer 90 time to take notes. As described above, according to the present embodiment, the presenter 80 can recognize that the viewer 90 is taking notes on the explanation, and can therefore take action appropriate to the viewer 90's situation (e.g., temporarily stopping the explanation, etc.). Note that, in the present embodiment, for example, input operation information of the notes acquired by the viewer operation recognition unit 214 of the viewer-side device 20 may be transmitted to the ideal gaze position estimator 304 of the presenter-side device 40. Then, in the present embodiment, for example, the ideal gaze position estimator 304 performs processing such as vectorization on the phrases (words and sentences) extracted from the input information, and generates data indicating the features of the phrases. Furthermore, in the present embodiment, for example, the ideal gaze position estimator 304 uses the generated data to determine whether the content of the notes matches or is similar to the explanation, etc., of the presenter 80.

[0137] Next, an example of information processing according to this embodiment will be described. As shown in Fig. 15B, first, the information processing system acquires the relative coordinates of the gaze position of the viewer 90 (step S151). Next, the information processing system acquires information on the ideal gaze position of the viewer 90 (step S152). Furthermore, the information processing system acquires input operation information of the viewer 90 (step S153).

[0138] Next, the information processing system determines whether the gaze position of the viewer 90 is within the frame of the document screen (step S154). If the information processing system determines that the gaze position of the viewer 90 is within the frame of the document screen (step S154: Yes), the information processing system proceeds to the processing of step S156. On the other hand, if the information processing system determines that the gaze position of the viewer 90 is outside the frame of the document screen (step S154: No), the information processing system proceeds to the processing of step S156. Then, the information processing system displays the gaze position of the viewer 90 with an in-frame cursor 500a (step S155).

[0139] Next, the information processing system determines whether the gaze position of the viewer 90 is near the ideal gaze position of the viewer 90 (step S156). Specifically, in step S156, the information processing system determines whether the distance of the gaze position of the viewer 90 from the ideal gaze position of the viewer 90 is within a predetermined threshold. If the information processing system determines that the distance of the gaze position of the viewer 90 from the ideal position is within the predetermined threshold (step S156: Yes), it proceeds to processing of step S157. On the other hand, if the information processing system determines that the distance of the gaze position of the viewer 90 from the ideal position is equal to or greater than the predetermined threshold (step S156: No), it proceeds to processing of step S158. Then, the information processing system displays the gaze position of the viewer 90 with the outside-frame cursor 500b (step S157).

[0140] Next, the information processing system determines whether the input content of the viewer 90 is close (matches or is similar) to the content of the materials or explanations based on the information on the input operation of the viewer 90 (step S158). If the information processing system determines that the input content of the viewer 90 matches or is similar to the content of the materials or explanations (step S158: Yes), it proceeds to processing of step S157. On the other hand, if the information processing system determines that the input content of the viewer 90 does not match or is not similar to the content of the materials or explanations (step S158: No), it proceeds to processing of step S159. Then, the information processing system displays the position of the viewer 90's gaze by blinking the in-frame cursor 500a (warning display) (step S159). The information processing system then ends processing.

[0141] It should be noted that this embodiment is not limited to the configurations shown in FIGS. 15A and 15B, and can be modified into various other configurations.

[0142] 12. Eleventh Embodiment Next, an eleventh embodiment of the present disclosure will be described with reference to Fig. 16A to Fig. 16C. Fig. 16A is an explanatory diagram for describing the operation in this embodiment, and Fig. 16B and Fig. 16C are flowcharts of an information processing method according to this embodiment.

[0143] However, sometimes viewers 90 who were not interested in a presentation may become interested during the presentation due to some trigger. In such a case, it is expected that a movement such as the viewer 90's gaze returning from outside the frame of the document screen to the frame is detected. Therefore, in the eleventh embodiment of the present disclosure, as shown in FIG. 16A , when the viewer 90's gaze returns from outside the frame of the document screen to the frame, the content of the part that the presenter 80 was explaining is displayed in a column 990 or the like. In this way, according to this embodiment, the presenter 80 can grasp the content that the viewer 90 was interested in. Note that, in this embodiment, the content of the part that the presenter 80 was explaining can be identified using the same method as that for estimating the ideal gaze position of the viewer 90 in the above-described embodiment. In detail, for example, the ideal gaze position estimator 304 estimates the ideal gaze position of the viewer 90 based on the gaze position of the presenter 80, the speech of the presenter 80, operation information of the presenter 80 (the position of the cursor pointing to a part of the document), etc. Therefore, in this embodiment, the content of the part that the presenter 80 was explaining can be identified based on the ideal position when the viewer's 90 line of sight returns from outside the frame of the document screen to inside the frame.

[0144] According to this embodiment, the presenter 80 can grasp the content that sparked the interest of the audience 90 during the presentation, and based on this understanding, can take action to make the presentation more effective and devise materials and explanations.

[0145] Next, an example of information processing according to this embodiment will be described. As shown in FIG. 16B, first, the information processing system acquires the relative coordinates of the gaze position of the viewer 90 (step S1161). Next, the information processing system acquires input operation information of the presenter 80 (step S1162). Then, the information processing system acquires information on the gaze position of the presenter 80 (step S1163). Furthermore, the information processing system acquires information on the speech of the presenter 80 (step S1164).

[0146] Next, the information processing system determines whether the gaze position of the viewer 90 is within the frame of the document screen (step S1165). If the information processing system determines that the gaze position of the viewer 90 is within the frame of the document screen (step S1165: Yes), the information processing system proceeds to the processing of step S1166. On the other hand, if the information processing system determines that the gaze position of the viewer 90 is outside the frame of the document screen (step S1165: No), the information processing system proceeds to the processing of step S1161.

[0147] Next, the information processing system determines whether the presenter 80 is moving the mouse while giving the explanation (step S1166). If the information processing system determines that the presenter 80 is moving the mouse while giving the explanation (step S1166: Yes), the process proceeds to step S1167. On the other hand, if the information processing system determines that the presenter 80 is not moving the mouse while giving the explanation (step S1166: No), the process proceeds to step S1168. Then, the information processing system analyzes what is written in the part of the document that the mouse is pointing at (step S1167).

[0148] Next, the information processing system determines whether the presenter 80 is following the explanation portion of the material with his / her eyes while giving his / her explanation (step S1168). If the information processing system determines that the presenter 80 is following the explanation portion with his / her eyes while giving his / her explanation (step S1168: Yes), the information processing system proceeds to the processing of step S1169. On the other hand, if the information processing system determines that the presenter 80 is not following the explanation portion with his / her eyes while giving his / her explanation (step S1168: No), the information processing system proceeds to the processing of step S1170. Then, the information processing system analyzes what is written in the portion of the material that corresponds to the position of the presenter 80's gaze (step S1169).

[0149] Next, the information processing system analyzes what the presenter 80 is explaining based on the speech of the presenter 80 (step S1170). Then, the information processing system displays the analysis result as information indicating what the viewer 90 is interested in (step S1171). The information processing system then ends the process.

[0150] FIG. 16C is a flowchart showing information processing according to a modified example of this embodiment. In this modified example, information on the time when the viewer 90's gaze returns from outside the frame of the document screen to inside the frame is acquired, and the time is stored together with the content in which the viewer 90 was interested. By using this time, for example, by comparing it with video and audio data of the presenter 80 captured during the presentation, it is possible to understand the explanatory audio and actions of the presenter 80 that the viewer 90 was interested in. Note that the flowchart of FIG. 16C is the same as FIG. 16B except that information on the time when the viewer 90's gaze returns from outside the frame of the document screen to inside the frame is acquired, and the time is stored together with the content in which the viewer 90 was interested, and therefore a description thereof will be omitted.

[0151] It should be noted that this embodiment is not limited to the configurations shown in FIGS. 16A to 16C, and can be modified into various other configurations.

[0152] <<13. Twelfth Embodiment>> Next, a twelfth embodiment of the present disclosure will be described with reference to Fig. 17A and Fig. 17B. Fig. 17A is an explanatory diagram for describing the operation in this embodiment, and Fig. 17B is a flowchart of an information processing method according to this embodiment. The twelfth embodiment of the present disclosure is an embodiment regarding calibration of detection of the gaze position of the viewer 90.

[0153] Typically, calibration for detecting the gaze position is performed before starting gaze detection by having the viewer 90 sequentially gaze at nine points, for example, the center, four corners, the center of the top and bottom edges of the viewer-side display unit 200 of the viewer-side device 20, and the centers of the right and left edges of the viewer-side display unit 200. By correlating the detected facial orientation, position, and eyeball (pupil) orientation of the viewer 90 gazing at each point with the position of each store, calibration for detecting the viewer's 90 gaze can be performed.

[0154] However, if the viewer 90 does not gaze at the correctly specified points during such calibration, the calibration cannot be performed properly. Even if the calibration can be performed properly, the estimated gaze position may deviate from the correct position over time. Therefore, in the twelfth embodiment of the present disclosure, calibration is performed sequentially while the viewer-side device 20 is being used (running calibration).

[0155] In this embodiment, as shown in Fig. 17A , when the viewer 90 moves the mouse cursor 500 on the viewer-side device 20, it is estimated that the viewer 90 is looking at the mouse cursor 550. In this embodiment, calibration is performed by matching the facial orientation, position, and eyeball (pupil) orientation of the viewer 90 gazing at the mouse cursor 550 with the position of the mouse cursor 550. Note that in this embodiment, calibration may be performed by determining the reference position of the mouse cursor 550 using a speech such as "It's here" from the viewer 90 using the viewer-side device 20.

[0156] As described above, according to this embodiment, calibration can be performed successively without any special operation while the viewer-side device 20 is being used. As a result, according to this embodiment, not only can calibration be performed successively without imposing a burden on the viewer 90, but the position of the viewer's 90's line of sight can also be detected accurately and continuously.

[0157] Next, an example of information processing according to this embodiment will be described. As shown in Fig. 17B, first, the information processing system acquires the relative coordinates of the gaze position of the viewer 90 (step S171). Next, the information processing system acquires input operation information of the viewer 90 (step S172).

[0158] Next, the information processing system determines whether the viewer 90 is moving the mouse (mouse cursor 550) (step S173). If the information processing system determines that the viewer 90 is moving the mouse (step S173: Yes), it proceeds to step S174. On the other hand, if the information processing system determines that the viewer 90 is not moving the mouse (step S173: No), it proceeds to step S173. Then, the information processing system estimates that the current face position, orientation, and eye direction of the viewer 90 indicate that the viewer 90 is looking at the mouse cursor 550, and performs calibration (step S174). The information processing system then ends the processing. Note that in this embodiment, the operation is not limited to mouse operation, and any operation to move a cursor, such as a controller or pen, may be used. Note that in this embodiment, the cursor may be in the form of a mouse cursor or a pointer used in presentations.

[0159] It should be noted that this embodiment is not limited to the configurations shown in FIGS. 17A and 17B, and can be modified into various other configurations.

[0160] <<14. Thirteenth Embodiment>> Next, a thirteenth embodiment of the present disclosure will be described with reference to Fig. 18. Fig. 18 is an explanatory diagram for describing the operation in this embodiment.

[0161] In a thirteenth embodiment of the present disclosure, a presenter 80 shares slides of materials with an audience 90 during the explanation, and the audience 90 is able to view any slide of the materials they like (for example, using the live function of PowerPoint (registered trademark)). In this embodiment, by utilizing each of the above-described embodiments of the present disclosure, information on which part of which slide of the materials the audience 90 is looking at can be obtained. Therefore, in this embodiment, the position of which slide of which part of the materials the audience 90 is looking at is displayed to the presenter 80.

[0162] For example, the example shown in the upper part of Fig. 18 indicates that the viewer 90's gaze position is on the "ABC block" shown on the slide 800b immediately preceding the slide 800a of the material being explained by the presenter 80. Furthermore, in this embodiment, if the viewer 90's gaze position is on the slide 800b immediately preceding the slide 800a of the material being explained by the presenter 80, it may be inferred that the viewer 90's gaze position is on the previous slide 800b because the viewer 90 is unable to understand the explanation. Then, in this embodiment, based on this inference, a message such as "It may be better to provide additional explanation" may be displayed to prompt the presenter 80 to provide additional explanation.

[0163] Furthermore, in this embodiment, if the viewer 90's gaze position is on slide 800c following slide 800a of the material being explained by the presenter 80, it may be inferred that the viewer 90 is tired of the explanation and therefore the viewer 90's gaze position is on the subsequent slide 800c. Based on this inference, in this embodiment, the presenter 80 may be prompted to explain the previous slide 800c using a phrase such as "It might be better to explain the previous slide."

[0164] In this manner, in this embodiment, it is possible to easily grasp which slide 800 the audience 90 is paying attention to, allowing the presenter 80 to give a more effective presentation.

[0165] Furthermore, in this embodiment, there may be a plurality of viewers 90. Therefore, in this embodiment, as shown in the lower part of Fig. 18 , the presenter 80 may be informed of which slides 800 in the materials the gaze positions of the plurality of viewers 90 are on. For example, a display 802 included in the example in the lower part of Fig. 18 uses gradation or the like to indicate which slides 800 the gaze positions of the plurality of viewers 90 are on. In detail, the display 802 is displayed so that the color of a slide 800 that includes more gaze positions of viewers 90 is darker.

[0166] 18 shows, by a heat map or the like, which slides 800 are receiving the gazes of the multiple viewers 90. In detail, the display 804 is displayed in a predetermined color tone in accordance with the location where the gazes of the viewers 90 are more frequently received.

[0167] In this manner, in this embodiment, the presenter 80 can grasp the level of understanding of the multiple viewers 90, the areas where explanation seems insufficient, and the areas where more explanation is desired, and can adjust the content and pace of the explanation accordingly. Furthermore, by applying this embodiment to, for example, a product sales video, the products that viewers 90 are interested in become clear, and this can be used for analyzing market forecasts of product sales. Furthermore, by applying this embodiment to, for example, a live performance video, the performances that viewers 90 are interested in become clear, and this can be used to create more captivating performances.

[0168] Furthermore, in this embodiment, not only can the gaze positions of multiple viewers 90 be displayed simultaneously, but the gaze positions of a specific one or more viewers 90 can also be displayed according to the settings of the presenter 80. In this way, according to this embodiment, the presenter 80 can grasp information about each viewer 90, and can therefore provide explanations that are tailored to each viewer's pace and level of understanding. For example, according to this embodiment, the presenter 80 can provide more appropriate feedback to the entire audience 90 or to each individual viewer 90. In other words, this embodiment can be applied, for example, to remote classes, etc.

[0169] It should be noted that this embodiment is not limited to the configuration shown in FIG. 18, and can be modified into various other configurations.

[0170] <<15. Fourteenth Embodiment>> Next, a fourteenth embodiment of the present disclosure will be described with reference to FIG. 19 . FIG. 19 is an explanatory diagram illustrating the operation of this embodiment. As described above, by utilizing each embodiment of the present disclosure, it is possible to grasp in real time the location (ideal position) that the presenter 80 wants the audience to look at. Therefore, in the fourteenth embodiment of the present disclosure, a predetermined effect is automatically applied to an image of a document or the like based on the grasped location (ideal position) that the presenter 80 wants the audience to look at, thereby creating a document or the like for a more effective presentation. In this embodiment, for example, such an effect can be applied by the drawing unit 306 of the presenter-side device 40. Note that, in this embodiment, the location that the presenter 80 wants the audience 90 to look at can also be identified using the same method as that for estimating the ideal gaze position of the audience 90 in the above-described embodiment. In detail, for example, the ideal gaze position estimator 304 estimates the ideal gaze position of the audience 90 based on the gaze position of the presenter 80, the speech of the presenter 80, operation information of the presenter 80 (the position of the cursor pointing to a part of the document), etc. Therefore, in this embodiment, the part that the presenter 80 wants the audience 90 to look at is specified based on the ideal position of the audience's line of sight.

[0171] In detail, in this embodiment, when a desired location (ideal position) is identified, the desired location may be automatically surrounded by a frame 900 or may be blinking, as shown in the upper part of Fig. 19. Also, in this embodiment, an image 902 of the desired location may be automatically zoomed in, as shown in the middle part of Fig. 19, to force the viewer 90 to see the image 902. Also, in this embodiment, a window 904 of the desired location may be automatically moved to the center, as shown in the lower part of Fig. 19, to force the viewer 90 to see the window 904 (camera switching).

[0172] As described above, by utilizing each embodiment of the present disclosure, it is possible to grasp in real time the parts that the presenter 80 wants viewers 90 to see. Therefore, in this embodiment, for example, when distributing already produced content, by adding effects to the content using information about the parts that the presenter 80 wants viewers 90 to see, the content can be made to attract more attention from the viewers 90. Therefore, in this embodiment, effects are added automatically in real time, so that the distributor (presenter 80) does not need to take the time to re-edit the content, and the distributor can concentrate on the distribution. This embodiment can be applied, for example, to the distribution of live video in which the distributor makes a presentation as the presenter 80 while simultaneously broadcasting.

[0173] It should be noted that this embodiment is not limited to the configuration shown in FIG. 19, and can be modified into various other configurations.

[0174] <<16. Summary>> As described above, in an embodiment of the present disclosure, the presenter 80 can easily grasp whether the viewer 90 is looking at the material, and further, what part of the material the viewer 90 is looking at. Furthermore, in this embodiment, the ideal gaze position of the viewer 90 can be estimated in real time, and by combining this with information on the gaze position of the viewer 90, useful information for the presenter 80 to give an effective presentation can be provided to the presenter 80. In other words, the present disclosure provides a technology that can make the communication of information provided by a system for sharing materials, audio, and the like between remote locations more effective.

[0175] Note that the above has described the embodiments of the present disclosure using examples in which they are mainly applied to a remote conference room system, etc. However, the embodiments of the present disclosure are not limited to application to such remote conference room systems, and may be applied to, for example, remote classes, business negotiations, remote consulting, interviews, sales, various performances (e.g., live performances such as plays and comedy), or to the creation of various videos (e.g., movies, commercial messages, promotional videos for companies and individuals, educational videos, cooking videos, how-to videos, web-based tests, etc.), analysis of videos during distribution (product sales videos, news videos, etc.), etc.

[0176] <<17. Hardware Configuration Example>> The remote conference system 10, the viewer-side device 20, the presenter-side device 40, and the like according to the embodiments of the present disclosure described above are realized by a computer 1000 configured as shown in FIG. 19, for example. FIG. 19 is a hardware configuration diagram showing an example of a computer that realizes the functions of each device according to the embodiments of the present disclosure. The computer 1000 includes a CPU (Central Processing Unit) 1100, a RAM (Random Access Memory) 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0177] The CPU 1100 operates based on programs stored in the ROM 1300 or the HDD 1400 and controls each unit. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs. Alternatively, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array) may be used instead of the CPU 1100. For example, the CPU 1100 can function as a viewer line-of-sight estimator 202, a viewer relative coordinate acquisition unit 204, a viewer line-of-sight display control unit 210, a state estimator 212, a viewer operation recognition unit 214, a calibration unit 216, etc. of the viewer-side device 20. Furthermore, for example, the CPU 1100 can function as a viewer gaze visualization unit 302, an ideal gaze position estimator 304, a drawing unit 306, a presenter shared material recognition unit 308, a presenter voice recognition unit 310, a presenter operation recognition unit 312, a presenter gaze estimator 314, etc. of the presenter side device 40.

[0178] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 is started, as well as programs that depend on the hardware of the computer 1000 .

[0179] HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 is a recording medium that records an information processing program according to the present disclosure, which is an example of program data 1450.

[0180] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0181] The input / output interface 1600 is an interface for connecting the input / output devices 1650 and the computer 1000. For example, the CPU 1100 receives data from the input / output devices 1650, such as a keyboard, mouse, microphone, camera, and various sensors, via the input / output interface 1600. The CPU 1100 also transmits data to output devices, such as a display, speaker, and printer, via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined recording medium. Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Discs), magneto-optical recording media such as MOs (Magneto-Optical Discs), tape media, magnetic recording media, and semiconductor memories. For example, the input / output interface 1600 can function as the viewer-side display unit 200, the audio output unit 220, etc. of the viewer-side device 20. Also, for example, the input / output interface 1600 can function as the presenter-side display unit 300, etc. of the presenter-side device 40.

[0182] For example, when the computer 1000 functions as the remote conference system 10, the viewer device 20, the presenter device 40, or the like according to an embodiment of the present disclosure, the CPU 1100 of the computer 1000 executes programs stored in the RAM 1200 to realize each function. Furthermore, the HDD 1400 stores processing programs and the like according to the present disclosure. The CPU 1100 reads and executes program data 1450 from the HDD 1400, but as another example, the CPU 1100 may obtain these programs from another device via an external network 1550.

[0183] Furthermore, the remote conference system 10, viewer-side device 20, presenter-side device 40, etc. according to the present embodiment may be applied to a system consisting of multiple devices that assumes connection to a network (or communication between devices), such as cloud computing, etc. In other words, the remote conference system 10, viewer-side device 20, and presenter-side device 40 according to the present embodiment described above can also be realized as an information processing system according to the present embodiment using, for example, multiple devices.

[0184] The above describes an example of the hardware configuration of the remote conference system 10, the viewer device 20, the presenter device 40, etc. Each of the above components may be configured using general-purpose components, or may be configured using hardware specialized for the function of each component. Such a configuration may be appropriately changed depending on the technical level at the time of implementation.

[0185] <<18. Supplementary Information>> An embodiment of the present disclosure may include, for example, a program that causes an information processing system such as that described above to execute an information processing method according to an embodiment of the present disclosure, and a non-transitory tangible medium on which the program is recorded. Furthermore, the program may be distributed via a communication line (including wireless communication) such as the Internet.

[0186] Furthermore, the components of each device shown in the figure are conceptual functional components and are not limited to being physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them may be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0187] Furthermore, the steps in the information processing method of the above-described embodiment of the present disclosure do not necessarily have to be processed in the order described. For example, the steps may be processed in a different order as appropriate. Furthermore, instead of being processed chronologically, the steps may be partially processed in parallel or individually. Furthermore, the processing of each step does not necessarily have to be performed in the manner described, and may be performed by another functional unit using another method.

[0188] Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical idea described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0189] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0190] The present technology can also be configured as follows. (1) An information processing system comprising: a gaze detection unit that detects a gaze position of a second user with respect to a second user terminal used by a second user that shares a presentation screen displayed on a first user terminal used by the first user; a relative coordinate conversion unit that calculates the gaze position of the second user as relative coordinates with respect to the presentation screen; and a display control unit that displays second user information on the first user terminal based on the relative coordinates of the gaze position of the second user. (2) The information processing system described in (1) above, wherein the relative coordinate conversion unit calculates the relative coordinates based on the coordinates of the gaze position of the second user with respect to a display unit of the second user terminal and the coordinates of an area of ​​the presentation screen on the display unit of the second user terminal. (3) The information processing system described in (1) or (2) above, wherein the display control unit displays the second user information so as to be superimposed on at least a portion of the presentation screen displayed on the first user terminal. (4) The information processing system according to (3), wherein the display control unit displays the gaze position of the second user. (5) The information processing system according to (4), wherein the display control unit, when the gaze position of the second user is within the presentation screen, displays the gaze position of the second user in a first display mode, and when the gaze position of the second user is outside the presentation screen, displays the gaze position of the second user in a second display mode different from the first display mode. (6) The information processing system according to (5), wherein the display control unit divides the presentation screen into a plurality of cells, displays the periphery of the cell including the gaze position of the second user as the first display mode, and displays a line segment constituting part of the periphery of the cell adjacent to the gaze position of the second user as the second display mode.(7) The information processing system according to (4) above, wherein the display control unit displays information regarding the gaze position of the second user in a first display mode when the gaze position of the second user is within the presentation screen and when the gaze position of the second user is outside the presentation screen and within a predetermined range from the presentation screen, and displays information regarding the gaze position of the second user in a second display mode different from the first display mode when the gaze position of the second user is outside the presentation screen and outside a predetermined range from the presentation screen. (8) The information processing system according to (3) above, further comprising an ideal gaze position estimator that estimates an ideal position of the gaze of the second user with respect to the presentation screen, and wherein the display control unit changes the display mode of the gaze position of the second user depending on a positional relationship between the gaze position of the second user and the ideal position. (9) The information processing system according to (8), wherein the ideal gaze position estimator estimates the ideal position based on at least one of the gaze position of the first user, a speech of the first user, and operation information of the first user. (10) The information processing system according to (8) or (9), wherein the display control unit displays the gaze position of the second user in a first display mode when the gaze position of the second user is within the presentation screen and a distance between the gaze position of the second user and the ideal position is within a predetermined distance, displays the gaze position of the second user in a second display mode different from the first display mode when the gaze position of the second user is outside the presentation screen, and displays the gaze position of the second user in a third display mode different from the first and second display modes when the gaze position of the second user is within the presentation screen and a distance between the gaze position of the second user and the ideal position is equal to or greater than a predetermined distance. (11) The information processing system according to any one of (8) to (10), wherein, when the display control unit detects that the second user's line of sight has moved from outside the presentation screen to inside the presentation screen, the display control unit identifies and displays the content presented by the first user based on the ideal position.(12) The information processing system according to (8), further comprising a cause estimator that, when the second user's gaze position is outside the presentation screen, estimates a cause of the second user's gaze position being outside the presentation screen, and the display control unit displays information related to the estimated cause. (13) The information processing system according to (12), wherein the cause estimator estimates a cause of the second user's gaze position being outside the presentation screen based on at least one of the second user's gaze position, environmental sound around the second user, and operation information of the second user. (14) The information processing system according to (12), wherein, when the cause of the second user's gaze position being outside the presentation screen is that the second user is viewing video of the first user, the display control unit displays information to the first user that the second user is viewing video of the first user. (15) The information processing system according to (12), wherein, when the reason why the gaze position of the second user is outside the presentation screen is because the second user is performing an input operation, the display control unit displays information to the first user that the second user is performing the input operation. (16) The information processing system according to (15), wherein, when information input to the second user terminal by the input operation of the second user matches or is similar to information specified based on the ideal position, the display control unit does not display information to the first user that the second user is performing the input operation. (17) The information processing system according to (13), further comprising a calibration unit that calibrates the gaze detection unit based on the gaze position of the second user and operation information of the second user on the second user terminal.(18) An information processing device comprising: a gaze detection unit that detects a gaze position of a second user with respect to a second user terminal used by a second user that shares a presentation screen displayed on a first user terminal used by the first user, the second user terminal being used by a second user; a relative coordinate conversion unit that calculates the gaze position of the second user as relative coordinates with respect to the presentation screen, and a display control unit that displays second user information based on the relative coordinates of the gaze position of the second user on the first user terminal. (19) An information processing method, including: an information processing device that detects the gaze position of the second user with respect to a second user terminal used by a second user that shares a presentation screen displayed on a first user terminal used by the first user, calculates the gaze position of the second user as relative coordinates with respect to the presentation screen, and displays second user information based on the relative coordinates of the gaze position of the second user on the first user terminal. (20) A program for causing a computer to function as: a gaze detection unit that detects the gaze position of a second user with respect to a second user terminal used by a second user that shares a presentation screen displayed on a first user terminal used by the first user; a relative coordinate conversion unit that calculates the gaze position of the second user as relative coordinates with respect to the presentation screen; and a display control unit that displays second user information based on the relative coordinates of the gaze position of the second user on the first user terminal.

[0191] 10 Remote conference system 20 Viewer-side device 40 Presenter-side device 80 Presenter 90 Viewer 91 Ideal position 92, 92a, 92b, 600 Gaze position 94, 96, 98a, 98b Change over time 100 Presenter screen sharing mechanism 200 Viewer-side display unit 202 Viewer gaze estimator 204 Viewer relative coordinate acquisition unit 210 Viewer gaze display control unit 212 State estimator 214 Viewer operation recognition unit 216 Calibration unit 220 Audio output unit 300 Presenter-side display unit 302 Viewer gaze visualization unit 304 Ideal gaze position estimator 306 Drawing unit 308 Presenter shared material recognition unit 310 Presenter voice recognition unit 312 Presenter operation recognition unit 314 Presenter gaze estimator 350 Grid 352 Cell 354 Line segment 400a, 400b Document screen 500, 500a, 500b, 550, 560, 960 Cursor 562, 562a, 562b, 562c, 562d, 562e Icon 700, 702 Button 750a, 750b, 904 Window 800, 800a, 800b, 800c Slide 802, 804 Display 900 Frame 902 Image 950 Video 980, 990 Column

Claims

1. An information processing system comprising: a gaze detection unit that detects the gaze position of a second user with respect to a second user terminal used by a second user that shares a presentation screen displayed on a first user terminal used by the first user; a relative coordinate conversion unit that calculates the gaze position of the second user as relative coordinates with respect to the presentation screen; and a display control unit that displays second user information based on the relative coordinates of the gaze position of the second user on the first user terminal.

2. The information processing system described in claim 1, wherein the relative coordinate conversion unit calculates the relative coordinates based on the coordinates of the second user's gaze position relative to the display unit of the second user terminal and the coordinates of the area of ​​the presentation screen on the display unit of the second user terminal.

3. The information processing system according to claim 1, wherein the display control unit displays the second user information so as to superimpose it on at least a portion of the presentation screen displayed on the first user terminal.

4. The information processing system according to claim 3, wherein the display control unit displays the gaze position of the second user.

5. The information processing system of claim 4, wherein the display control unit displays the second user's gaze position in a first display mode when the second user's gaze position is within the presentation screen, and displays the second user's gaze position in a second display mode different from the first display mode when the second user's gaze position is outside the presentation screen.

6. The information processing system of claim 5, wherein the display control unit divides the presentation screen into a plurality of cells, displays the periphery of the cell including the gaze position of the second user as the first display mode, and displays a line segment that forms part of the periphery of the cell adjacent to the gaze position of the second user as the second display mode.

7. The information processing system of claim 4, wherein the display control unit displays information regarding the second user's gaze position in a first display mode when the second user's gaze position is within the presentation screen, and when the second user's gaze position is outside the presentation screen and within a predetermined range from the presentation screen, and displays information regarding the second user's gaze position in a second display mode different from the first display mode when the second user's gaze position is outside the presentation screen and outside a predetermined range from the presentation screen.

8. An information processing system as described in claim 3, further comprising an ideal gaze position estimator that estimates the ideal position of the gaze of the second user relative to the presentation screen, and wherein the display control unit changes the display mode of the gaze position of the second user depending on the positional relationship between the gaze position of the second user and the ideal position.

9. The information processing system according to claim 8, wherein the ideal gaze position estimator estimates the ideal position based on at least one of the gaze position of the first user, the speech of the first user, and operation information of the first user.

10. The information processing system of claim 8, wherein the display control unit: displays the gaze position of the second user in a first display mode when the gaze position of the second user is within the presentation screen and the distance between the gaze position of the second user and the ideal position is within a predetermined distance; displays the gaze position of the second user in a second display mode different from the first display mode when the gaze position of the second user is outside the presentation screen; and displays the gaze position of the second user in a third display mode different from the first and second display modes when the gaze position of the second user is within the presentation screen and the distance between the gaze position of the second user and the ideal position is equal to or greater than a predetermined distance.

11. An information processing system as described in claim 8, wherein when the display control unit detects that the second user's line of sight has moved from outside the presentation screen to inside the presentation screen, the display control unit identifies and displays the content presented by the first user based on the ideal position.

12. The information processing system of claim 8, further comprising a cause estimator that, when the second user's gaze position is outside the presentation screen, estimates the cause of the second user's gaze position being outside the presentation screen, and the display control unit displays information regarding the estimated cause.

13. The information processing system of claim 12, wherein the cause estimator estimates the cause of the second user's gaze position being outside the presentation screen based on at least one of the second user's gaze position, environmental sounds around the second user, and operation information of the second user.

14. An information processing system as described in claim 12, wherein, if the reason the second user's gaze position is outside the presentation screen is that the second user is looking at the video of the first user, the display control unit displays information to the first user that the second user is looking at the video of the first user.

15. An information processing system as described in claim 12, wherein, if the reason the second user's gaze position is outside the presentation screen is that the second user is performing an input operation, the display control unit displays information to the first user that the second user is performing the input operation.

16. An information processing system as described in claim 15, wherein, when information input to the second user terminal by the second user's input operation matches or is similar to information identified based on the ideal position, the display control unit does not display information to the first user that the second user is performing the input operation.

17. An information processing system as described in claim 13, further comprising a calibration unit that calibrates the gaze detection unit based on the gaze position of the second user and operation information of the second user on the second user terminal.

18. An information processing device comprising: a gaze detection unit that detects the gaze position of a second user with respect to a second user terminal used by a second user that shares a presentation screen displayed on a first user terminal used by the first user; a relative coordinate conversion unit that calculates the gaze position of the second user as relative coordinates with respect to the presentation screen; and a display control unit that displays second user information based on the relative coordinates of the gaze position of the second user on the first user terminal.

19. An information processing method, comprising: an information processing device detecting a gaze position of a second user relative to a second user terminal used by a second user that shares a presentation screen displayed on a first user terminal used by the first user; calculating the gaze position of the second user as relative coordinates with respect to the presentation screen; and displaying second user information based on the relative coordinates of the gaze position of the second user on the first user terminal.

20. A program for causing a computer to function as: a gaze detection unit that detects the gaze position of a second user with respect to a second user terminal used by a second user that shares a presentation screen displayed on a first user terminal used by the first user; a relative coordinate conversion unit that calculates the gaze position of the second user as relative coordinates with respect to the presentation screen; and a display control unit that displays second user information based on the relative coordinates of the gaze position of the second user on the first user terminal.

Images