Information processing system, information processing apparatus, information processing method, and program

Abstract

Provided is an information processing system including: a line-of-sight detecting unit that detects a line-of-sight position of a second user with respect to a second user side terminal used by the second user, the second user side terminal sharing a presentation screen displayed on a first user side terminal used by a first user; a relative coordinate converting unit that calculates the line-of-sight position of the second user as a relative coordinate with respect to the presentation screen; and a display control unit that displays second user information on the first user side terminal based on the relative coordinate of the line-of-sight position of the second user.

Description

Technical Field

[0001] This disclosure relates to information processing systems, information processing devices, information processing methods, and procedures. Background Technology

[0002] In recent years, remote conferencing systems that enable the sharing of data, audio, and other information between remote locations have become widely used. By using such remote conference room systems, people in remote locations can easily hold meetings without having to travel long distances.

[0003] Reference List

[0004] Patent documents

[0005] PTL 1: JP 2014-50018 A Summary of the Invention

[0006] Technical issues

[0007] However, for example, in remote meetings, even if materials can be shared between remote locations, presenters have difficulty knowing whether viewers at remote locations are actually viewing the materials, and further, which part of the materials they are viewing. Therefore, because presenters cannot monitor viewers in real time, they cannot provide explanations based on viewers' interests and understanding, and it is difficult to effectively conduct presentations (transmit information).

[0008] Therefore, this disclosure provides information processing systems, information processing apparatuses, information processing methods and programs that enable more efficient information transmission provided by systems that share data, audio, etc., between remote locations.

[0009] Solution to the problem

[0010] According to this disclosure, an information processing system is provided, comprising: a gaze detection unit configured to detect the position of a second user's gaze relative to a second user-side terminal used by the second user, the second user-side terminal sharing a demonstration screen displayed on a first user-side terminal used by a first user; a relative coordinate transformation unit configured to calculate the position of the second user's gaze as relative coordinates to the demonstration screen; and a display control unit configured to display second user information on the first user-side terminal based on the relative coordinates of the second user's gaze position.

[0011] Furthermore, according to this disclosure, an information processing apparatus is provided, comprising: a gaze detection unit configured to detect the position of a second user's gaze relative to a second user-side terminal used by the second user, the second user-side terminal sharing a demonstration screen displayed on a first user-side terminal used by a first user; a relative coordinate transformation unit configured to calculate the position of the second user's gaze as relative coordinates to the demonstration screen; and a display control unit configured to display second user information on the first user-side terminal based on the relative coordinates of the second user's gaze position.

[0012] Furthermore, according to this disclosure, an information processing method is provided, comprising: detecting, by an information processing device, the position of a second user's gaze relative to a second user-side terminal used by the second user, the second user-side terminal sharing a demonstration screen displayed on a first user-side terminal used by a first user; calculating, by the information processing device, the position of the second user's gaze as relative coordinates to the demonstration screen; and displaying second user information on the first user-side terminal by the information processing device based on the relative coordinates of the second user's gaze position.

[0013] Furthermore, according to this disclosure, a program is provided for enabling a computer to function as: a gaze detection unit configured to detect the position of a second user's gaze relative to a second user-side terminal used by the second user, the second user-side terminal sharing a presentation screen displayed on a first user-side terminal used by a first user; a relative coordinate transformation unit configured to calculate the position of the second user's gaze as relative coordinates to the presentation screen; and a display control unit configured to display second user information on the first user-side terminal based on the relative coordinates of the second user's gaze position. Attached Figure Description

[0014] [ Figure 1 ]

[0015] Figure 1 This is a functional block diagram of an information processing system according to a first embodiment of the present disclosure.

[0016] [ Figure 2 ]

[0017] Figure 2 This is an explanatory diagram illustrating the operation of the viewer relative coordinate acquisition unit according to the first embodiment of this disclosure.

[0018] [ Figure 3 ]

[0019] Figure 3This is an explanatory diagram illustrating the operation of an ideal line-of-sight position estimator according to a first embodiment of the present disclosure.

[0020] [ Figure 4A ]

[0021] Figure 4A This is an explanatory diagram (part 1) used to describe the operation of the second embodiment of the present disclosure.

[0022] [ Figure 4B ]

[0023] Figure 4B This is a flowchart (part 1) of an information processing method according to a second embodiment of the present disclosure.

[0024] [ Figure 5A ]

[0025] Figure 5A This is an explanatory diagram (part 2) used to describe the operation in the second embodiment of the present disclosure.

[0026] [ Figure 5B ]

[0027] Figure 5B This is a flowchart (part 2) of an information processing method according to a second embodiment of the present disclosure.

[0028] [ Figure 6A ]

[0029] Figure 6A This is an explanatory diagram illustrating the operation of a third embodiment of the present disclosure.

[0030] [ Figure 6B ]

[0031] Figure 6B This is a flowchart of an information processing method according to a third embodiment of the present disclosure.

[0032] [ Figure 7 ]

[0033] Figure 7 This is an explanatory diagram illustrating the operation of the fourth embodiment of the present disclosure.

[0034] [ Figure 8A ]

[0035] Figure 8A This is an explanatory diagram illustrating the operation of the fifth embodiment of the present disclosure.

[0036] [ Figure 8B ]

[0037] Figure 8BThis is a flowchart of an information processing method according to the fifth embodiment of this disclosure.

[0038] [ Figure 9A ]

[0039] Figure 9A This is an explanatory diagram illustrating the operation of the sixth embodiment of the present disclosure.

[0040] [ Figure 9B ]

[0041] Figure 9B This is a flowchart of an information processing method according to the sixth embodiment of this disclosure.

[0042] [ Figure 10A ]

[0043] Figure 10A This is an explanatory diagram illustrating the operation of the seventh embodiment of the present disclosure.

[0044] [ Figure 10B ]

[0045] Figure 10B This is a flowchart of an information processing method according to the seventh embodiment of this disclosure.

[0046] [ Figure 11A ]

[0047] Figure 11A This is an explanatory diagram (part 1) used to describe the operation of the eighth embodiment of the present disclosure.

[0048] [ Figure 11B ]

[0049] Figure 11B This is a flowchart (part 1) of the information processing method according to the eighth embodiment of this disclosure.

[0050] [ Figure 12A ]

[0051] Figure 12A This is an explanatory diagram (part 2) used to describe the operation of the eighth embodiment of the present disclosure.

[0052] [ Figure 12B ]

[0053] Figure 12B This is a flowchart (part 2) of the information processing method according to the eighth embodiment of this disclosure.

[0054] [ Figure 13A ]

[0055] Figure 13AThis is an explanatory diagram (part 3) used to describe the operation of the eighth embodiment of the present disclosure.

[0056] [ Figure 13B ]

[0057] Figure 13B This is a flowchart (part 3) of the information processing method according to the eighth embodiment of this disclosure.

[0058] [ Figure 14A ]

[0059] Figure 14A This is an explanatory diagram illustrating the operation of the ninth embodiment of the present disclosure.

[0060] [ Figure 14B ]

[0061] Figure 14B This is a flowchart of an information processing method according to the ninth embodiment of this disclosure.

[0062] [ Figure 15A ]

[0063] Figure 15A This is an explanatory diagram illustrating the operation of the tenth embodiment of the present disclosure.

[0064] [ Figure 15B ]

[0065] Figure 15B This is a flowchart of an information processing method according to the tenth embodiment of the present disclosure.

[0066] [ Figure 16A ]

[0067] Figure 16A This is an explanatory diagram illustrating the operation of the eleventh embodiment of the present disclosure.

[0068] [ Figure 16B ]

[0069] Figure 16B This is a flowchart (part 1) of the information processing method according to the eleventh embodiment of this disclosure.

[0070] [ Figure 16C ]

[0071] Figure 16C This is a flowchart (part 2) of the information processing method according to the eleventh embodiment of this disclosure.

[0072] [ Figure 17A ]

[0073] Figure 17AThis is an explanatory diagram illustrating the operation of the twelfth embodiment of the present disclosure.

[0074] [ Figure 17B ]

[0075] Figure 17B This is a flowchart of an information processing method according to the twelfth embodiment of the present disclosure.

[0076] [ Figure 18 ]

[0077] Figure 18 This is an explanatory diagram illustrating the operation of the thirteenth embodiment of the present disclosure.

[0078] [ Figure 19 ]

[0079] Figure 19 This is an explanatory diagram illustrating the operation of the fourteenth embodiment of the present disclosure.

[0080] [ Figure 20 ]

[0081] Figure 20 This is a hardware configuration diagram illustrating an example of a computer that implements the functions of an apparatus according to an embodiment of the present disclosure. Detailed Implementation

[0082] In the following, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that in this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted. Furthermore, in this specification and the drawings, multiple components having substantially the same or similar functional configurations can be distinguished from each other by adding different letters after the same reference numerals. However, when it is not necessary to specifically distinguish multiple components having substantially the same or similar functional configurations, only the same reference numerals are assigned.

[0083] Note that the descriptions will be given in the following order.

[0084] 1. Background

[0085] 2. First Implementation Method

[0086] 2.1 Configuration of the Information Processing System

[0087] 2.2 Viewer-side device configuration

[0088] 2.3 Configuration of presenter-side equipment

[0089] 3. Second Implementation Method

[0090] 4. Third Implementation Method

[0091] 5. Fourth Implementation Method

[0092] 6. Fifth Implementation Method

[0093] 7. Sixth Implementation Method

[0094] 8. Seventh Implementation Method

[0095] 9. Eighth Implementation Method

[0096] 10. Ninth Implementation Method

[0097] 11. Tenth Implementation Method

[0098] 12. Eleventh Implementation Method

[0099] 13. Twelfth Implementation Method

[0100] 14. Thirteenth Implementation Method

[0101] 15. Fourteenth Implementation Method

[0102] 16. Summary

[0103] 17. Hardware Configuration Example

[0104] 18. Supplement

[0105] 1. Background

[0106] First, the background that prompted the inventors to create this disclosure will be described.

[0107] As mentioned earlier, remote conference room systems allow people in remote locations to hold meetings freely without having to travel long distances. However, even when materials and audio can be shared between remote locations, it is difficult to monitor the participants at each location. For example, when a presenter is explaining material while displaying it on a screen using a remote conferencing system, the presenter may find it difficult to determine whether viewers are viewing the material and, if so, which part of it.

[0108] Specifically, when a viewer's gaze leaves the displayed material, it is assumed that the viewer is not interested in the presentation. Furthermore, even when the viewer's gaze is within the material, if their gaze shifts away from the part the presenter is explaining, it is assumed that the viewer needs time to comprehend the content. In such cases, the presenter can effectively deliver the presentation by changing the way they explain or by taking actions to attract the viewer's attention. However, in remote meetings, because the presenter cannot grasp the viewer's viewpoint, they cannot provide explanations tailored to the viewer's interest and comprehension, making it difficult to deliver an effective presentation.

[0109] Even when using a remote conferencing system that can detect the position of a viewer's gaze, if the position of the viewer's gaze relative to the information displayed on the screen cannot be identified, it is impossible to determine whether the viewer is viewing the information, and further, which part of the information the viewer is viewing.

[0110] In view of this situation, the present inventors conceived the following embodiments of the present disclosure. In the embodiments of the present disclosure, the viewer's gaze position on the viewer's side is converted into relative coordinates relative to the area of ​​the data screen displayed on the viewer's screen. Then, in the embodiments of the present disclosure, based on the converted relative coordinates, the viewer's gaze position is displayed in the area of ​​the data displayed on the presenter's screen or superimposed on a frame indicating the area of ​​the data displayed on the presenter's screen. In this embodiment, by performing such a display, the presenter can intuitively grasp the position of the viewer's gaze relative to the data screen. Therefore, according to this embodiment, the presenter can easily grasp the position of the viewer's gaze, thereby changing the manner and pace of the explanation, and taking actions to attract the viewer's attention. Therefore, according to this embodiment, the presenter can perform the presentation effectively. Hereinafter, details of the embodiments of the present disclosure created by the present inventors will be described in sequence.

[0111] Note that the following description will illustrate embodiments of this disclosure applied to remote conference room systems. However, embodiments of this disclosure are not limited to such remote conference room systems, but can be applied to, for example, remote courses, business negotiations, consultations between remote locations, interviews, sales, etc., or to video creation, video distribution, etc.

[0112] 2. First Implementation Method

[0113] 2.1 Configuration of the Information Processing System

[0114] First, refer to Figure 1 The configuration of the information processing system according to the first embodiment of this disclosure is described. Figure 1This is a functional block diagram of the information processing system according to this embodiment.

[0115] The information processing system according to this embodiment is a system for holding remote meetings, for example, between the location of the presenter (first user) 80 and the location of the viewer (second user) 90. Note that the information processing system is not limited to remote meetings between two locations, and can also perform remote meetings between three or more locations, or it can be a case where a meeting is held at the same location using another terminal.

[0116] Specifically, such as Figure 1 As shown, the information processing system according to this embodiment includes, for example, a viewer-side device (second user-side terminal) 20 located at the location of the viewer 90, and a presenter-side device (first user-side terminal) 40 located at the location of the presenter 80. The information processing system according to this embodiment also includes a remote conferencing system 10, which has the function of sharing video (images), audio, various data, etc., between the viewer-side device 20 and the presenter-side device 40. In this embodiment, the viewer-side device 20, the remote conferencing system 10, and the presenter-side device 40 are connected to each other via a communication network (not shown). In this embodiment, any communication scheme can be applied as the communication scheme used in the communication network, regardless of whether it is wired or wireless, but it is desirable to use a communication scheme that can maintain stable operation. Note that in this embodiment, the viewer-side device 20, the remote conferencing system 10, and the presenter-side device 40 can be, for example, Figure 1 The standalone devices shown, or all devices, can be configured as an integrated device. Furthermore, in this embodiment, each of the viewer-side device 20, the remote conferencing system 10, and the presenter-side device 40 can be a single device, or can be configured as multiple devices. Hereinafter, each of the devices included in the information processing system according to this embodiment will be described sequentially.

[0117] Remote conferencing system 10

[0118] As described above, the remote conferencing system 10 has the function of sharing video, audio, various data, etc., between the viewer-side device 20 and the presenter-side device 40. The remote conferencing system 10 can be a device such as a server, desktop personal computer (PC), tablet PC, laptop PC, notebook PC, or smartphone. In this embodiment, the remote conferencing system 10 includes a presenter screen sharing mechanism 100. Specifically, for example, the presenter screen sharing mechanism 100 allows the viewer-side device 20 to share and display the screen displayed by the presenter 80 on the presenter-side device 40. Furthermore, for example, the presenter screen sharing mechanism 100 allows the presenter-side device 40 to share information indicating the state of the screen displayed on the viewer-side device 20 (e.g., multiple windows open).

[0119] Viewer-side device 20

[0120] Viewer-side device 20 can share the screen (demonstration screen) displayed by presenter 80 on presenter-side device 40 with viewer 90, and can detect the position of viewer 90's gaze and send that information to presenter-side device 40. Additionally, viewer-side device 20 can be a device such as a desktop PC, tablet PC, laptop PC, notebook PC, smartphone, or head-mounted display (HMD).

[0121] The viewer-side device 20 includes a camera device (not shown) capable of detecting the gaze of the viewer 90 as an input device. Specifically, the camera device may be, for example, a webcam device mounted on the viewer-side device 20 that captures an image of the viewer 90's face. Furthermore, the camera device is not limited to being mounted on the viewer-side device 20, and may be a device separate from the viewer-side device 20, or, for example, a camera device built into a smartphone that works in conjunction with the viewer-side device 20.

[0122] The viewer-side device 20 includes touch panels, buttons, toggle switches, joysticks, keyboards, controllers, mice, etc. (not shown) as input devices for receiving input operations from the viewer 90. The input device is not limited to being mounted on the viewer-side device 20, and can be a device separate from the viewer-side device 20, and can be, for example, the touch panel of a tablet PC that works with the viewer-side device 20. Furthermore, in this embodiment, the camera device can detect hand movements of the viewer 90 to receive input operations from the viewer 90. Alternatively, in this embodiment, an inertial measurement unit (IMU) (not shown) worn on a part of the viewer 90's body can detect hand movements of the viewer 90 to receive input operations from the viewer 90. Note that the IMU may include, for example, an accelerometer sensor as an inertial sensor for acquiring acceleration, a gyroscope sensor (angular velocity sensor) as an inertial sensor for acquiring angular velocity, etc.

[0123] Additionally, the viewer-side device 20 may include a microphone (hereinafter referred to as a microphone) as an input device for acquiring the voice or ambient sounds emitted by the viewer 90.

[0124] Furthermore, the viewer-side device 20 includes an audio output unit 220, a viewer-side display portion 200, and the like as output devices. In this embodiment, the aforementioned input and output devices may be separate from the viewer-side device 20, or they may be integrated with the viewer-side device 20. Note that a detailed configuration of the viewer-side device 20 will be described later.

[0125] Demonstrator-side device 40

[0126] The presenter-side device 40 can display a screen of the material to the presenter 80 or show the position of the viewer 90's gaze. The presenter-side device 40 can be a device such as a desktop PC, tablet PC, laptop PC, notebook PC, smartphone, or HMD.

[0127] The presenter-side device 40 may include a camera device (not shown) capable of detecting the presenter 80's gaze as an input device. Specifically, for example, the camera device may be one or more cameras mounted on the presenter 80's side. Furthermore, the presenter-side device 40 includes touch panels, buttons, toggle switches, joysticks, keyboards, controllers, mice, etc. (not shown) as input devices for receiving input operations from the presenter 80. In this embodiment, the camera device may detect hand movements of the presenter 80 to receive input operations from the viewer 80. Alternatively, in this embodiment, input operations from the presenter 80 may be received by detecting hand movements of the presenter 80 using an IMU (not shown) worn on a part of the presenter 80's body.

[0128] The presenter-side device 40 includes a microphone (not shown) as an input device for acquiring the speech emitted by the presenter 80. Furthermore, the presenter-side device 40 includes a presenter-side display portion 300 and the like as output devices. In this embodiment, the aforementioned input and output devices may be devices separate from the presenter-side device 40, or they may be devices integrated with the presenter-side device 40. Note that a detailed configuration of the presenter-side device 40 will be described later.

[0129] Note, refer to Figure 1 The configuration of the information processing system described is merely an example, and the configuration of the information processing system according to this embodiment is not limited to such an example. That is, the configuration of the information processing system according to this embodiment can be flexibly modified according to specifications and operation.

[0130] 2.2 Viewer-side device configuration

[0131] In addition, refer to Figure 1 and Figure 2 The detailed configuration of the viewer-side device 20 according to this embodiment will be described. Figure 2 This is an explanatory diagram illustrating the operation of the viewer relative coordinate acquisition unit (relative coordinate transformation unit) 204 according to this embodiment.

[0132] like Figure 1 As shown, the viewer-side device 20 mainly includes a viewer-side display section 200, a viewer gaze estimator (gaze detection unit) 202, a viewer relative coordinate acquisition unit 204, a viewer gaze display control unit 210, a state estimator (cause estimator) 212, a viewer operation recognition unit 214, a calibration unit 216, an audio output unit 220, etc. The functional units of the viewer-side device 20 will be described in sequence below.

[0133] Viewer-side display section 200

[0134] The viewer-side display portion 200 is a display device that displays video (images) to the viewer 90. The display device can be, for example, a liquid crystal display (LCD), an organic electroluminescent display (OLED), etc. Alternatively, in this embodiment, the display device can be an HMD worn on the viewer 90's head and displaying video, etc. Furthermore, in this embodiment, the display device can be a projector that projects video, etc., around the viewer 90 (e.g., onto a wall, a screen, etc.).

[0135] Viewer gaze estimator 202

[0136] Viewer gaze estimator 202 detects the position of viewer 90's gaze relative to viewer-side device 20. Specifically, viewer gaze estimator 202 analyzes images from a camera device capturing images of viewer 90 and identifies the direction of viewer 90's gaze based on the position and orientation of viewer 90's face, pupil position, pupil diameter, eye movement, etc. Then, viewer gaze estimator 202 estimates the position (gaze position) of the gaze relative to the viewer-side display portion 200 of viewer-side device 20. Note that viewer gaze estimator 202 can detect the amount of movement of the gaze position and obtain fixed information about the gaze position, such as gaze duration.

[0137] Viewer relative coordinate acquisition unit 204

[0138] The viewer relative coordinate acquisition unit 204 converts the position of the viewer 90's gaze from the viewer gaze estimator 202 into relative coordinates with respect to the image (demonstration screen) of the data, etc., displayed on the viewer-side display portion 200. For example, such as... Figure 2As shown, the viewer relative coordinate acquisition unit 204 acquires coordinates (200, 800) as coordinates within the viewer-side display portion 200, and coordinates representing the position 92a of the viewer 90's gaze relative to the viewer-side display portion 200 of the presenter-side device 40. Furthermore, the viewer relative coordinate acquisition unit 204 acquires coordinates ((0, 300), (1600, 1080)) as coordinates within the viewer-side display portion 200, and coordinates indicating the area of ​​the data screen 400a displayed on the viewer-side display portion 200 of the viewer-side device 20. Then, the viewer relative coordinate acquisition unit 204 uses the acquired coordinates to convert the position of the viewer 90's gaze into relative coordinates relative to the data screen 400a. In the example above, the relative coordinates of the viewer 90's gaze position are (0.13, 0.74).

[0139] In this embodiment, relative coordinates are sent to the presenter-side device 40, and the presenter-side device 40 can display the position of the viewer 90's gaze as a cursor 500a based on the relative coordinates. Specifically, the presenter-side device 40 can perform the display using relative coordinates to faithfully display the position 92a of the viewer 90's gaze relative to the area of ​​the data screen 400b displayed on the presenter-side display portion 300, as in the case of the viewer 90's side. As described above, even if the size of the display portion differs between the viewer 90's side and the presenter 80's side, or if the position and size of the data screen relative to the display portion differ, the position of the viewer 90's gaze can be faithfully reproduced using relative coordinates. Therefore, according to this embodiment, the presenter 80 can easily determine whether the viewer 90 is viewing data, and further, which part of the data the viewer 90 is viewing.

[0140] In this embodiment, even when the viewer 90 is not looking at the data screen 400a, as long as the position 92b of the viewer 90's gaze is within the detection range of the viewer gaze estimator 202, the position 92b of the gaze can be converted into relative coordinates with respect to the data screen 400a. In this embodiment, the presenter-side device 40 displays the position 600 of the viewer 90's gaze obtained using the relative coordinates as a cursor 500b on the outer periphery of the area of ​​the data screen 400b adjacent to position 600. In this embodiment, position 600 is displayed as cursor 500b, which is different from cursor 500a. According to this embodiment, by performing such a display, the presenter 80 can easily determine that the viewer 90's gaze is outside the frame of the data screen and grasp the approximate position of the viewer 90's gaze. According to this embodiment, by performing the display in this way, the presenter 80 can be informed whether the position of the viewer 90's gaze has been detected, without hindering the display of other information on the presenter-side device 40.

[0141] Viewer gaze display control unit 210

[0142] The viewer gaze display control unit 210 causes the viewer-side display portion 200 to display the estimation result of the viewer gaze estimator 202 of the position of the viewer's gaze 90. Note that in this embodiment, the viewer gaze display control unit 210 may be provided as an option or may be omitted.

[0143] State Estimator 212

[0144] The state estimator 212 estimates the reason why the viewer 90's gaze is outside the area (outside the frame) of the data screen. The state estimator 212 estimates the reason based on the viewer 90's gaze position, ambient sounds around the viewer 90, and the viewer 90's operational information. In this embodiment, the state estimator 212 can estimate the viewer 90's psychological state based on eye movements and estimate the reason based on the estimated psychological state. Furthermore, the state estimator 212 can estimate the viewer 90's psychological state based on sensing data from various sensors (not shown), which are installed in a wearable device worn on the viewer 90's body and detect the viewer 90's state. For example, the sensors may be blood flow sensors, sweat sensors, electroencephalogram (EEG) sensors, skin temperature sensors, etc. Note that in this embodiment, the state estimator 212 may be provided as an option or may be omitted.

[0145] Viewer operation recognition unit 214

[0146] Viewer operation recognition unit 214 recognizes input operations of viewer 90, such as input operations by viewer 90 to an input device, which may be a touch panel, button, toggle switch, joystick, keyboard, controller, or mouse. Note that in this embodiment, viewer operation recognition unit 214 may be provided as an option or may be omitted.

[0147] Calibration Unit 216

[0148] The calibration unit 216 can perform calibration of the viewer gaze estimator 202 based on the coordinates of the viewer 90's gaze position and the viewer 90's input operation information. Note that the operation of the calibration unit 216 will be described in detail later. Note that in this embodiment, the calibration unit 216 may be provided as an option or may be omitted.

[0149] Audio output unit 220

[0150] The audio output unit 220 may be, for example, a speaker that outputs audio to the viewer 90. Note that in this embodiment, the audio output unit 220 may be provided as an option or may be omitted.

[0151] In this embodiment, the viewer-side device 20 is provided with a communication unit and a storage section (not shown). The communication unit can send and receive information to and from external devices. In other words, the communication unit is a communication interface with the function of sending and receiving data, and is implemented by a communication device such as a communication antenna, a transmit / receive circuit, or a port. The storage section stores programs, information, etc., for the aforementioned functional units to perform various processes, as well as information obtained through processing. Note that the storage section is implemented by a non-volatile memory such as flash memory.

[0152] Note, refer to Figure 1 The configuration of the viewer-side device 20 described is merely an example, and the configuration of the viewer-side device 20 according to this embodiment is not limited to such an example. That is, the configuration of the viewer-side device 20 according to this embodiment can be flexibly modified according to specifications and operation.

[0153] 2.3 Configuration of presenter-side equipment

[0154] In addition, refer to Figures 1 to 3 The detailed configuration of the demonstrator-side device 40 according to this embodiment will be described. Figure 3 This is an explanatory diagram illustrating the operation of the ideal line-of-sight position estimator 304 according to this embodiment.

[0155] like Figure 1 As shown, the presenter-side device 40 mainly includes a presenter-side display section 300, a viewer gaze visualization unit (display control unit) 302, an ideal gaze position estimator 304, a drawing unit (display control unit) 306, a presenter shared data recognition unit 308, a presenter voice recognition unit 310, a presenter operation recognition unit 312, and a presenter gaze estimator 314. The functional units of the presenter-side device 40 will be described in sequence below.

[0156] Presenter-side display section 300

[0157] The presenter-side display portion 300 is a display device that displays video (images) to the presenter 80. The display device can be, for example, a liquid crystal display (LCD), an organic EL display, etc. Alternatively, in this embodiment, the display device can be an HMD worn on the presenter 80's head. Furthermore, in this embodiment, the display device can be a projector that projects video or the like around the presenter 80 (e.g., onto walls, screens, etc.).

[0158] Viewer's gaze visualization unit 302

[0159] The viewer gaze visualization unit 302 can display information about the position of the viewer 90's gaze (second user information) on the presenter-side display portion 300 based on the relative coordinates of the position of the viewer 90's gaze sent from the viewer-side device 20.

[0160] As mentioned above, in this embodiment, the viewer's gaze visualization unit 302 can display the position of the viewer 90's gaze as a cursor 500a (first display mode) based on relative coordinates, such as... Figure 2 As shown in the diagram. In this embodiment, when the viewer 90's gaze is outside the frame of the data image, the viewer gaze visualization unit 302 displays the viewer 90's gaze position 600, obtained using relative coordinates, as a cursor 500b (second display mode) on the outer periphery of the area of ​​the data screen 400b adjacent to position 600. The cursor 500b has a display mode different from that of the cursor 500a. For example, the cursor 500b differs from the cursor 500a in size, shape, color, etc. According to this embodiment, by performing such a display, the presenter 80 can easily grasp that the viewer 90's gaze is outside the frame of the data screen and grasp the approximate position of the viewer 90's gaze.

[0161] Ideal line-of-sight position estimator 304

[0162] Ideal gaze position estimator 304 estimates the ideal position of viewer 90's gaze relative to the data screen. For example, ideal gaze position estimator 304 estimates the ideal position of viewer 90's gaze based on the position of presenter 80's gaze, the voice emitted by presenter 80, and operational information of presenter 80 (e.g., the position of a cursor indicating a portion of the data). Note that in this embodiment, ideal gaze position estimator 304 may be provided as an option or may be omitted.

[0163] For example, such as Figure 3 As shown, the ideal gaze position estimator 304 performs processing such as vectorization on the wording (words or sentences) extracted from the speech of the presenter 80, which is recognized by the presenter speech recognition unit 310, described later, and generates data indicating the features of the wording. Figure 3 In the example, presenter 80 says "Speaking of apples,...", thus generating data associated with "apples" and related wording (e.g., "fruit"). Ideal gaze position estimator 304 identifies the material used by presenter 80 (output by presenter shared material recognition unit 308, described later) and extracts the content described in that material and its location. Figure 3In the example, since the image of "apple" and phrases such as "orange" and "grape" are included in the material, information about "apple," "orange," and "grape" is extracted. The ideal gaze position estimator 304 performs processing such as vectorization on the extracted information in the same manner as described above, generating data indicating the features of the phrases. Then, the ideal gaze position estimator 304 searches the material for information that matches the information from the spoken speech, or for information that has feature data similar to the feature data of the information from the spoken speech (string matching search, vector search, etc.). Then, when the information extracted from the material matches the information from the spoken speech, or when the information extracted from the material has feature data similar to the feature data of the information from the spoken speech, the ideal gaze position estimator 304 estimates the position of the information on the material as the ideal position for the viewer's gaze. For example... Figure 3 As shown, the ideal gaze position estimator 304 can estimate the position of the cursor 550 pointed to by the presenter 80 as the ideal gaze position of the viewer 90.

[0164] As described above, in this embodiment, since the ideal position of the viewer 90's gaze can be estimated in real time, by combining the information of the ideal position of the viewer 90's gaze with the information of the position of the viewer 90's gaze, useful information for effectively performing the presentation can be provided to the presenter 80. Note that the details of such an operation will be described later.

[0165] Drawing Unit 306

[0166] The drawing unit 306 causes the presenter-side display portion 300 to display, for example, information about the state of the viewer 90 that can be estimated based on the line of sight, and information about the ideal position of the estimated line of sight of the viewer 90, as superimposed on at least a portion of the displayed data screen (presentation screen).

[0167] Presenter Shared Data Recognition Unit 308

[0168] The presenter shared data identification unit 308 identifies the data used by the presenter 80. Note that in this embodiment, the presenter shared data identification unit 308 may be provided as an option or may be omitted.

[0169] Demonstrator speech recognition unit 310

[0170] The presenter speech recognition unit 310 recognizes the speech emitted by the presenter 80, which is captured by the microphone. Note that in this embodiment, the presenter speech recognition unit 310 may be provided as an option or may be omitted.

[0171] Demonstrator operation recognition unit 312

[0172] The presenter operation recognition unit 312 recognizes input operations, such as input operations by the presenter 80 to an input device, which may be a touch panel, button, toggle switch, joystick, keyboard, controller, or mouse. Note that in this embodiment, the presenter operation recognition unit 312 may be provided as an option or may be omitted.

[0173] Presenter gaze estimator 314

[0174] The presenter gaze estimator 314 analyzes images from the camera device capturing images of the presenter 80, identifies the presenter 80's gaze direction based on the position and orientation of the presenter 80's face, the position of the pupils, the size of the pupil diameter, eye movements, etc., and estimates the position (gaze position) of the gaze relative to the presenter-side display portion 300. Note that in this embodiment, the presenter gaze estimator 314 may be provided as an option or may be omitted.

[0175] In this embodiment, the presenter-side device 40 is provided with a communication unit and a storage section (not shown). The communication unit can send and receive information to and from external devices. In other words, the communication unit is a communication interface with the function of sending and receiving data, and is implemented by a communication device such as a communication antenna, a transmit / receive circuit, or a port. The storage section stores programs, information, etc., for the aforementioned functional units to perform various processes, as well as information obtained through processing. Note that the storage section is implemented by a non-volatile memory such as flash memory. Furthermore, in this embodiment, the presenter-side device 40 may be provided with an audio output unit (not shown). The audio output unit may be, for example, a speaker or headphone speaker that outputs the voice of the viewer 90 to the presenter 80.

[0176] Note, refer to Figure 1 The configuration of the presenter-side device 40 described is merely an example, and the configuration of the presenter-side device 40 according to this embodiment is not limited to such an example. That is, the configuration of the presenter-side device 40 according to this embodiment can be flexibly modified according to specifications and operation.

[0177] As described above, in this embodiment, the position of the viewer 90's gaze is converted into relative coordinates with respect to the data screen displayed on the viewer-side display portion 200 and processed. Therefore, in this embodiment, the presenter-side device 40 can accurately display the position of the viewer 90's gaze relative to the area of ​​the data screen 400b displayed on the presenter-side display portion 300 using relative coordinates, as in the case of the viewer 90's side. According to this embodiment, even if the size of the display portion differs between the viewer 90's side and the presenter 80's side, or if the position and size of the data screen relative to the display portion differ, the position of the viewer 90's gaze can be accurately reproduced using relative coordinates. Therefore, according to this embodiment, the presenter 80 can easily determine whether the viewer 90 is viewing data, and further, which part of the data the viewer 90 is viewing.

[0178] In this embodiment, when the viewer 90's gaze is outside the frame of the data image, the position 600 of the viewer 90's gaze, obtained using relative coordinates, is displayed as a cursor 500b on the outer periphery of the area of ​​the data screen 400b adjacent to position 600. This cursor 500b has a different pattern than the cursor 500a when the viewer 90's gaze is inside the frame of the data image. According to this embodiment, by performing such a display, the presenter 80 can easily grasp that the viewer 90's gaze is outside the frame of the data screen and understand the approximate position of the viewer 90's gaze.

[0179] In this embodiment, since the ideal position of the viewer 90's gaze can be estimated in real time, by combining the information of the ideal position of the viewer 90's gaze with the information of the position of the viewer 90's gaze, useful information for effectively performing the presentation can be provided to the presenter 80. The details of specific implementations using the above-described functions will be described below in sequence.

[0180] Note that the embodiments of this disclosure described below are performed by the information processing system according to the first embodiment. However, in the first embodiment and the following embodiments of this disclosure, the configuration of the information processing system and the configuration of the devices included in the information processing system are not limited to... Figure 1 The pattern shown can be modified into various patterns. For example, the information processing system according to the first embodiment can be configured as an information processing device. Furthermore, in the first embodiment and the following embodiments of this disclosure, each functional unit is not limited to being included in... Figure 1 The device shown can be included in another device or can be included in multiple devices.

[0181] Furthermore, in the first embodiment and the following embodiments of this disclosure, the operation of the functional blocks included in the information processing system is not limited to... Figure 2 and Figure 3 The pattern shown can be modified into various other patterns.

[0182] 3. Second Implementation Method

[0183] First, as one embodiment of this disclosure, it is conceivable to directly display the viewer 90's gaze position on the presenter-side device 40. For example, it is conceivable to display the time-varying changes in the viewer 90's gaze position. However, in such a case, the line indicating the time change would slightly shift upwards, downwards, to the left, and to the right due to possible errors in estimating the gaze position or the viewer 90's blinking (saccades). Therefore, even if the time-varying changes in the viewer 90's gaze position are displayed to the presenter 80, the time change is difficult to see, and thus, the time change cannot provide useful information to the presenter 80.

[0184] Furthermore, depending on the data transmission rate between the viewer-side device 20 and the presenter-side device 40, information about the viewer 90's gaze position may be temporarily lost, thus the movement of points indicating time changes may become discontinuous and difficult to see. Therefore, in the second embodiment of this disclosure, when displaying the time change of the viewer 90's gaze position, predetermined processing is performed to make the time change easily visible and to provide useful information to the presenter 80. Hereinafter, reference will be made to... Figures 4A to 5B The second embodiment is described. Figure 4A and Figure 5A These are explanatory diagrams used to describe the operation in this embodiment, and Figure 4B and Figure 5B This is a flowchart of the information processing method according to this embodiment.

[0185] First, such as Figure 4A As shown, when the time change 94 of the viewer 90's gaze position is directly displayed on the presenter-side device 40, the time change 94 gradually varies in both the vertical and horizontal directions due to errors that may occur when estimating the gaze position and noise caused by the viewer 90's blinking (saccades). Therefore, in this embodiment, the time change 96 is displayed after removing noise from the viewer 90's gaze position information using a finite impulse response (FIR) filter or an infinite impulse response (IIR) filter. Figure 4AAs shown, compared to time variation 94, the time variation 96 obtained through processing does not exhibit gradual changes in the vertical and horizontal directions. Because the IIR filter has an impulse response function that returns a non-zero value over an infinitely long time, all coordinates in the history are fed back to determine the information to be output. Because the FIR filter has a finite-time impulse response function, coordinates within a predetermined time range in the history are fed back to determine the information to be output. Therefore, it is possible to obtain... Figure 4A The smoothed time variation is shown in Figure 96.

[0186] Next, an example of information processing according to this embodiment will be described. First, as Figure 4B As shown, the information processing system acquires information about the viewer 90's gaze position (specifically, relative coordinates) (step S41). Next, the information processing system stores the acquired viewer 90's gaze position information as coordinate history in a storage section (not shown) (step S42). Then, the information processing system determines whether a sufficient number of viewer 90 gaze position information entries have been stored to display the time-varying changes in the viewer 90's gaze position (step S43). When it is determined that a sufficient number of viewer 90 gaze position information entries have been stored (step S43: Yes), the information processing system proceeds to step S44. On the other hand, when it is determined that a sufficient number of viewer 90 gaze position information entries have not been stored (step S43: No), the information processing system proceeds to step S45. Furthermore, the information processing system uses an FIR filter to remove noise from the stored multiple viewer 90 gaze position information entries and smooths the information (step S44). Next, the information processing system displays the time-varying changes in the viewer 90's gaze position (step S45) and ends the processing.

[0187] like Figure 5A As shown, when the time change 94 of the viewer 90's gaze position is directly displayed on the presenter-side device 40, the information about the viewer 90's gaze position may be temporarily lost due to the data transmission rate between the viewer-side device 20 and the presenter-side device 40, and the dot group indicating the time change 94 may become discontinuous, as in the dot group indicated by large circles. Therefore, in this embodiment, processing such as drawing frame interpolation (linear interpolation, etc.) is performed, and the dot group including small circles is displayed as the time change 96. For example, as Figure 5A As shown, the time change 96 obtained through the above processing is not discontinuous, but smooth, compared to the group of points indicated by large circles according to the time change 94.

[0188] Next, examples of information processing according to this embodiment will be described. For example... Figure 5BAs shown, the information processing system acquires information about the viewer 90's gaze position (step S51). Next, based on the acquired viewer 90's gaze position, the information processing system causes the viewer-side device 20 to display the time change of the viewer 90's gaze position (step S52). Next, based on the data transmission rate (linear speed) between the viewer-side device 20 and the presenter-side device 40, the information processing system performs throttling control to temporarily suppress the transmission of information about the viewer 90's gaze position from the viewer-side device 20 to the presenter-side device 40 (step S53). Then, the information processing system allows some of the viewer 90's gaze position information to pass through (step S53: Yes) and sends this information to the presenter-side device 40 (step S54). On the other hand, the information processing system does not allow some of the viewer 90's gaze position information to pass through (step S53: No). Furthermore, the information processing system performs interpolation processing, etc., on the information of multiple gaze positions (step S55), causes the presenter-side device 40 to display the time change of the viewer 90's gaze position (step S56), and ends the processing.

[0189] As described above, in this embodiment, when displaying the time change of the viewer 90's gaze position, a smooth time change 96 that does not gradually change in the vertical and horizontal directions can be obtained by performing predetermined processing. Therefore, according to this embodiment, the time change can be made easier to see, and useful information can be provided to the presenter 80.

[0190] Note that this implementation method is not limited to Figures 4A to 5B The pattern shown can be modified into various other patterns.

[0191] 4. Third Implementation Method

[0192] In the third embodiment of this disclosure, a display format for changing the gaze position of viewer 90 is used, calculated based on the degree of fixation based on the amount of gaze movement per unit time. Specifically, in this embodiment, by changing the display format for changing the gaze position of viewer 90 based on the degree of fixation, the presenter 80 can easily grasp information such as where viewer 90 has been focusing for a long time or where viewer 90's gaze is not fixed due to confusion. Referring below... Figure 6A and Figure 6B The third implementation method is described. Figure 6A These are explanatory diagrams used to describe the operation in this embodiment, and Figure 6B This is a flowchart of the information processing method according to this embodiment.

[0193] like Figure 6AAs shown in the upper part, in this embodiment, for example, when displaying the time change of the viewer 90's gaze position, if the movement of the gaze per unit time is considered small and the degree of fixation is high, the line 98a indicating the time change can be drawn thick and dark. In this embodiment, for example, when the degree of fixation is considered high, the cursor indicating the gaze position can be drawn thick and dark. On the other hand, as... Figure 6A As shown in the lower part, in this embodiment, when displaying the time change of the viewer 90's gaze position, for example, if the movement of the gaze per unit time is considered small and the degree of fixation is low, the line 98b indicating the time change can be drawn thin and light. In this embodiment, for example, when the degree of fixation is considered low, the cursor indicating the gaze position can be drawn thin and light.

[0194] For example, in this embodiment, the degree of fixation indicating the fixation of the gaze position can be obtained by calculating the amount of gaze movement per predetermined unit of time based on information about the viewer 90's gaze position, and further calculating the reciprocal of the calculated movement. In this embodiment, the display format of the temporal change in the viewer 90's gaze position is changed based on the aforementioned degree of fixation. Therefore, according to this embodiment, the presenter 80 can easily and intuitively grasp information such as where the viewer 90 has been focusing for a long time or whether the viewer 90's gaze is not fixed due to confusion.

[0195] Next, an example of information processing according to this embodiment will be described. First, as Figure 6B As shown, the information processing system acquires information about the viewer 90's gaze position (specifically, relative coordinates) (step S61). Next, the information processing system stores the acquired viewer 90's gaze position information as coordinate history in a storage section (not shown) (step S62). Then, the information processing system determines whether a sufficient number of viewer 90 gaze position information entries have been stored in order to display the time change of the viewer 90's gaze position (step S63). When it is determined that a sufficient number of viewer 90 gaze position information entries have been stored (step S63: Yes), the information processing system proceeds to step S64. On the other hand, when it is determined that a sufficient number of viewer 90 gaze position information entries have not been stored (step S63: No), the information processing system proceeds to step S68.

[0196] Next, the information processing system uses the stored information on the gaze positions of multiple viewers 90 to calculate the degree of fixation (step S64). The information processing system then compares the calculated degree of fixation with a predetermined threshold (step S65). When it is determined that the calculated degree of fixation is equal to or greater than the predetermined threshold (step S65: Yes), the information processing system proceeds to step S67. On the other hand, when it is determined that the calculated degree of fixation is less than the predetermined threshold (step S65: No), the information processing system proceeds to step S66. Furthermore, the information processing system draws a cursor or line indicating the gaze position of the viewers 90 in a thin and light manner (step S66). The information processing system draws the cursor or line indicating the gaze position of the viewers 90 in a thick and dark manner (step S67). Finally, the information processing system displays the cursor or line indicating the gaze position of the viewers 90 (step S68) and ends the process.

[0197] Note that this implementation method is not limited to Figure 6A and Figure 6B The pattern shown can be modified into various other patterns.

[0198] 5. Fourth Implementation Method

[0199] In each of the above embodiments of this disclosure, since the viewer 90's gaze position is displayed in real time, such a display can actually distract the presenter 80. Furthermore, in each of the above embodiments of this disclosure, since the viewer 90's gaze position is displayed in real time, the viewer 90 may feel uncomfortable from a privacy perspective. Therefore, in the fourth embodiment of this disclosure, the estimation and display of the gaze position can be stopped by either the presenter 80 or the viewer 90 at their own discretion. Hereinafter, reference will be made to... Figure 7 The fourth implementation method is described. Figure 7 This is an explanatory diagram used to describe the operation in this embodiment.

[0200] For example, such as Figure 7 As shown, when the display of the viewer 90's gaze position is stopped, the presenter 80 operates a button 702 displayed on the video 950 of each viewer on the presenter-side display portion 300, which is shown along with the name of each viewer. According to this embodiment, by operating such a button 702, the presenter 80 can stop the display of the viewer 90's gaze position or display the viewer 90's gaze position again.

[0201] like Figure 7As shown, when the detection of the viewer 90's gaze position is stopped, the viewer 90 operates the button 700 displayed on the viewer-side display portion 200. According to this embodiment, by operating such a button 700, the viewer 90 can stop or restart the detection of his / her own gaze position.

[0202] As described above, according to this embodiment, if the real-time display of the viewer 90's gaze position is distracting, the presenter 80 can stop the display of the viewer 90's gaze position. Furthermore, according to this embodiment, if the viewer 90 feels uncomfortable from a privacy perspective regarding the detection of his / her own gaze position, the viewer 90 can stop the detection of his / her own gaze position.

[0203] Note that this implementation method is not limited to Figure 7 The pattern shown can be modified into various other patterns.

[0204] 6. Fifth Implementation Method

[0205] In each of the above embodiments of this disclosure, the viewer 90's gaze position is displayed by a cursor or the like. However, the embodiments of this disclosure are not limited to displaying the viewer 90's gaze position with a cursor. Therefore, in the fifth embodiment of this disclosure, the viewer 90's gaze position is displayed in a display form other than a cursor. Hereinafter, reference will be made to... Figure 8A and Figure 8B The fifth implementation method is described. Figure 8A These are explanatory diagrams used to describe the operation in this embodiment, and Figure 8B This is a flowchart of the information processing method according to this embodiment.

[0206] For example, in this embodiment, a grid 350 is provided on the data screen (demonstration screen) to divide the data screen into multiple units 352. In this embodiment, when the viewer 90's gaze position is within the data screen, the unit 352 including the viewer 90's gaze position (specifically, the entire unit 352 or the frame (outer perimeter) of the unit 352) is displayed in a flashing manner, such as... Figure 8A As shown in the upper part. Additionally, in this embodiment, when the viewer 90's line of sight is outside the data frame, as... Figure 8A As shown in the lower part, among the four line segments 354 that form the outer periphery of the unit 352 adjacent to the viewer's line of sight position, the line segment 354 adjacent to the viewer's line of sight position is displayed in a flashing manner.

[0207] In this embodiment, the viewer 90's gaze position is not displayed by a cursor, but rather by a unit 352 that includes the viewer 90's gaze position or a line segment 354 adjacent to the viewer 90's gaze position. Therefore, in this embodiment, even when the estimation accuracy of the viewer 90's gaze position is not high, the viewer 90's gaze position can still be presented to the presenter 80. Furthermore, in this embodiment, since the unit 352 to be displayed is identified based on whether the gaze position is included in a unit 352 with a predetermined area, even if the viewer 90's gaze position moves rapidly, the unit 352 to be displayed will not immediately switch, thus avoiding strain on the presenter 80's eyes.

[0208] Next, examples of information processing according to this embodiment will be described. For example... Figure 8B As shown, firstly, the information processing system obtains the relative coordinates of the viewer 90's gaze position (step S81). Next, the information processing system determines whether the viewer 90's gaze position is within the frame of the data screen (step S82). When it is determined that the viewer 90's gaze position is within the frame of the data screen (step S82: Yes), the information processing system proceeds to step S83. On the other hand, when it is determined that the viewer 90's gaze position is outside the frame of the data screen (step S82: No), the information processing system proceeds to step S86.

[0209] Next, the information processing system creates a grid within the frame dividing the data display, and identifies cells 352 from the multiple cells 352 generated from the grid that include the viewer 90's line of sight (step S83). Then, the information processing system draws the identified cells 352 as illuminated (step S84). Next, the information processing system creates a grid within the frame dividing the data display, and identifies line segments 354 from the multiple cells 352 generated from the grid that are adjacent to the viewer 90's line of sight (step S85). The information processing system then draws the identified line segments 354 as illuminated (step S86). Finally, the information processing system displays the drawing operations in steps S84 and S86 (step S87) and ends the processing.

[0210] Note that this implementation method is not limited to Figure 8A and Figure 8B The pattern shown can be modified into various other patterns.

[0211] 7. Sixth Implementation Method

[0212] Next, we will refer to Figure 9A and Figure 9B The sixth embodiment of the present disclosure is described. Figure 9AThese are explanatory diagrams used to describe the operation in this embodiment, and Figure 9B This is a flowchart of the information processing method according to this embodiment.

[0213] In each of the above embodiments of this disclosure, such as Figure 9A As shown in the upper part, when the viewer 90's gaze position is within the frame of the data screen, the viewer 90's gaze position is displayed by cursor 500a (in-frame cursor), and when the viewer 90's gaze position is outside the frame of the data screen, the viewer 90's gaze position is displayed by cursor 500b (out-of-frame cursor). That is, in each of the above embodiments of this disclosure, the display mode of the viewer 90's gaze position changes depending on whether the viewer 90's gaze position is within the frame of the data screen (in... Figure 9A (The shape and size of the cursor 500 vary in the upper part). Therefore, in each embodiment of this disclosure, the display mode is switched at the moment when the viewer 90's line of sight moves out of the frame of the data screen, or at the moment when the viewer 90's line of sight moves into and out of the frame. When the display mode is switched frequently, the presenter 80 will be distracted and may be unable to focus on the explanation, etc.

[0214] Therefore, in the sixth embodiment of this disclosure, even when the viewer 90's gaze position is outside the frame of the data screen, if the distance from the data screen (frame) is within a predetermined threshold (predetermined range), the display is performed in the same manner as when the viewer 90's gaze position is inside the frame of the data screen (in-frame cursor 500a), such as... Figure 9A As shown in the lower part. In this embodiment, when the viewer 90's line of sight appears outside the frame of the data screen and the distance from the data screen (frame) is equal to or greater than a predetermined threshold, the display is performed in the display form (outside the frame cursor 500b) when the viewer 90's line of sight appears outside the frame of the data screen.

[0215] In this embodiment, even when the viewer 90's gaze is outside the frame of the data display, the cursor 500a is displayed within the frame when the viewer 90's gaze is within a predetermined range from the frame of the data display. In this embodiment, the cursor 500b is displayed only when the viewer 90's gaze is significantly deviated from the frame of the data display. That is, in this embodiment, a buffer is provided at the moment the display mode of the viewer 90's gaze position is switched, thereby suppressing frequent switching of the display mode. Therefore, according to this embodiment, the presenter 80 only pays attention to the cursor 500b outside the frame when the viewer 90's gaze position is significantly deviated from the frame of the data display, and can focus on the explanation, etc. Note that in this embodiment, it is preferable to set the buffer (i.e., a predetermined threshold) based on the estimation accuracy of the gaze position.

[0216] Next, examples of information processing according to this embodiment will be described. For example... Figure 9B As shown, firstly, the information processing system acquires the relative coordinates of the viewer 90's gaze position (step S91). Next, the information processing system determines whether the viewer 90's gaze position is within the frame of the data screen (step S92). When it is determined that the viewer 90's gaze position is within the frame of the data screen (step S92: Yes), the information processing system proceeds to step S93. On the other hand, when it is determined that the viewer 90's gaze position is outside the frame of the data screen (step S92: No), the information processing system proceeds to step S94. Then, the information processing system displays the viewer 90's gaze position using an in-frame cursor 500a (step S93).

[0217] Next, the information processing system determines whether the distance between the viewer 90's gaze position and the frame of the data screen is within a predetermined threshold (step S94). When it is determined that the distance between the viewer 90's gaze position and the frame of the data screen is within the predetermined threshold (step S94: Yes), the information processing system proceeds to step S93. On the other hand, when it is determined that the distance between the viewer 90's gaze position and the frame of the data screen is equal to or greater than the predetermined threshold (step S94: No), the information processing system proceeds to step S95. Then, the information processing system displays the viewer 90's gaze position using an outside-frame cursor 500b (step S95). Finally, the information processing system ends the process.

[0218] Note that this implementation method is not limited to Figure 9A and Figure 9B The pattern shown can be modified into various other patterns.

[0219] 8. Seventh Implementation Method

[0220] Next, we will refer to Figure 10A and Figure 10B The seventh embodiment of this disclosure is described. Figure 10A These are explanatory diagrams used to describe the operation in this embodiment, and Figure 10B This is a flowchart of the information processing method according to this embodiment.

[0221] like Figure 10AAs shown, even when the viewer 90's line of sight 92 is within the window 750a displaying the data screen, for example, since the viewer 90 is not interested in the presenter 80's explanation, the viewer 90 may be looking at another window 750b superimposed on the window 750a. Therefore, in the seventh embodiment of this disclosure, a warning indicating that the viewer is looking at another window 750b superimposed on the data screen is displayed so that the presenter 80 can recognize such a situation. In this embodiment, the warning display can be performed, for example, by blinking a cursor within a frame.

[0222] According to this embodiment, the presenter 80 can recognize that the viewer 90 is viewing another window 750b superimposed on the document window 750a. According to this embodiment, the presenter 80 can grasp the detailed state of the viewer 90, and therefore can perform an effective presentation, such as taking actions that attract the viewer 90's interest.

[0223] Next, examples of information processing according to this embodiment will be described. For example... Figure 10B As shown, firstly, the information processing system acquires the relative coordinates of the viewer 90's gaze (step S101). Next, the information processing system determines whether the viewer 90's gaze position is within the frame of the data screen (step S102). When it is determined that the viewer 90's gaze position is within the frame of the data screen (step S102: Yes), the information processing system proceeds to step S103. On the other hand, when it is determined that the viewer 90's gaze position is outside the frame of the data screen (step S102: No), the information processing system proceeds to step S105.

[0224] Next, the information processing system determines whether the viewer 90's gaze position is outside the coordinate range of another window 750b superimposed on the window 750a displaying the data image (step S103). When it is determined that the viewer 90's gaze position is outside the coordinate range of the other window 750b (step S103: Yes), the information processing system proceeds to step S104. On the other hand, when it is determined that the viewer 90's gaze position is within the coordinate range of the other window 750b (step S103: No), the information processing system proceeds to step S106.

[0225] Then, the information processing system selects to display the viewer 90's gaze position using an in-frame cursor 500a (step S104). Next, the information processing system displays the viewer 90's gaze position using an out-of-frame cursor 500b (step S105). Furthermore, the information processing system displays the viewer 90's gaze position by, for example, causing the in-frame cursor 500a to blink (displaying a warning cursor) (step S106). Then, the information processing system terminates the process.

[0226] Note that this implementation method is not limited to Figure 10A and Figure 10B The pattern shown can be modified into various other patterns.

[0227] 9. Eighth Implementation Method

[0228] Next, we will refer to Figures 11A to 13B The eighth embodiment of the present disclosure is described. Figure 11A , Figure 12A and Figure 13A These are explanatory diagrams used to describe the operation in this embodiment, and Figure 11B , Figure 12B and Figure 13B This is a flowchart of the information processing method according to this embodiment. As previously described, in embodiments of this disclosure, the ideal gaze position estimator 304 can estimate the ideal gaze position of the viewer 90 in real time. Therefore, in the eighth embodiment of this disclosure, information about the ideal gaze position of the viewer 90 and information about the actual gaze position of the viewer 90 are combined to provide useful information to the presenter 80 for effectively performing the presentation. Specifically, in this embodiment, the display mode of the viewer 90's gaze position is changed according to the positional relationship between the viewer 90's gaze position and the ideal position.

[0229] In this embodiment, for example, such as Figure 11A As shown, when the presenter 80 is explaining a portion of the material while pointing with the cursor (mouse cursor) 960, the ideal gaze position estimator 304 estimates that the position pointed to by the cursor 960 is the ideal gaze position for the viewer 90. For example, as Figure 11A As shown, when the viewer 90's actual line of sight is at position 92, compared to the case where the viewer 90's line of sight is at position 91 relative to the ideal position 960, the viewer 90's line of sight has deviated from the ideal position 960. Therefore, in this embodiment, when the line of sight is at position 91, the viewer 90's line of sight has left the part that the presenter 80 wants the viewer 90 to see. Therefore, the presenter 80 is given an emphasis display (warning display) to inform the presenter 80 of this fact. Thus, according to this embodiment, the presenter 80 can recognize that the viewer 90's line of sight has left the part that the presenter 80 wants the viewer 90 to see due to the emphasis display, and can therefore take immediate action to perform an effective presentation.

[0230] For example, if the presenter 80 explains faster than the viewer 90 understands, assume that the viewer 90's gaze position 92 gradually deviates from the viewer 90's ideal gaze position 91. Therefore, according to this embodiment, since the presenter 80 can be immediately notified that the viewer 90's gaze position 92 has deviated from the viewer 90's ideal gaze position 91, the presenter 80 can slow down the explanation or repeat the explanation.

[0231] Next, examples of information processing according to this embodiment will be described. For example... Figure 11B As shown, firstly, the information processing system acquires the relative coordinates of the viewer 90's gaze position (step S111). Next, the information processing system acquires information about the ideal position of the viewer 90's gaze (step S112). Furthermore, the information processing system determines whether the viewer 90's gaze position is within the frame of the data screen (step S113). When it is determined that the viewer 90's gaze position is within the frame of the data screen (step S113: Yes), the information processing system proceeds to step S114. On the other hand, when it is determined that the viewer 90's gaze position is outside the frame of the data screen (step S113: No), the information processing system proceeds to step S116.

[0232] Next, the information processing system determines whether the distance between the viewer 90's line of sight and the ideal position is within a predetermined threshold (step S114). When it is determined that the distance between the viewer 90's line of sight and the ideal position is within the predetermined threshold (step S114: Yes), the information processing system proceeds to step S115. On the other hand, when it is determined that the distance between the viewer 90's line of sight and the ideal position is equal to or greater than the predetermined threshold (step S114: No), the information processing system proceeds to step S117.

[0233] Then, the information processing system selects to display the viewer 90's gaze position using an in-frame cursor 500a (step S115). Next, the information processing system displays the viewer 90's gaze position using an out-of-frame cursor 500b (step S116). Furthermore, the information processing system displays the viewer 90's gaze position by, for example, causing the in-frame cursor 500a to blink (displaying a warning cursor) (third display mode) (step S117). Then, the information processing system ends the processing.

[0234] Furthermore, modifications to this embodiment will be described. Here, for example, such as... Figure 12AThe following description illustrates a modification applicable to a scenario where data is configured using multiple sections 980. In this case, it is assumed that the viewer 90's ideal gaze moves sequentially within the areas of each section 980 according to the presentation order given by the presenter 80. However, some sections 980, including those in such ideal positions, may never be noticed by the viewer 90. In such cases, the viewer 90 is highly likely not to recognize the content described in the corresponding section 980. Therefore, in this modification, sections 980 that the viewer 90 has never paid attention to are continuously displayed to the presenter 80. Thus, according to this modification, such a display allows the presenter 80 to notice the existence of sections 980 that the viewer 90 has never paid attention to, even after a series of explanations has been completed. Therefore, according to this modification, the presenter 80 can take action to perform an effective presentation, such as pointing back to a section 980 to bring the viewer 90's gaze back to that section. Furthermore, in this embodiment, the display of the presence of a section 980 that the viewer 90 has never paid attention to continues, so that after the description is completed, the viewer 90 can notice the corresponding section 980 and review the description in the section 980.

[0235] Next, examples of information processing according to this embodiment will be described. For example... Figure 12B As shown, firstly, the information processing system obtains the relative coordinates of the viewer 90's gaze position (step S121). Next, the information processing system obtains information about the ideal position of the viewer 90's gaze (step S122). Furthermore, the information processing system obtains information about the relative coordinates of the area of ​​the column 980 included in the data on the data screen (step S123).

[0236] Next, the information processing system determines whether the viewer 90's gaze is within the frame of the data screen (step S124). When it is determined that the viewer 90's gaze is within the frame of the data screen (step S124: Yes), the information processing system proceeds to step S126. On the other hand, when it is determined that the viewer 90's gaze is outside the frame of the data screen (step S124: No), the information processing system proceeds to step S127.

[0237] Next, the information processing system determines whether the ideal position of viewer 90's gaze is within column 980 (step S125). When it is determined that the ideal position of viewer 90's gaze is within column 980 (step S125: Yes), the information processing system identifies column 980 that includes the ideal position of viewer 90's gaze as the corresponding column 980 and proceeds to step S126. On the other hand, when it is determined that the ideal position of viewer 90's gaze is outside column 980 (step S125: No), the information processing system returns to the beginning.

[0238] Next, the information processing system determines whether the distance between the viewer 90's gaze position and the center of gravity of the corresponding column 980 is within a predetermined threshold (step S126). When it is determined that the distance between the viewer 90's gaze position and the center of gravity of the column 980 is within the predetermined threshold (step S126: Yes), the information processing system proceeds to step S128. On the other hand, when it is determined that the distance between the viewer 90's gaze position and the center of gravity of the column 980 is equal to or greater than the predetermined threshold (step S126: No), the information processing system proceeds to step S129.

[0239] Then, the information processing system displays the viewer 90's gaze position using an outside-frame cursor 500b (step S127). Next, the information processing system displays the viewer 90's gaze position using an inside-frame cursor 500a (step S128). Furthermore, the information processing system displays the viewer 90's gaze position by, for example, blinking the inside-frame cursor 500a (displaying a warning cursor) and blinking the corresponding column 980 identified in step S125 (step S129). In step S129, it is preferable to continue displaying the corresponding column 980 until the predetermined processing is performed. Then, the information processing system ends the processing.

[0240] Furthermore, another modification to this embodiment will be described. The reason why the viewer 90's gaze position 92 deviates from the viewer 90's ideal gaze position 91 is that the viewer 90 has difficulty understanding the description written in a certain part of the material, and even when the presenter 80 has moved on to the next part of the description, the viewer 90's gaze remains fixed on that part. In such a case, in this embodiment and the above modification, as Figure 13A As shown, when the viewer 90's line of sight position 92 deviates from the viewer 90's ideal line of sight position 91, the viewer 90's line of sight position 92 at that moment is displayed to the presenter 80 via cursor 560, etc. The presenter 80 can then notice the display of cursor 560 and can reiterate the explanation above. However, if the display of cursor 560 continues when reiterating the explanation, it can be cumbersome. Therefore, in this modification, when the viewer 90's line of sight position and the viewer 90's ideal line of sight position 91 are aligned, the presenter 80 erases the display of cursor 560, for example, by moving the mouse cursor 550 to the position corresponding to cursor 560 to reiterate the content corresponding to cursor 560, or by providing an explanation corresponding to cursor 560.

[0241] Next, examples of information processing according to this embodiment will be described. For example... Figure 13BAs shown, the information processing system first obtains information about the ideal position of the viewer's gaze from the mouse cursor 550 operated by the presenter 80 (step S131). Next, the information processing system obtains the relative coordinates of the viewer 90's gaze position (step S132).

[0242] Next, the information processing system determines whether the viewer 90's gaze is within the frame of the data screen (step S133). When it is determined that the viewer 90's gaze is within the frame of the data screen (step S133: Yes), the information processing system proceeds to step S134. On the other hand, when it is determined that the viewer 90's gaze is outside the frame of the data screen (step S133: No), the information processing system proceeds to step S136.

[0243] Next, the information processing system determines whether the distance between the viewer 90's gaze position and the ideal position is within a predetermined threshold (step S134). When it is determined that the distance between the viewer 90's gaze position and the ideal position is within the predetermined threshold (step S134: Yes), the information processing system proceeds to step S135. On the other hand, when it is determined that the distance between the viewer 90's gaze position and the cursor 550 is equal to or greater than the predetermined threshold (step S134: No), the information processing system proceeds to step S137.

[0244] Then, the information processing system displays the viewer 90's gaze position using an in-frame cursor 500a (step S135). Next, the information processing system displays the viewer 90's gaze position using an out-of-frame cursor 500b (step S136). Furthermore, the information processing system displays the viewer 90's gaze position by, for example, setting a predetermined marker (display of a warning marker) based on the in-frame cursor 500a (step S137). In step S137, the warning marker continues to be displayed until the ideal gaze position approaches the warning marker. Then, the information processing system terminates the process.

[0245] Note that this implementation method and its modifications are not limited to... Figures 11A to 13B The pattern shown can be modified into various other patterns.

[0246] 10. Ninth Implementation Method

[0247] Next, we will refer to Figure 14A and Figure 14B The ninth embodiment of the present disclosure is described. Figure 14A These are explanatory diagrams used to describe the operation in this embodiment, and Figure 14BThis is a flowchart of the information processing method according to this embodiment. In the ninth embodiment of this disclosure, when the viewer 90's gaze position is outside the frame of the data screen, the state estimator 212 estimates the reason why the viewer 90's gaze position is outside the frame of the data screen. In this embodiment, the state estimator 212 estimates the reason based on, for example, the viewer 90's gaze position, ambient sounds around the viewer 90, and the viewer 90's operation information. In this embodiment, the estimated reason is displayed to the presenter 80.

[0248] In this embodiment, for example, such as Figure 14A As shown in the upper part, information about the reasons for the estimation is displayed by icons 562a to 562d, etc. In this embodiment, for example, icons 562a to 562d are displayed in an overlay on the frame of the data screen, so as not to interfere with the data. Specifically, icon 562a is the case where the viewer 90's gaze position is estimated to be outside the frame of the data screen because an input operation is performed on a keyboard (not shown). In this case, the state estimator 212 can estimate the reason based on, for example, the viewer 90's operation information on the keyboard. Icon 562b is the case where the viewer 90's gaze position is estimated to be outside the frame of the data screen because a visitor is present. In this case, the state estimator 212 can estimate the reason based on, for example, the sound of a walkie-talkie (not shown) detected as ambient sound around the viewer 90. Icon 562c is the case where the viewer 90's gaze position is estimated to be outside the frame of the data screen because the viewer 90 is thinking. In this case, the state estimator 212 can estimate the reason based on, for example, the viewer 90's gaze position or eye movement. Icon 562d indicates a situation where viewer 90 is viewing an image from another viewer 90's camera device, and the estimated gaze position of viewer 90 is outside the frame of the data screen. In this case, the state estimator 212 can estimate the cause based on, for example, the gaze position of viewer 90. Note that in this embodiment, when displaying the cause, the gaze position of viewer 90 can be displayed using the outside-frame cursor 500b, and the ideal position of viewer 90's gaze can also be displayed.

[0249] As described above, according to this embodiment, since the presenter 80 can grasp the detailed state of the viewer 90, he / she can take actions based on the viewer 90's situation (e.g., temporarily stop the explanation), thereby enabling an effective presentation.

[0250] In this embodiment, such as Figure 14A As shown in the lower part, an icon 562e can be displayed to indicate that the viewer 90 is watching the image of the camera device of the presenter 80, and such a display enables the presenter 80 to respond immediately when the viewer 90 speaks to the presenter 80.

[0251] Next, examples of information processing according to this embodiment will be described. For example... Figure 14B As shown, firstly, the information processing system acquires the relative coordinates of the viewer 90's gaze position (step S141). Furthermore, the information processing system determines whether the viewer 90's gaze position is outside the frame of the data screen (step S142). When it is determined that the viewer 90's gaze position is outside the frame of the data screen (step S142: Yes), the information processing system proceeds to step S143. On the other hand, when it is determined that the viewer 90's gaze position is inside the frame of the data screen (step S142: No), the information processing system proceeds to step S146.

[0252] Next, the information processing system determines whether the viewer 90's gaze is within the image of the camera device of the presenter 80 and the other viewer 90 (step S143). When it is determined that the viewer 90's gaze is within the image of the camera device of the presenter 80 and the other viewer 90 (step S143: Yes), the information processing system proceeds to step S144. On the other hand, when it is determined that the viewer 90's gaze is not within the image of the camera device of the presenter 80 and the other viewer 90 (step S143: No), the information processing system proceeds to step S145.

[0253] Next, the information processing system determines whether the viewer 90's gaze is within the image of the presenter 80's camera device (step S144). When it is determined that the viewer 90's gaze is within the image of the presenter 80's camera device (step S144: Yes), the information processing system proceeds to step S147. On the other hand, when it is determined that the viewer 90's gaze is not within the image of the presenter 80's camera device (step S144: No), the information processing system proceeds to step S148.

[0254] Next, the information processing system determines whether the reason estimated by the state estimator 212 is that the viewer 90 is thinking (step S145). When it is determined that the estimated reason is that the viewer 90 is thinking (step S145: Yes), the information processing system proceeds to step S147. On the other hand, when it is determined that the estimated reason is not that the viewer 90 is thinking (step S145: No), the information processing system proceeds to step S148.

[0255] Next, the information processing system selects the cursor 500a within the display box (step S146). Then, the information processing system highlights the cursor 500b outside the box (e.g., displays the cursor 500b outside the box in a blinking manner) and displays an icon 562 indicating the reason (step S147). For example, as Figure 14AAs shown in the lower part, an icon 562e is displayed indicating that the viewer 90 is watching the image from the camera device of the presenter 80. Additionally, the information processing system displays a cursor 500b outside the frame and an icon 562 indicating the reason (step S148). The information processing system then terminates processing.

[0256] Note that this implementation method is not limited to Figure 14A and Figure 14B The pattern shown can be modified into various other patterns.

[0257] 11. Tenth Implementation Method

[0258] Next, we will refer to Figure 15A and Figure 15B The tenth embodiment of this disclosure is described. Figure 15A These are explanatory diagrams used to describe the operation in this embodiment, and Figure 15B This is a flowchart of the information processing method according to this embodiment.

[0259] In each of the above embodiments of this disclosure, a warning is displayed when the viewer 90's gaze is outside the frame of the data screen. However, in such embodiments, for example, when the accuracy of gaze position detection is not high, an incorrect warning may be displayed even when the viewer 90 is viewing the data. Therefore, the tenth embodiment of this disclosure is an embodiment for avoiding such incorrect warning displays. In this embodiment, as... Figure 15A As shown in the upper part, when the presenter 80 points the cursor 550 at a portion of the frame near the data screen while giving explanations, even if the viewer 90's gaze is detected to be outside the frame of the data screen, only the gaze position is displayed with the cursor 500b, without performing a warning display such as flashing. In this embodiment, as... Figure 15A As shown in the lower part, when the presenter 80 is giving instructions while pointing the cursor 550 at a portion of the frame away from the data screen, and when it is detected that the viewer 90's gaze is outside the frame of the data screen, the position is indicated by blinking the cursor 500b. Note that in this embodiment, the ideal position of the viewer 90's gaze can be used instead of the position of the cursor 550.

[0260] As described above, according to this embodiment, even when the detection accuracy of the gaze position is not high, the presenter 80 will not recognize the warning display that does not correctly reflect the viewer 90, and therefore, an effective presentation can be performed.

[0261] Furthermore, when viewer 90 is listening to presenter 80's explanation, viewer 90's gaze can be positioned outside the frame of the information screen to take notes. In such cases, even if viewer 90's gaze is outside the frame of the information screen, it is preferable not to display a warning to presenter 80, because viewer 90 is interested in the information. Therefore, in this embodiment, for example, information about viewer 90's input operations for note-taking (keyboard operation information) is obtained. In this embodiment, the content of the memo is determined to be similar to the explanation based on whether the content of the memo matches or is similar to the content at the position of the cursor 550 used by presenter 80, or whether the content of the memo matches or is similar to the speech uttered by presenter 80. Furthermore, in this embodiment, when it is determined that the content of the memo matches or is similar to the explanation, no warning is displayed. Note that in this embodiment, instead of a warning display, an icon (not shown) can be displayed to instruct presenter 80 to temporarily stop the explanation to ensure time for viewer 90 to take notes. As described above, according to this embodiment, the presenter 80 can detect that the viewer 90 is taking notes on the explanation, and therefore can take action based on the viewer 90's situation (e.g., temporarily pausing the explanation). Note that in this embodiment, for example, the input operation information of the memo obtained by the viewer operation recognition unit 214 of the viewer-side device 20 can be sent to the ideal gaze position estimator 304 of the presenter-side device 40, etc. In this embodiment, for example, the ideal gaze position estimator 304 performs processing such as vectorization on the wording (words or sentences) extracted from the input information and generates data indicating the characteristics of the wording. Furthermore, in this embodiment, for example, the ideal gaze position estimator 304 uses the generated data to determine whether the content of the memo matches or is similar to the explanation of the presenter 80, etc.

[0262] Next, examples of information processing according to this embodiment will be described. For example... Figure 15B As shown, firstly, the information processing system obtains the relative coordinates of the viewer 90's gaze position (step S151). Next, the information processing system obtains information about the ideal position of the viewer 90's gaze (step S152). Furthermore, the information processing system obtains the viewer 90's input operation information (step S153).

[0263] Next, the information processing system determines whether the viewer 90's gaze is within the frame of the data screen (step S154). When it is determined that the viewer 90's gaze is within the frame of the data screen (step S154: Yes), the information processing system proceeds to step S155. On the other hand, when it is determined that the viewer 90's gaze is outside the frame of the data screen (step S154: No), the information processing system proceeds to step S156. Then, the information processing system displays the viewer 90's gaze position using an in-frame cursor 500a (step S155).

[0264] Next, the information processing system determines whether the viewer 90's gaze position is near the ideal gaze position (step S156). Specifically, in step S156, the information processing system determines whether the distance between the viewer 90's gaze position and the ideal gaze position is within a predetermined threshold. When it is determined that the distance between the viewer 90's gaze position and the ideal position is within the predetermined threshold (step S156: Yes), the information processing system proceeds to step S157. On the other hand, when it is determined that the distance between the viewer 90's gaze position and the ideal position is equal to or greater than the predetermined threshold (step S156: No), the information processing system proceeds to step S158. Then, the information processing system displays the viewer 90's gaze position using an out-of-frame cursor 500b (step S157).

[0265] Next, based on the input information from viewer 90, the information processing system determines whether the content input by viewer 90 is close to (matches or is similar to) the content of the materials or instructions (step S158). When it is determined that the content input by viewer 90 matches or is similar to the content of the materials or instructions (step S158: Yes), the information processing system proceeds to step S157. On the other hand, when it is determined that the content input by viewer 90 does not match or is dissimilar to the content of the materials or instructions (step S158: No), the information processing system proceeds to step S159. Then, the information processing system displays the viewer 90's gaze position by causing the cursor 500a within the frame to blink (warning display) (step S159). The information processing system then terminates the process.

[0266] Note that this implementation method is not limited to Figure 15A and Figure 15B The pattern shown can be modified into various other patterns.

[0267] 12. Eleventh Implementation Method

[0268] Next, we will refer to Figures 16A to 16C The eleventh embodiment of this disclosure is described. Figure 16A These are explanatory diagrams used to describe the operation in this embodiment, and Figure 16B and Figure 16C This is a flowchart of the information processing method according to this embodiment.

[0269] Incidentally, a viewer 90 who is not interested in the presentation may become interested due to certain triggers. In such a case, it is assumed that a movement of the viewer 90's gaze from outside the frame of the data screen back into the frame of the data screen is detected. Therefore, in the eleventh embodiment of this disclosure, when the viewer 90's gaze returns from outside the frame of the data screen to the frame of the data screen, the content explained by the presenter 80 is displayed in a column 990, etc. Figure 16A As shown in the diagram. In this way, according to this embodiment, the presenter 80 can grasp the content that the viewer 90 is interested in. Note that in this embodiment, the same method as the method for estimating the ideal position of the viewer 90's gaze in the above embodiments can be used to identify the portion of content explained by the presenter 80. Specifically, for example, the ideal gaze position estimator 304 estimates the ideal position of the viewer 90's gaze based on the presenter 80's gaze position, the voice emitted by the presenter 80, the presenter 80's operational information (e.g., the position of the cursor indicating a portion of the data), etc. Therefore, in this embodiment, when the viewer 90's gaze returns from outside the frame of the data screen to inside the frame of the data screen, the portion of content explained by the presenter 80 can be identified based on this ideal position.

[0270] According to this embodiment, the presenter 80 can grasp the content that has triggered the viewer 90's interest in the presentation, and therefore, based on this grasp, can take action to achieve an effective presentation or design materials or explanations.

[0271] Next, examples of information processing according to this embodiment will be described. For example... Figure 16B As shown, firstly, the information processing system acquires the relative coordinates of the viewer 90's gaze position (step S1161). Next, the information processing system acquires the input operation information of the presenter 80 (step S1162). Then, the information processing system acquires the information of the presenter 80's gaze position (step S1163). Furthermore, the information processing system acquires the information of the speech emitted by the presenter 80 (step S1164).

[0272] Next, the information processing system determines whether the viewer 90's gaze is within the frame of the data screen (step S1165). When it is determined that the viewer 90's gaze is within the frame of the data screen (step S1165: Yes), the information processing system proceeds to step S1166. On the other hand, when it is determined that the viewer 90's gaze is outside the frame of the data screen (step S1165: No), the information processing system proceeds to step S1161.

[0273] Next, the information processing system determines whether the presenter 80 is giving instructions while moving the mouse (step S1166). If it is determined that the presenter 80 is giving instructions while moving the mouse (step S1166: Yes), the information processing system proceeds to step S1167. On the other hand, if it is determined that the presenter 80 is not giving instructions while moving the mouse (step S1166: No), the information processing system proceeds to step S1168. Then, the information processing system analyzes the content written in the part of the data indicated by the mouse (step S1167).

[0274] Next, the information processing system determines whether the presenter 80 is giving explanations while following the explanatory section of the material with his / her gaze (step S1168). When it is determined that the presenter 80 is giving explanations while following the explanatory section with his / her gaze (step S1168: Yes), the information processing system proceeds to step S1169. On the other hand, when it is determined that the presenter 80 is not giving explanations while following the explanatory section with his / her gaze (step S1168: No), the information processing system proceeds to step S1170. Then, the information processing system analyzes the content written in the material at the section corresponding to the presenter 80's gaze position (step S1169).

[0275] Next, the information processing system analyzes the content explained by the presenter 80 based on the voice emitted by the presenter 80 (step S1170). The information processing system displays the analysis results as information indicating the content of interest to the viewer 90 (step S1171). The information processing system then terminates the process.

[0276] Notice, Figure 16C This is a flowchart illustrating the modified information processing of this embodiment. In this modification, information about the time when the viewer 90's gaze returns from outside the frame of the data screen to inside the frame is acquired, and this time is stored along with content of interest to the viewer 90. Because this time is used, for example, by comparing this time with video and audio data of the presenter 80 captured during a presentation, information about the explanatory audio that the viewer 90 is interested in, the actions of the presenter 80, etc., can be determined. Note that, in addition to acquiring information about the time when the viewer 90's gaze returns from outside the frame of the data screen to inside the frame and storing this time along with content of interest to the viewer 90, Figure 16C Flowcharts and Figure 16B The flowchart is the same as that of the previous one, so its description will be omitted.

[0277] Note that this implementation method is not limited to Figures 16A to 16C The pattern shown can be modified into various other patterns.

[0278] 13. Twelfth Implementation Method

[0279] Next, we will refer to Figure 17A and Figure 17B The twelfth embodiment of this disclosure is described. Figure 17A These are explanatory diagrams used to describe the operation in this embodiment, and Figure 17B This is a flowchart of the information processing method according to this embodiment. The twelfth embodiment of this disclosure is an embodiment that calibrates the detection of the viewer's (90's) gaze position.

[0280] Typically, before starting gaze detection, calibration for detecting gaze position is performed by having the viewer 90 sequentially focus on nine points: the center and four corners of the viewer-side display portion 200 of the viewer-side device 20, the center of the upper and lower ends of the viewer-side display portion 200, and the center of the right and left ends of the viewer-side display portion 200. The orientation and position of the viewer 90's face, as well as the orientation of their eyeballs (pupils), detected at each point, are correlated with the position of each point, thereby enabling calibration for detecting the viewer 90's gaze.

[0281] However, calibration cannot be properly performed when the viewer 90 does not focus correctly on the designated point during such calibration. Even if calibration can be performed properly, the estimated line-of-sight position may deviate from the correct position over time. Therefore, in the twelfth embodiment of this disclosure, calibration (run-calibration) is performed sequentially during the use of the viewer-side device 20.

[0282] In this embodiment, such as Figure 17A As shown, when viewer 90 moves the mouse cursor 550 in viewer-side device 20, it is estimated that viewer 90 is looking at the mouse cursor 550. In this embodiment, calibration is performed by associating the orientation and position of the viewer 90's face, which is focused on the mouse cursor 550, and the orientation of their eyeballs (pupils) with the position of the mouse cursor 550. Note that in this embodiment, calibration can also be performed by using a voice, such as "here," uttered by viewer 90 using viewer-side device 20 to determine the position of the mouse cursor 550 as a reference.

[0283] As described above, according to this embodiment, calibration can be performed sequentially while using the viewer-side device 20 without requiring special operations. Therefore, according to this embodiment, not only can calibration be performed sequentially without placing a burden on the viewer 90, but the gaze position of the viewer 90 can also be continuously detected with high accuracy.

[0284] Next, examples of information processing according to this embodiment will be described. For example... Figure 17BAs shown, firstly, the information processing system obtains the relative coordinates of the viewer 90's gaze position (step S171). Next, the information processing system obtains the viewer 90's input operation information (step S172).

[0285] Next, the information processing system determines whether the viewer 90 is moving the mouse (mouse cursor 550) (step S173). When it is determined that the viewer 90 is moving the mouse (step S173: Yes), the information processing system proceeds to step S174. On the other hand, when it is determined that the viewer 90 is not moving the mouse (step S173: No), the information processing system proceeds to step S173. Then, the information processing system estimates that the viewer 90's current facial position and orientation, as well as the direction of their eyes, are in a situation where the viewer 90 is looking at the mouse cursor 550, and performs calibration (step S174). The information processing system ends the process. Note that in this embodiment, the operation is not limited to mouse operation, but can also be an operation of moving the cursor using, for example, a controller or a pen. Note that in this embodiment, the cursor can be a mouse cursor or pointer used in a presentation, etc.

[0286] Note that this implementation method is not limited to Figure 17A and Figure 17B The pattern shown can be modified into various other patterns.

[0287] 14. Thirteenth Implementation Method

[0288] Next, we will refer to Figure 18 The thirteenth embodiment of this disclosure is described. Figure 18 This is an explanatory diagram used to describe the operation in this embodiment.

[0289] In the thirteenth embodiment of this disclosure, presenter 80 and viewer 90 share the slides of the material being explained. It is assumed that viewer 90 is in a situation where he / she can view a preferred slide of the material (e.g., the Live feature of PowerPoint). In this embodiment, by using each of the above embodiments of this disclosure, information about which slide and which part of the material viewer 90 is viewing can be obtained. Therefore, in this embodiment, presenter 80 is shown which slide and which part of the material viewer 90's gaze is on.

[0290] For example, Figure 18The example shown at the top indicates that the viewer 90's gaze is positioned within the "ABC box" shown on the slide 800b immediately preceding the slide 800a of the material being explained by the presenter 80. Furthermore, in this embodiment, when the viewer 90's gaze is positioned on the slide 800b preceding the slide 800a of the material being explained by the presenter 80, it can be estimated that the viewer 90 cannot understand the explanation; therefore, the viewer 90's gaze is positioned on the preceding slide 800b. In this embodiment, based on such an estimation, wording such as "It would be best to provide further explanation" can be displayed to prompt the presenter 80 to provide further clarification.

[0291] In this embodiment, when the viewer 90's gaze is on the slide 800c following the slide 800a of the material being explained by the presenter 80, it can be estimated that the viewer 90 is bored with the explanation, and therefore the viewer 90's gaze is directed to the next slide 800c. Based on this estimation, in this embodiment, wording such as "It might be better to explain the next slide" can be used to prompt the presenter 80 to explain the previous slide 800c.

[0292] As described above, in this embodiment, the presenter 80 can easily grasp which slide 800 the viewer 90 is focusing on, and therefore, the presenter 80 can perform a more effective presentation.

[0293] In this embodiment, there can be multiple viewers 90. Therefore, in this embodiment, as... Figure 18 As shown in the lower part, the presenter 80 can be shown which slide 800 of the document the gaze of multiple viewers 90 is on. For example, Figure 18 The example in the lower part includes a display unit 802 that uses gradients or the like to indicate which slide 800 the gaze positions of multiple viewers 90 are located on. Specifically, the display unit 802 is configured such that slides 800 that include the gaze positions of more viewers 90 are displayed in a darker color.

[0294] In addition, Figure 18 In the example below, the display unit 804 indicates, via heatmaps or similar means, which slide 800 the gaze positions of multiple viewers 90 are located at. Specifically, the display unit 804 is configured such that the portion where the gaze positions of more viewers 90 are located has a predetermined hue.

[0295] As described above, in this embodiment, the presenter 80 can grasp the comprehension levels of multiple viewers 90, identify areas where explanations are insufficient, and determine areas requiring further explanation. Therefore, the presenter can adjust the content and pacing of the explanation based on this understanding. Furthermore, for example, by applying this embodiment to product sales videos, the products that viewers 90 are interested in become clear, thus this embodiment can be used to analyze, for example, market forecasts of product sales volume. Additionally, for example, by applying this embodiment to live performance videos, the performances that viewers 90 are interested in become clear, thus this embodiment can be used to create performances that attract more attention.

[0296] In this embodiment, not only are the gaze positions of multiple viewers 90 displayed simultaneously, but the gaze positions of one or more specific viewers 90 can also be displayed according to the presenter 80's settings. In this way, according to this embodiment, the presenter 80 can grasp the information of each viewer 90, and therefore can provide explanations tailored to individual pace and comprehension levels. For example, according to this embodiment, the presenter 80 can provide more appropriate feedback to all viewers 90 or an individual. That is, this embodiment can be applied to, for example, remote courses.

[0297] Note that this implementation method is not limited to Figure 18 The pattern shown can be modified into various other patterns.

[0298] 15. Fourteenth Implementation Method

[0299] Next, we will refer to Figure 19 The fourteenth embodiment of this disclosure is described. Figure 19 This is an explanatory diagram illustrating the operation in this embodiment. As described above, by using each embodiment of this disclosure, the portion (ideal position) that the presenter 80 wants the viewer 90 to see in real time can be determined. Therefore, in the fourteenth embodiment of this disclosure, based on the determined portion (ideal position) that the presenter 80 wants the viewer 90 to see, a predetermined effect is automatically applied to the image of materials, etc., and materials, etc., are created for performing a more effective presentation. In this embodiment, for example, such an effect can be performed by the drawing unit 306 of the presenter-side device 40, etc. Note that in this embodiment, the same method as the method for estimating the ideal position of the viewer 90's gaze in the above embodiments can also be used to identify the portion that the presenter 80 wants the viewer 90 to see. Specifically, for example, the ideal gaze position estimator 304 estimates the ideal gaze position of the viewer 90 based on the presenter 80's gaze position, the voice emitted by the presenter 80, the presenter 80's operation information (e.g., the position of the cursor indicating a part of the materials), etc. Therefore, in this embodiment, the portion that the presenter 80 wants the viewer 90 to see is identified based on the ideal gaze position of the viewer 90.

[0300] Specifically, in this embodiment, when the desired portion (ideal location) is identified, for example, such as... Figure 19 As shown in the upper part, the portion to be seen can be automatically surrounded by box 900 or can be made to blink. Additionally, in this embodiment, for example, as... Figure 19 As shown in the middle section, the portion of image 902 that is expected to be seen can be automatically scaled to force image 902 to be displayed to viewer 90. Additionally, in this embodiment, for example, as... Figure 19 As shown in the lower part, the window 904 of the part that is expected to be seen can be automatically moved to the center to force the window 904 to be displayed to the viewer 90 (switching of the camera device).

[0301] As described above, by using each embodiment of this disclosure, the portion that the presenter 80 wants the viewer 90 to see can be determined in real time. Therefore, in this embodiment, for example, when distributing created content, the presenter 80 adds effects to the content using information about the portion they want the viewer 90 to see, thereby obtaining content that attracts more attention from the viewer 90. Thus, in this embodiment, since the effects are added automatically in real time, the distributor (presenter 80) does not need to spend time and effort re-editing the content, and can focus on distribution. This embodiment can be applied, for example, to the distribution of live event images, where the distributor performs the transmission while performing a presentation as presenter 80.

[0302] Note that this implementation method is not limited to Figure 19 The pattern shown can be modified into various other patterns.

[0303] 16. Summary

[0304] As described above, in embodiments of this disclosure, the presenter 80 can easily determine whether the viewer 90 is viewing material, and further, which part of the material the viewer 90 is viewing. In this embodiment, since the ideal position of the viewer 90's gaze can be estimated in real time, by combining the information of the ideal position of the viewer 90's gaze with the information of the viewer 90's gaze position, useful information can be provided to the presenter 80 to enable the presenter 80 to effectively perform the presentation. In other words, this disclosure provides a technique that enables more efficient information transmission provided by systems that share data, audio, etc., between remote locations.

[0305] Note that the above description has described examples of embodiments of this disclosure primarily applied to remote conference room systems, etc. However, embodiments of this disclosure are not limited to such remote conference room systems, and can also be applied to, for example, remote courses, business negotiations, consultations between remote locations, interviews, sales, and various performances (e.g., live performances such as plays and comedies), or to the creation of various videos (e.g., movies, commercials, promotional videos for companies and individuals, educational videos, cooking videos, instructional videos, and tests of network usage), and the analysis of video distribution (product sales videos, news videos, etc.).

[0306] 17. Hardware Configuration Example

[0307] The remote conferencing system 10, viewer-side device 20, presenter-side device 40, etc., according to the above embodiments of this disclosure are provided by, for example, having Figure 20 The computer 1000 with the configuration shown is implemented. Figure 20 This is a hardware configuration diagram illustrating an example of a computer that implements the functions of an apparatus according to embodiments of the present disclosure. The computer 1000 includes a central processing unit (CPU) 1100, random access memory (RAM) 1200, read-only memory (ROM) 1300, a hard disk drive (HDD) 1400, a communication interface 1500, and an input / output interface 1600. The units of the computer 1000 are connected together via a bus 1050.

[0308] The CPU 1100 operates based on programs stored in ROM 1300 or HDD 1400 and controls each unit. For example, the CPU 1100 deploys programs stored in ROM 1300 or HDD 1400 onto RAM 1200 and executes processing corresponding to various programs. For example, integrated circuits such as Application-Specific Integrated Circuits (ASICs) and Field-Programmable Gate Arrays (FPGAs) can be used instead of the CPU 1100. For example, the CPU 1100 can be used as a viewer gaze estimator 202, a viewer relative coordinate acquisition unit 204, a viewer gaze display control unit 210, a state estimator 212, a viewer operation recognition unit 214, a calibration unit 216, etc., in the viewer-side device 20. For example, the CPU 1100 can be used as a viewer gaze visualization unit 302, an ideal gaze position estimator 304, a drawing unit 306, a presenter shared data recognition unit 308, a presenter speech recognition unit 310, a presenter operation recognition unit 312, a presenter gaze estimator 314, etc. in the presenter-side device 40.

[0309] ROM 1300 stores boot programs such as the Basic Input / Output System (BIOS) executed by CPU 1100 when computer 1000 starts up, and programs that depend on the hardware of computer 1000.

[0310] HDD 1400 is a computer-readable recording medium that non-temporarily stores programs executed by CPU 1100, data used by such programs, etc. Specifically, HDD 1400 is a recording medium that records an information processing program according to the present disclosure, the information processing program being an example of program data 1450.

[0311] Communication interface 1500 is an interface for connecting computer 1000 to an external network 1550 (e.g., the Internet). For example, CPU 1100 receives data from another device via communication interface 1500 and sends data generated by CPU 1100 to another device.

[0312] Input / output interface 1600 is an interface for connecting input / output device 1650 and computer 1000. For example, CPU 1100 receives data from input / output device 1650, such as a keyboard, mouse, microphone, camera, or various sensors, via input / output interface 1600. CPU 1100 also sends data to output devices, such as a monitor, speaker, or printer, via input / output interface 1600. Input / output interface 1600 can be used as a media interface for reading programs recorded on a predetermined recording medium (medium). The medium is, for example, an optical recording medium such as a digital multifunction disc (DVD) or phase-change rewritable disc (PD), a magneto-optical recording medium such as a magneto-optical disc (MO), magnetic tape, magnetic recording media, semiconductor memory, etc. For example, input / output interface 1600 can be used as a viewer-side display section 200, an audio output unit 220, etc., of viewer-side device 20. Furthermore, for example, input / output interface 1600 can be used as a presenter-side display section 300, etc., of presenter-side device 40.

[0313] For example, when computer 1000 is used as a remote conferencing system 10, viewer-side device 20, presenter-side device 40, etc., according to embodiments of the present disclosure, the CPU 1100 of computer 1000 executes programs stored in RAM 1200 to perform functions. HDD 1400 stores processing programs, etc., according to the present disclosure. Note that CPU 1100 reads program data 1450 from HDD 1400 and executes the program, but in another example, CPU 1100 may obtain these programs from another device via external network 1550.

[0314] Furthermore, the remote conferencing system 10, viewer-side device 20, and presenter-side device 40 according to this embodiment can be applied to systems that include multiple devices assumed to be connected to a network (or communication between devices), such as cloud computing. That is, the remote conferencing system 10, viewer-side device 20, and presenter-side device 40 according to this embodiment can also be implemented as, for example, an information processing system according to this embodiment using multiple devices.

[0315] The hardware configuration examples of the remote conferencing system 10, the viewer-side device 20, and the presenter-side device 40 have been described above. Each of these components can be constructed from general-purpose parts or from hardware specifically designed for the function of each component. Such configurations can be appropriately modified depending on the technological level available at the time of implementation.

[0316] 18. Supplement

[0317] Embodiments of this disclosure may include, for example, a program for causing the aforementioned information processing system to execute an information processing method according to an embodiment of this disclosure, and a non-transitory tangible medium for recording the program. The program may be distributed via communication lines such as the Internet (including wireless communication).

[0318] The components of the device shown in the accompanying drawings are functional concepts and are not necessarily physically configured as shown in the drawings. In other words, the specific patterns of device distribution and integration are not limited to those shown in the accompanying drawings, and all or some of the devices in the device can be configured to be functionally or physically distributed or integrated in any unit, depending on various loads, usage conditions, etc.

[0319] Furthermore, each of the above-described steps in the information processing method of the embodiments of this disclosure does not necessarily have to be processed in the order described above. For example, the order of the steps in the processing can be appropriately changed. Furthermore, each of the steps in the processing can be executed in partial parallelism or individually, rather than in a time-series manner. Moreover, the processing of each step does not necessarily have to be performed according to the above method, but can be performed by another functional unit using another method.

[0320] Although preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It will be apparent to those skilled in the art that various changes or modifications can be conceived within the scope of the technical concept described in the claims, and it should be understood that these naturally also fall within the technical scope of the present disclosure.

[0321] Furthermore, the effects described in this specification are merely illustrative or exemplary, and not restrictive. That is, the technology based on this disclosure may exhibit other effects, in conjunction with or in lieu of the effects described herein, that would be apparent to a person skilled in the art.

[0322] This technology can be configured as described below. (1)

[0324] An information processing system, comprising:

[0325] A gaze detection unit is configured to detect the position of a second user's gaze relative to a second user-side terminal used by the second user, the second user-side terminal sharing a demonstration screen displayed on a first user-side terminal used by a first user;

[0326] A relative coordinate transformation unit, configured to calculate the position of the second user's line of sight as relative coordinates to the demonstration screen; and

[0327] The display control unit is configured to display second user information on the first user-side terminal based on the relative coordinates of the second user's line of sight. (2)

[0329] According to the information processing system described in (1), the relative coordinate transformation unit calculates the relative coordinates based on the coordinates of the position of the second user's line of sight relative to the display portion of the second user-side terminal and the coordinates of the area of ​​the demonstration screen in the display portion of the second user-side terminal. (3)

[0331] According to the information processing system described in (1) or (2), the display control unit displays the second user information as superimposed on at least a portion of the demonstration screen displayed on the first user-side terminal. (4)

[0333] According to the information processing system described in (3), the display control unit displays the position of the second user's line of sight. (5)

[0335] According to the information processing system described in (4), wherein

[0336] The display control unit performs the following operations:

[0337] When the second user's line of sight appears within the demonstration screen, the position of the second user's line of sight is displayed in a first display mode, and

[0338] When the position of the second user's gaze appears outside the demonstration screen, the position of the second user's gaze is displayed in a second display mode different from the first display mode. (6)

[0340] According to the information processing system described in (5), wherein

[0341] The display control unit performs the following operations:

[0342] The demonstration screen is divided into multiple units.

[0343] As the first display mode, the outer periphery of the unit displaying the position of the second user's gaze is shown, and

[0344] As a second display mode, a line segment is displayed that forms part of the outer periphery of a unit adjacent to the position of the second user's line of sight. (7)

[0346] According to the information processing system described in (4), wherein

[0347] The display control unit performs the following operations:

[0348] When the second user's gaze is within the demonstration screen, and when the second user's gaze is outside the demonstration screen but within a predetermined range from the demonstration screen, information about the position of the second user's gaze is displayed in a first display mode.

[0349] When the second user's gaze is outside the demonstration screen and beyond a predetermined range from the demonstration screen, information about the position of the second user's gaze is displayed in a second display mode, different from the first display mode. (8)

[0351] The information processing system according to (3) further includes an ideal gaze position estimator, which is configured to estimate the ideal position of the second user's gaze relative to the presentation screen, wherein...

[0352] The display control unit changes the display mode of the second user's line of sight position based on the positional relationship between the second user's line of sight position and the ideal position. (9)

[0354] According to the information processing system described in (8), the ideal line-of-sight position estimator estimates the ideal position based on at least one of the position of the first user's line of sight, the voice emitted by the first user, or the operation information of the first user. (10)

[0356] According to the information processing system described in (8) or (9), wherein

[0357] The display control unit performs the following operations:

[0358] When the second user's line of sight appears within the demonstration screen and the distance between the second user's line of sight and the ideal position is within a predetermined distance, the position of the second user's line of sight is displayed in the first display mode.

[0359] When the second user's gaze position appears outside the demonstration screen, the position of the second user's gaze is displayed in a second display mode different from the first display mode.

[0360] When the position of the second user's gaze appears in the demonstration screen and the distance between the position of the second user's gaze and the ideal position is equal to or greater than a predetermined distance, the position of the second user's gaze is displayed in a third display mode that is different from the first display mode and the second display mode. (11)

[0362] According to any one of (8) to (10) of the information processing system, when it is detected that the second user’s gaze has moved from outside the demonstration screen to inside the demonstration screen, the display control unit identifies and displays the content demonstrated by the first user based on the ideal position. (12)

[0364] The information processing system according to (8) further includes a cause estimator, which is configured to estimate the cause of the second user's gaze position appearing outside the demonstration screen when the position of the second user's gaze appears outside the demonstration screen, wherein...

[0365] The display control unit shows information about the estimated cause. (13)

[0367] According to the information processing system described in (12), the cause estimator estimates the reason why the position of the second user's gaze appears outside the demonstration screen based on at least one of the position of the second user's gaze, the ambient sound around the second user, or the operation information of the second user. (14)

[0369] According to the information processing system described in (12), when the reason why the second user's gaze appears outside the demonstration screen is that the second user is watching the first user's video, the display control unit displays information to the first user indicating that the second user is watching the first user's video. (15)

[0371] According to the information processing system described in (12), when the reason why the second user's line of sight appears outside the demonstration screen is that the second user is performing an input operation, the display control unit displays information to the first user indicating that the second user is performing the input operation. (16)

[0373] According to the information processing system described in (15), when the information input to the second user-side terminal through the input operation of the second user matches or is similar to the information identified based on the ideal location, the display control unit does not display information indicating that the second user is performing the input operation to the first user. (17)

[0375] The information processing system according to (13) further includes a calibration unit configured to perform calibration of the gaze detection unit based on the position of the second user's gaze and the second user's operation information on the second user-side terminal. (18)

[0377] An information processing apparatus, comprising:

[0378] A gaze detection unit is configured to detect the position of a second user's gaze relative to a second user-side terminal used by the second user, the second user-side terminal sharing a demonstration screen displayed on a first user-side terminal used by a first user;

[0379] A relative coordinate transformation unit, configured to calculate the position of the second user's line of sight as relative coordinates to the demonstration screen; and

[0380] The display control unit is configured to display second user information on the first user-side terminal based on the relative coordinates of the second user's line of sight. (19)

[0382] An information processing method, comprising:

[0383] The information processing device detects the position of the second user's line of sight relative to the second user-side terminal used by the second user, and the second user-side terminal shares the demonstration screen displayed on the first user-side terminal used by the first user;

[0384] The information processing device calculates the position of the second user's gaze as relative coordinates with respect to the demonstration screen; and

[0385] The information processing device displays the second user's information on the first user's terminal based on the relative coordinates of the second user's line of sight. (20)

[0387] A program for causing a computer to function as:

[0388] A gaze detection unit is configured to detect the position of a second user's gaze relative to a second user-side terminal used by the second user, the second user-side terminal sharing a demonstration screen displayed on a first user-side terminal used by a first user;

[0389] A relative coordinate transformation unit, configured to calculate the position of the second user's line of sight as relative coordinates to the demonstration screen; and

[0390] The display control unit is configured to display second user information on the first user-side terminal based on the relative coordinates of the second user's line of sight.

[0391] List of reference numerals

[0392] 10. Remote conferencing system

[0393] 20 Viewer-side devices

[0394] 40 Demonstrator-side equipment

[0395] 80 demonstrators

[0396] 90 viewers

[0397] 91 Ideal Location

[0398] Line of sight positions: 92, 92a, 92b, 600

[0399] Changes over time in 94, 96, 98a, and 98b

[0400] 100 presenters screen sharing mechanism

[0401] 200 Viewer Side Display

[0402] 202 Viewer gaze estimator

[0403] 204 Viewer Relative Coordinate Acquisition Unit

[0404] 210 Viewer gaze display control unit

[0405] 212 State Estimator

[0406] 214 Viewer Operation Recognition Unit

[0407] 216 Calibration Unit

[0408] 220 audio output unit

[0409] 300 Presenter-side Display Section

[0410] 302 Viewer Gaze Visualization Unit

[0411] 304 Ideal line-of-sight position estimator

[0412] 306 Drawing Unit

[0413] 308 Presenter Shared Data Recognition Unit

[0414] 310 Demonstrator Speech Recognition Unit

[0415] 312 Demonstrator Operation Recognition Unit

[0416] 314 Presenter gaze estimator

[0417] 350 grid

[0418] Unit 352

[0419] 354 line segments

[0420] 400a and 400b data images

[0421] 500, 500a, 500b, 550, 560, 960 cursors

[0422] Icons 562, 562a, 562b, 562c, 562d, and 562e

[0423] Buttons 700 and 702

[0424] 750a, 750b, 904 windows

[0425] 800, 800a, 800b, 800c Slides

[0426] 802, 804 Display Section

[0427] 900 frames

[0428] Image 902

[0429] 950 Video

[0430] 980 and 990 columns

Claims

1. An information processing system, comprising: A gaze detection unit is configured to detect the position of a second user's gaze relative to a second user-side terminal used by the second user, the second user-side terminal sharing a demonstration screen displayed on a first user-side terminal used by a first user; A relative coordinate transformation unit is configured to calculate the position of the second user's line of sight as a relative coordinate with respect to the demonstration screen; as well as The display control unit is configured to display second user information on the first user-side terminal based on the relative coordinates of the second user's line of sight.

2. The information processing system according to claim 1, wherein... The relative coordinate transformation unit calculates the relative coordinates based on the coordinates of the second user's line of sight relative to the display portion of the second user-side terminal and the coordinates of the area of ​​the demonstration screen in the display portion of the second user-side terminal.

3. The information processing system according to claim 1, wherein... The display control unit displays the second user information as an overlay on at least a portion of the demonstration screen displayed on the first user-side terminal.

4. The information processing system according to claim 3, wherein The display control unit displays the position of the second user's gaze.

5. The information processing system according to claim 4, wherein... The display control unit performs the following operations: When the second user's line of sight appears within the demonstration screen, the position of the second user's line of sight is displayed in a first display mode, and When the position of the second user's gaze appears outside the demonstration screen, the position of the second user's gaze is displayed in a second display mode different from the first display mode.

6. The information processing system according to claim 5, wherein... The display control unit performs the following operations: The demonstration screen is divided into multiple units. As the first display mode, the outer periphery of the unit displaying the position of the second user's gaze is shown, and As a second display mode, a line segment is displayed that forms part of the outer periphery of a unit adjacent to the position of the second user's line of sight.

7. The information processing system according to claim 4, wherein The display control unit performs the following operations: When the second user's gaze is within the demonstration screen, and when the second user's gaze is outside the demonstration screen but within a predetermined range from the demonstration screen, information about the position of the second user's gaze is displayed in a first display mode. When the second user's gaze is outside the demonstration screen and beyond a predetermined range from the demonstration screen, information about the position of the second user's gaze is displayed in a second display mode, different from the first display mode.

8. The information processing system of claim 3 further includes an ideal gaze position estimator, the ideal gaze position estimator being configured to estimate the ideal position of the second user's gaze relative to the presentation screen, wherein... The display control unit changes the display mode of the second user's line of sight position according to the positional relationship between the second user's line of sight position and the ideal position.

9. The information processing system according to claim 8, wherein The ideal gaze position estimator estimates the ideal position based on at least one of the following: the position of the first user's gaze, the voice emitted by the first user, or the operation information of the first user.

10. The information processing system according to claim 8, wherein The display control unit performs the following operations: When the second user's line of sight appears within the demonstration screen and the distance between the second user's line of sight and the ideal position is within a predetermined distance, the position of the second user's line of sight is displayed in the first display mode. When the second user's gaze position appears outside the demonstration screen, the position of the second user's gaze is displayed in a second display mode different from the first display mode. When the position of the second user's gaze appears in the demonstration screen and the distance between the position of the second user's gaze and the ideal position is equal to or greater than a predetermined distance, the position of the second user's gaze is displayed in a third display mode that is different from the first display mode and the second display mode.

11. The information processing system according to claim 8, wherein When the display control unit detects that the second user's gaze has moved from outside the demonstration screen to inside the demonstration screen, it identifies and displays the content demonstrated by the first user based on the ideal position.

12. The information processing system according to claim 8, further comprising a cause estimator, the cause estimator being configured to: when the position of the second user's gaze appears outside the demonstration screen, estimate the cause why the position of the second user's gaze appears outside the demonstration screen, wherein... The display control unit displays information about the estimated cause.

13. The information processing system according to claim 12, wherein... The cause estimator estimates the reason why the second user's gaze position appears outside the demonstration screen based on at least one of the second user's gaze position, ambient sounds around the second user, or the second user's operation information.

14. The information processing system according to claim 12, wherein... When the second user's gaze appears outside the demonstration screen because the second user is watching the first user's video, the display control unit displays information to the first user indicating that the second user is watching the first user's video.

15. The information processing system according to claim 12, wherein... The reason why the second user's line of sight appears outside the demonstration screen is that the second user is performing an input operation. At this time, the display control unit displays information to the first user indicating that the second user is performing the input operation.

16. The information processing system according to claim 15, wherein When the information input to the second user's terminal through the second user's input operation matches or is similar to the information identified based on the ideal location, the display control unit does not display information to the first user indicating that the second user is performing the input operation.

17. The information processing system of claim 13 further includes a calibration unit configured to perform calibration of the gaze detection unit based on the position of the second user's gaze and the second user's operation information on the second user-side terminal.

18. An information processing apparatus, comprising: A gaze detection unit is configured to detect the position of a second user's gaze relative to a second user-side terminal used by the second user, the second user-side terminal sharing a demonstration screen displayed on a first user-side terminal used by a first user; A relative coordinate transformation unit is configured to calculate the position of the second user's line of sight as a relative coordinate with respect to the demonstration screen; as well as The display control unit is configured to display second user information on the first user-side terminal based on the relative coordinates of the second user's line of sight.

19. An information processing method, comprising: The information processing device detects the position of the second user's line of sight relative to the second user-side terminal used by the second user, and the second user-side terminal shares the demonstration screen displayed on the first user-side terminal used by the first user; The information processing device calculates the position of the second user's gaze as a relative coordinate with respect to the demonstration screen; as well as The information processing device displays the second user's information on the first user's terminal based on the relative coordinates of the second user's line of sight.

20. A program that causes a computer to function as: A gaze detection unit is configured to detect the position of a second user's gaze relative to a second user-side terminal used by the second user, the second user-side terminal sharing a demonstration screen displayed on a first user-side terminal used by a first user; A relative coordinate transformation unit is configured to calculate the position of the second user's line of sight as a relative coordinate with respect to the demonstration screen; as well as The display control unit is configured to display second user information on the first user-side terminal based on the relative coordinates of the second user's line of sight.

Images