On-site participation system and on-site participation method

Abstract

To increase the on-site presence of online participants.SOLUTION: A control device displays an online participant video on a display screen on the basis of online participant gaze information so as to make the online participant appear to be looking in the on-site direction, extracts the video of the viewing direction from the omnidirectional on-site video on the basis of the online participant gaze information, and transmits the extracted on-site video for each online participant together with the on-site audio to the online participant's terminal. The online participant's terminal displays the on-site video for each online participant and outputs the on-site audio.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an on-site participation system and an on-site participation method.

Background Art

[0002] In the conference video display system described in Patent Document 1, a conference server arranges objects such as conference participants, conference tables, and materials to generate a video representing the state of the conference, and displays the video on a display device connected to the conference server. In response to requests from each participant terminal, videos obtained by subjecting the video to predetermined processing are generated for each participant terminal and transmitted to the terminal that received the request. The participant terminal receives the video transmitted from the conference server and displays it in the conference state window on the participant terminal screen. The virtual audience forming device described in Patent Document 2 synthesizes and outputs the live video at the shooting site and the viewer video of the viewer transmitted from the viewer side, and synthesizes and outputs the live audio at the shooting site and the viewer audio of the viewer transmitted from the viewer side.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, with the technologies described in Patent Documents 1 and 2 above, it is difficult for online participants who are participating online in a conference, lecture, etc. actually held on-site to feel a sense of presence.

[0005] The present invention has been made in consideration of such circumstances, and an object thereof is to improve the on-site presence of online participants.

Means for Solving the Problem

[0006] One aspect of the present invention includes an imaging unit capable of photographing in all directions, a display unit disposed on each side of a polygon or on a circumference such that a display screen can be photographed by the imaging unit and a plurality of the display screens face each other, a plurality of speakers provided close to each of the plurality of the display screens, a sound collection unit capable of collecting sound in all directions, terminals of each online participant, and a control device. The imaging unit, the display unit, the plurality of speakers, and the sound collection unit are provided on-site. The control device, for each online participant, based on online participant line-of-sight information indicating the viewing direction of the online participant within the on-site, causes an online participant video showing the online participant to be displayed on the display screen that makes it appear as if the online participant is looking in the viewing direction. The control device, for each online participant, causes the speaker close to the display screen on which the online participant video is displayed to output the voice of the online participant. The control device, for each online participant, based on the online participant line-of-sight information, extracts the video in the viewing direction from the all-directional on-site video photographed by the imaging unit, and transmits the on-site video for each online participant, which is the extracted video, together with the on-site voice collected by the sound collection unit to the terminal of the online participant. The terminal of the online participant displays the on-site video for each online participant and outputs the on-site voice. This is an on-site participation system. One aspect of the present invention is the above on-site participation system, wherein when there are a plurality of online participants in the same viewing direction, the control device displays the online participant videos of the plurality of online participants separately on adjacent display screens, and rotates the online participant videos so as to match the appearance of the online participants reflected on the display screens in the same viewing direction according to the orientations of the adjacent display screens, and then displays them on the display screens. This is an on-site participation system. One aspect of the present invention is an on-site participation system as described above, wherein the display unit further includes a plurality of the display screens arranged on each side of the polygon or on the circumference, outside the plurality of the display screens arranged on each side of the polygon or on the circumference, and the control device, when there are a plurality of online participants in the same viewing direction, divides and displays the online participant videos of the plurality of online participants in the same viewing direction on a plurality of the display screens so as to appear as if the online participants are looking in the same viewing direction. One aspect of the present invention is an on-site participation system as described above, wherein the control device transmits image data displayed on a display screen provided on the on-site separately from the display unit and capable of being photographed by the imaging unit to the terminals of each online participant, and the terminal of the online participant displays the image data in a semi-transparent display area that is superimposed on the display area of the display screen in the on-site in the on-site video for each online participant. One aspect of the present invention is an on-site participation system as described above, wherein the control device receives an online participant line-of-sight information change instruction for forcibly changing the online participant line-of-sight information, and changes the online participant line-of-sight information held by the control device according to the received online participant line-of-sight information change instruction. One aspect of the present invention is an on-site participation system as described above, wherein the display unit includes a separate rear-side display screen on the rear side for each of the plurality of the display screens arranged on each side of the polygon or on the circumference, and the control device displays an online participant rear video showing the back of the online participant on the rear-side display screen of the display screen on which the online participant video is displayed. One aspect of the present invention is an on-site participation system as described above, wherein a movable body including the imaging unit, the display unit, the plurality of speakers, and the sound collection unit and capable of being remotely controlled is provided.

[0007] One aspect of the present invention includes an imaging unit capable of capturing images in all directions, a display unit disposed on each side of a polygon or on the circumference so that a display screen can be captured by the imaging unit and a plurality of the display screens face each other, a plurality of speakers provided close to each of the plurality of display screens, a sound collection unit capable of collecting sound in all directions, terminals of each online participant, and a control device. The imaging unit, the display unit, the plurality of speakers, and the sound collection unit are provided on-site. The control device, for each online participant, based on online participant line-of-sight information indicating the viewing direction of the online participant within the on-site area, displays an online participant video showing the online participant on the display screen that makes it appear as if the online participant is looking in the viewing direction. The control device, for each online participant, outputs the voice of the online participant from the speaker close to the display screen on which the online participant video is displayed. The control device, for each online participant, based on the online participant line-of-sight information, extracts the video in the viewing direction from the all-directional on-site video captured by the imaging unit, and transmits the on-site video for each online participant, which is the extracted video, together with the on-site sound collected by the sound collection unit to the terminal of the online participant. The terminal of the online participant displays the on-site video for each online participant and outputs the on-site sound. This is an on-site participation method including these steps.

Advantages of the Invention

[0008] According to the present invention, an effect is obtained in that it is possible to improve the sense of presence of online participants on-site.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Embodiments for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 and FIG. 2 are diagrams showing a configuration example of an on-site participation system according to an embodiment. The on-site participation system 1 is a system for participating online in, for example, a meeting or a lecture actually held at the on-site (site) 100.

[0011] As an example of this embodiment, the case where a lecture is being held at the on-site 100 will be described. At the on-site 100, for example, the on-site participant US-k who is a lecturer is actually giving a lecture using the video projected on the projector screen 150. At the on-site 100, there are a plurality of on-site participants US-1, ···, US-m who are listeners. Also, a plurality of online participants UL-1, UL-2, ···, UL-i, ···, UL-n who are listeners are taking the lecture actually held at the on-site 100 online from each online participation location 200-1, 200-2, ···, 200-i, ···, 200-n.

[0012] As shown in FIG. 1, the on-site participation system 1 includes an imaging unit 110, a display unit 120, a plurality of speakers (speaker unit) 130, and a sound collection unit 140 provided at the on-site 100, a control device 300, and online participant terminals 201-1, 201-2, ···, 201-n used by each of the online participants UL-1, UL-2, ···, UL-n.

[0013] Hereinafter, when the online participants UL-1, UL-2, ···, UL-n are not particularly distinguished, they are referred to as the online participant UL. When the online participant terminals 201-1, 201-2, ···, 201-n are not particularly distinguished, they are referred to as the online participant terminal 201. When the online participation locations 200-1, 200-2, ···, 200-i, ···, 200-n are not particularly distinguished, they are referred to as the online participation location 200. Also, when the on-site participants US-1, ···, US-k, ···, US-m are not particularly distinguished, they are referred to as the on-site participant US.

[0014] The imaging unit 110 is configured to be able to photograph in all directions. The imaging unit 110 may be, for example, an omnidirectional camera (also called a 360-degree camera or a full-sphere camera) capable of photographing in all directions. The imaging unit 110 may be, for example, a plurality of cameras (such as web cameras) arranged on the same circumference. It is preferable that the imaging unit 110 can photograph the entire space of the on-site 100. The imaging unit 110 transmits the data of the photographed video to the control device 300.

[0015] The display unit 120 includes a plurality of display screens, and the plurality of display screens are arranged so as to be photographable by the imaging unit 110. Also, the plurality of display screens are arranged on each side of a polygon or on the circumference so as to face each other.

[0016] The plurality of speakers of the speaker unit 130 are provided close to each of the plurality of display screens of the display unit 120.

[0017] The sound collection unit 140 is configured to be able to collect omnidirectional voices. The sound collection unit 140 may be, for example, a 3D (three-dimensional) microphone. The sound collection unit 140 transmits the data of the collected voice to the control device 300.

[0018] The control device 300 controls the on-site participation system 1. The control device 300 communicates with the imaging unit 110, the display unit 120, the speaker unit 130, and the sound collection unit 140 via a communication line. Further, the control device 300 communicates with each online participant terminal 201 via a communication line.

[0019] The control device 300 includes computer hardware such as a CPU (Central Processing Unit) and a memory, and the functions of the control device 300 are realized by the CPU executing a computer program stored in the memory. Note that the control device 300 may be configured using a general-purpose computer device, or may be configured as a dedicated hardware device. For example, the control device 300 may be configured using a server computer connected to a communication network such as the Internet. Further, each function of the control device 300 may be realized by cloud computing. Also, the control device 300 may be realized by a single computer, or the functions of the control device 300 may be realized by being distributed among a plurality of computers. Further, the control device 300 may be configured to open a website using, for example, a WWW system or the like.

[0020] The online participant terminal 201 is equipped with computer hardware such as a CPU and memory. The functions of the online participant terminal 201 are realized by the CPU executing a computer program stored in the memory. As the online participant terminal 201, a general-purpose computer device may be used, or it may be configured as a dedicated hardware device. Also, as the online participant terminal 201, a mobile communication terminal device such as a smartphone, a tablet-type computer device (tablet PC), etc. may be utilized. Further, a head-mounted display (HMD) may be used as the online participant terminal 201-i.

[0021] FIG. 3 and FIG. 4 are diagrams showing configuration examples of the imaging unit 110, the display unit 120, the speaker unit 130, and the sound collection unit 140 according to the present embodiment. FIG. 3 is a perspective view, and FIG. 4 is a top view. In the configuration examples of FIGS. 3 and 4, the display unit 120 is arranged on each side of a hexagon such that the display screens 120-1, 120-2, ···, 120-6 of six flat panel displays face each other.

[0022] The display screens 120-1, 120-2, ···, 120-6 are arranged at positions where they can be visually recognized by the on-site participant US at the on-site 100. For example, when the on-site 100 is a lecture hall with seats arranged in a stepped manner, the display screens 120-1, 120-2, ···, 120-6 are arranged at a lower position in front of the frontmost seat as shown in FIG. 2. Also, the display screens 120-1, 120-2, ···, 120-6 are installed upward so that the on-site participant US can visually recognize them from above.

[0023] Also, the six speakers 130-1, 130-2, ···, 130-6 of the speaker unit 130 are provided in the vicinity of each of the six display screens 120-1, 120-2, ···, 120-6.

[0024] In addition, the omnidirectional camera 110 as the imaging unit 110 is disposed at the central portion of a hexagon where the display screens 120-1, 120-2, ···, 120-6 are arranged. Also, the omnidirectional camera 110 is disposed at a position higher than the display screens 120-1, 120-2, ···, 120-6 so that the display screens 120-1, 120-2, ···, 120-6 and their surroundings can be photographed. The omnidirectional camera 110 is arranged so that it can photograph the display screens 120-1, 120-2, ···, 120-6 and at least all or part of the on-site participants US and the projector screen 150 in the space of the on-site 100. More preferably, the omnidirectional camera 110 is arranged so that it can photograph the display screens 120-1, 120-2, ···, 120-6 and the entire space of the on-site 100.

[0025] The 3D microphone 140 as the sound collection unit 140 is disposed at the central portion of a hexagon where the display screens 120-1, 120-2, ···, 120-6 are arranged, similar to the omnidirectional camera 110. Also, the 3D microphone 140 is arranged so that it can collect the sound output from the six speakers 130-1, 130-2, ···, 130-6 and the sound from its surroundings (such as the sound of on-site participants US).

[0026] Note that in the configuration examples of FIGS. 3 and 4, the display unit 120 has display screens arranged on each side of a hexagon, but it is not limited to this. For example, the display unit 120 may have display screens arranged on each side of a polygon having more sides than a hexagon. Also, as shown in FIG. 5, the display unit 120 may use a frustum of a cone-shaped projector screen (hemispherical shell). In the frustum of a cone-shaped projector screen, a plurality of display screens are arranged on the circumference so as to face each other.

[0027] Hereinafter, as shown in FIG. 2, a case where the imaging unit 110, the display unit 120, the speaker unit 130, and the sound collection unit 140 having the configurations shown in FIGS. 3 and 4 are provided at the on-site 100 will be described as an example.

[0028] For each online participant UL, the control device 300 causes the online participant video V indicating the online participant UL to be displayed on a display screen 120-l (where l is an integer from 1 to 6) such that the online participant UL appears to be looking in the viewing direction within the on-site 100 based on the online participant's line-of-sight information indicating the viewing direction within the on-site 100 of the online participant UL.

[0029] For each online participant UL, the control device 300 causes the speaker 130-l adjacent to the display screen 120-l on which the online participant video V is displayed to output the voice of the online participant UL. The voice data of the online participant UL is transmitted from the online participant terminal 201 to the control device 300.

[0030] For each online participant UL, based on the online participant's line-of-sight information, the control device 300 extracts the video in the viewing direction of the online participant UL from the omnidirectional on-site video captured by the omnidirectional camera 110, and transmits the on-site video for each online participant, which is the extracted video, together with the on-site voice picked up by the 3D microphone 140 to the online participant terminal 201 of the online participant.

[0031] The online participant terminal 201 of the online participant UL displays the on-site video for each online participant and outputs the on-site voice. As a result, as shown in FIG. 2, on the online participant terminal 201 of each online participant UL, the on-site video for each online participant, which is the on-site video in the viewing direction within the on-site 100 of the own online participant UL, is displayed and the on-site voice is output. Thereby, the online participant UL can view the video within the on-site 100 in the direction he / she wants to see and hear the three-dimensional voice within the on-site 100. Therefore, even in an online lecture, the lecture being held at the on-site 100 can be attended with a sense of presence.

[0032] The online participant's line-of-sight information may be set by the online participant UL using the online participant terminal 201 to set the viewing direction within the on-site 100. Alternatively, the online participant's line-of-sight information may be generated by the online participant terminal 201 detecting the line of sight of the online participant UL, estimating the viewing direction within the on-site 100 of the online participant UL, and generating the online participant's line-of-sight information from the estimated result. Also, the online participant's line-of-sight information of the online participant UL-i using the head-mounted display as the online participant terminal 201-i may be generated from the detected orientation of the head-mounted display by detecting the orientation of the head-mounted display. The online participant terminal 201 transmits the online participant's line-of-sight information to the control device 300.

[0033] The online participant video V may be a video of the online participant UL during online lectures taken by a camera provided at the online participation location 200. In this case, the online participant terminal 201 transmits the video of the online participant UL as the online participant video V together with the audio data of the online participant UL to the control device 300. The audio of the online participant UL is input to the online participant terminal 201 as the audio data of the online participant UL collected by a microphone provided at the online participation location 200.

[0034] Alternatively, the online participant video V may be an avatar image of the online participant UL generated by CG (computer graphics). For example, for the online participant UL-i using the head-mounted display as the online participant terminal 201-i, the avatar image of the online participant UL-i is used for the online participant video V. The avatar image of the online participant UL may be generated by the online participant terminal 201 and transmitted to the control device 300, or may be generated by the control device 300.

[0035] The display screen 120-l that appears to show the viewing direction within the on-site 100 of the online participant UL is previously set in the control device 300 with the association between the viewing direction within the on-site 100 and the display screen 120-l.

[0036] The association between the display screen 120-l and the speaker 130-l adjacent to the display screen 120-l is previously set in the control device 300.

[0037] The association between the orientation of the omnidirectional on-site video captured by the omnidirectional camera 110 and the orientation in the on-site 100 is previously set in the control device 300. The viewing direction within the on-site 100 indicated by the online participant's line-of-sight information is represented as the orientation in the on-site 100. Based on the association of the orientation, the control device 300 extracts the video of the viewing direction of the online participant UL (the on-site video for each online participant) from the omnidirectional on-site video captured by the omnidirectional camera 110.

[0038] In FIG. 2, the online participant UL-1 sets the direction of the projector screen 150, which is forward within the on-site 100, to the viewing direction E-1 within the on-site 100. Accordingly, the control device 300 causes the online participant video V-1 of the online participant UL-1 to be displayed on the display screen 120-1 so as to appear as if the online participant UL-1 is looking in the direction of the projector screen 150. Further, the control device 300 extracts the video in the direction of the projector screen 150, which is the viewing direction of the online participant UL-1, from the omnidirectional on-site video captured by the omnidirectional camera 110, and transmits the extracted on-site video for each online participant of the online participant UL-1 together with the on-site audio picked up by the 3D microphone 140 to the online participant terminal 201-1 of the online participant UL-1. Accordingly, on the online participant terminal 201-1 of the online participant UL-1, the video in the direction of the projector screen 150 is displayed and the on-site audio is output. In the example of FIG. 2, the video in the direction of the projector screen 150 shows the projector screen 150 within the on-site 100, the on-site participants US-k, the display screens 120-3 and 120-4 facing the display screen 120-1, and the like.

[0039] Also, in FIG. 2, the online participant UL-2 sets the side direction within the site 100 to the viewing direction E-2 within the site 100. Thereby, the control device 300 causes the online participant video V-2 of the online participant UL-2 to be displayed on the display screen 120-3 that makes it appear as if the online participant UL-2 is looking at the side direction within the site 100. Further, the control device 300 extracts the video of the side direction within the site 100, which is the viewing direction of the online participant UL-2, from the omnidirectional site video captured by the omnidirectional camera 110, and transmits the extracted site video for each online participant of the online participant UL-2 together with the site audio picked up by the 3D microphone 140 to the online participant terminal 201-2 of the online participant UL-2. Thereby, on the online participant terminal 201-2 of the online participant UL-2, the video of the side direction within the site 100 is displayed and the site audio is output. In the example of FIG. 2, the display screens 120-6, 120-1, etc. facing the display screen 120-3 within the site 100 are reflected in the video of the side direction within the site 100.

[0040] Also, in FIG. 2, the online participant UL-n sets the backward direction within the site 100 to the viewing direction E-n within the site 100. Thereby, the control device 300 causes the online participant video V-n of the online participant UL-n to be displayed on the display screen 120-4 so as to appear as if the online participant UL-n is looking backward within the site 100. Further, the control device 300 extracts the video in the backward direction within the site 100, which is the viewing direction of the online participant UL-n, from the omnidirectional site video captured by the omnidirectional camera 110, and transmits the extracted site video for each online participant of the online participant UL-n together with the site audio picked up by the 3D microphone 140 to the online participant terminal 201-n of the online participant UL-n. Thereby, on the online participant terminal 201-n of the online participant UL-n, the video in the backward direction within the site 100 is displayed and the site audio is output. In the example of FIG. 2, the video in the backward direction within the site 100 shows the site participants US-1, US-m within the site 100 and the display screens 120-6, 120-1, 120-2 facing the display screen 120-4, etc.

[0041] [Regarding the case where multiple online participants UL look in the same direction within the site] In the case where a lecture is being held at the site 100 as illustrated in FIG. 2, etc., it is highly possible that a large number of online participants UL set the same direction within the site 100 (the direction of the projector screen 150 in FIG. 2) as the viewing direction within the site 100. Then, the online participant videos V of a large number of online participants UL will be displayed on the same display screen 120-1, making it difficult to visually recognize the online participant videos V of each online participant UL. Examples 1-1, 1-2, and 1-3 of the countermeasures for this problem are shown below.

[0042] (Example of countermeasure 1-1) In Example 1-1 of the countermeasure method, the display screen 120-1 is divided into a number of display areas, and in each display area, the online participant video V of each online participant UL with the same viewing direction within the on-site 100 is displayed. Since the size of the display is limited, this Example 1-1 of the countermeasure method is preferably used when the number of online participants UL is small.

[0043] As shown in FIG. 6, the online participant videos V of a plurality of online participants UL, where the direction of the projector screen 150 is the viewing direction E-1 within the on-site 100, are separately displayed in a plurality of divided display areas within the display screen 120-1.

[0044] In addition, for the display unit 120 used when there is a high possibility that a large number of online participants UL, such as when a lecture is held at the on-site 100, set the same direction as the viewing direction within the on-site 100, a display with a large display screen is used. Thereby, on one display screen, it is possible to divide into more display areas and display the online participant videos V of more online participants UL.

[0045] (Example 1-2 of the countermeasure method) In Example 1-2 of the countermeasure method, the online participant videos V of a plurality of online participants UL with the same viewing direction within the on-site 100 are displayed separately on a plurality of adjacent display screens. At this time, since the directions of each of the plurality of adjacent display screens are different, the viewing directions within the on-site 100 are different for each display screen. Therefore, according to the direction of each of the plurality of adjacent display screens, the online participant video V is rotated and displayed on the display screen so that the appearance of the online participant UL reflected on the display screen is adjusted to the same viewing direction within the on-site 100.

[0046] As shown in FIG. 6, online participant videos V of a plurality of online participants UL, in which the direction of the projector screen 150 is the viewing direction E-1 within the on-site 100, are separately displayed on adjacent display screens 120-1 and 120-2. Since the display screen 120-1 corresponds to the viewing direction E-1 within the on-site 100, the online participant video V-1 of the online participant UL-1 displayed on the display screen 120-1 is displayed as if the online participant UL-1 is looking in the viewing direction E-1 within the on-site 100.

[0047] On the other hand, since the display screen 120-2 has a different orientation from the display screen 120-1, even if the online participant video V-j of the online participant UL-j is directly displayed on the display screen 120-2, it is displayed as if the online participant UL-j is looking in a direction different from the viewing direction E-1 within the on-site 100. For this reason, according to the orientation of the display screen 120-2, the online participant video V-j is rotated and displayed on the display screen 120-2 so that the appearance of the online participant UL-j reflected on the display screen 120-2 matches the viewing direction E-1 within the on-site 100. Here, since the display screen 120-1 and the display screen 120-2 have different orientations by 60 degrees, the online participant video V-j is rotated by 60 degrees and displayed on the display screen 120-2 so as to match the orientation of the display screen 120-1. Thereby, the online participant video V-j of the online participant UL-j displayed on the display screen 120-2 is displayed as if the online participant UL-j is looking in the viewing direction E-1 within the on-site 100.

[0048] (Example 1-3 of the countermeasure method) In Example 1-3 of the countermeasure method, as shown in FIG. 6, outside the six display screens (first layer) arranged on each side of the hexagon, there are further twelve display screens (second layer) arranged on each side of the dodecagon. Then, the online participant videos V of a plurality of online participants UL, in which the direction of the projector screen 150 is the viewing direction E-1 within the on-site 100, are separately displayed on the display screens 120-1 of the first layer and the display screens 120-21, 120-22 of the second layer corresponding to the viewing direction E-1 within the on-site 100.

[0049] Note that, as shown in FIG. 6, the above-described Example 1-1, 1-2, and 1-3 of the countermeasure method may be used in combination, or any one or more of Example 1-1, 1-2, and 1-3 of the countermeasure method may be used.

[0050] [Regarding the method for outputting the voice of online participants] When outputting the voice of the online participant UL, the voice of the online participant UL is output from the speaker 130-l close to the display screen 120-l that displays the online participant video V of the online participant UL. In FIG. 4, when outputting the voice of the online participant UL-2, the voice of the online participant UL-2 is output from the two speakers 130-2 and 130-3 close to the display screen 120-3 that displays the online participant video V-2 of the online participant UL-2. At this time, the volume of the voice of the online participant UL-2 is inversely proportional to the distance between the display screen 120-3 and each of the speakers 130-2 and 130-3. As a result, the on-site participant US can feel the voice from the online participant UL-2 (the speaker) on the online participant video V-2.

[0051] Note that even for an online participant UL using 3D headphones, by adjusting the volume of each channel of the 3D headphones according to the positional relationship between the display screen and each speaker, the position of the voice source can be perceived.

[0052] [Regarding the semi-transparent virtual screen on the online participant terminal] In FIG. 7, the projector screen 150 provided at the on-site 100 can be photographed by the imaging unit 110. Image data SV-A is displayed on the projector screen 150. The control device 300 transmits the image data SV-A displayed on the projector screen 150 to the online participant terminal 201 of each online participant UL.

[0053] In FIG. 7, the online participant terminal 201-1 of the online participant UL-1 displays the on-site video SV-1 of the online participant UL-1 different from other online participants. The on-site video SV-1 of the online participant UL-1 different from other online participants shows the projector screen 150 within the on-site 100. The online participant terminal 201-1 displays a semi-transparent display area (referred to as a semi-transparent virtual screen for convenience of explanation) 201v-1 superimposed on the display area of the projector screen 150 within the on-site video SV-1 of the online participant UL-1 different from other online participants. The online participant terminal 201-1 displays the image data SV-A on the semi-transparent virtual screen 201v-1.

[0054] Although the projector screen 150 within the on-site 100 is shown in the on-site video SV-1 of the online participant UL-1 different from other online participants, in the on-site video SV-1 displayed on the display screen of the online participant terminal 201-1, the image data SV-A displayed on the projector screen 150 within the on-site 100 may be unclear and unrecognizable. As a countermeasure to this problem, the image data SV-A itself is displayed on the display screen of the online participant terminal 201-1. Thereby, it becomes easier for the online participant UL-1 to grasp the image data SV-A displayed on the projector screen 150 within the on-site 100. Also, by displaying the image data SV-A on the semi-transparent virtual screen 201v-1 provided on the display screen of the online participant terminal 201, the online participant UL-1 can also view the on-site video SV-1 of the online participant UL-1 different from other online participants under which the semi-transparent virtual screen 201v-1 is superimposed. For example, since the actions of the on-site participant US-k who is the instructor can be seen through the semi-transparent virtual screen 201v-1, the online participant UL-1 can easily grasp the actions such as the pointing of the on-site participant US-k.

[0055] [Regarding the function of forcibly changing the viewing direction within the on-site of the online participant] There may be a situation where an on-site participant US (such as an instructor) wants an online participant UL to look in a certain direction. At this time, even if the online participant UL is requested to change the viewing direction setting within the on-site area, it may not be possible to get the setting changed. Therefore, a function is provided to forcibly change the viewing direction of the online participant within the on-site area.

[0056] The control device 300 receives an online participant line-of-sight information change instruction for forcibly changing the online participant line-of-sight information, and changes the online participant line-of-sight information held by the control device 300 according to the received online participant line-of-sight information change instruction.

[0057] In FIG. 8, the on-site participant US-k who is an instructor performs an operation to forcibly change the viewing direction of the online participant UL-2 within the on-site area 100 so that the online participant UL-2, who has set the side direction within the on-site area 100 to the viewing direction E-2 within the on-site area 100 (see FIG. 2), looks at the direction of the projector screen 150. In the example of FIG. 8, the operation is a drag & drop. In this case, a touch screen is used for the display unit 120.

[0058] The on-site participant US-k drags and drops the online participant video V-2 of the online participant UL-2 displayed on the display screen 120-3 to the display screen 120-2. The display unit 120 transmits a drag & drop operation signal indicating the drag & drop operation to the control device 300. The control device 300 receives the drag & drop operation signal as an online participant line-of-sight information change instruction. The control device 300 changes the online participant line-of-sight information of the online participant UL-2, which was displayed on the original display screen 120-3 among the online participant line-of-sight information it holds, to the online participant line-of-sight information indicating the viewing direction within the on-site area 100 associated with the destination display screen 120-2 based on the drag & drop operation signal.

[0059] With this change in the online participant's line-of-sight information, the online participant video V-2 of the online participant UL-2 that was being displayed on the display screen 120-3 is now displayed on the display screen 120-2, and the off-site video of the online participant UL-2 from a different online participant is changed to the video in the viewing direction (the direction of the projector screen 150) within the off-site 100 that is associated with the display screen 120-2 and is then displayed on the online participant terminal 201 of the online participant UL-2. As a result, the video within the off-site 100 that the online participant UL-2 views on the online participant terminal 201 is forcibly switched from the video in the side direction within the off-site 100 to the video in the direction of the projector screen 150.

[0060] Note that the operation for forcibly changing the viewing direction of the online participant UL within the off-site 100 is not limited to the above-described drag & drop. For example, the control device 300 determines the pointing direction of the off-site participant US-k who is the instructor based on the video captured by the imaging unit 110, and changes the online participant's line-of-sight information of the online participant UL to be changed to the online participant's line-of-sight information indicating the determined pointing direction. For example, when it is desired for all of the online participants UL to look in the same direction within the off-site 100, the online participant's line-of-sight information of all of the online participants UL is changed to the online participant's line-of-sight information indicating the determined pointing direction.

[0061] [Regarding the Rear-Side Display] Since the multiple display screens 120-l of the display unit 120 face each other, it is considered difficult for the off-site participant US to see. Therefore, as shown in FIG. 9, a separate display screen (rear-side display screen) 120-lb is provided on the rear side of each display screen 120-l. In the example of FIG. 9, the display screens 120-1, 120-2, ···, 120-6 of six flat-panel displays are arranged on each side of a hexagon so as to face each other. On the rear side of each of the display screens 120-1, 120-2, ···, 120-6, six more flat-panel display screens (rear-side display screens) 120-1b, 120-2b, ···, 120-6b are arranged facing outward.

[0062] The control device 300 displays an online participant rear image showing the back of the online participant UL on the back display screen 120-lb of the display screen 120-l on which the online participant video V is displayed. The online participant rear image may be an image of the actual back of the online participant UL, or may be an avatar image of the back of the online participant UL generated by CG. As a result, a plurality of on-site participants US-1, ···, US-m who are learners and the on-site participant US-k who is an instructor can easily recognize the presence of the on-site participant US from the online participant rear image displayed on the back display screen 120-lb.

[0063] [Regarding the mobile robot] FIG. 10 shows a mobile robot (mobile body) 400 that can move within the on-site 100. The mobile robot 400 is configured to be movable and remotely controllable, and includes an imaging unit 110, a display unit 120, a plurality of speakers (speaker unit) 130, and a sound collection unit 140.

[0064] In FIG. 10, the mobile robot 400 includes a traveling unit 410 and a mounting unit 420. The traveling unit 410 travels automatically within the on-site 100 by remote control. The mounting unit 420 is fixed to the traveling unit 410 and has a structure for mounting the imaging unit 110, the display unit 120, the speaker unit 130, and the sound collection unit 140. In the example of FIG. 10, the imaging unit 110, the display unit 120, the speaker unit 130, and the sound collection unit 140 having the configuration shown in FIG. 6 are mounted on the mounting unit 420. As a result, the imaging unit 110, the display unit 120, the speaker unit 130, and the sound collection unit 140 move within the on-site 100 by the automatic traveling of the traveling unit 410. For example, an arbitrary online participant UL can change the positions of the imaging unit 110, the display unit 120, the speaker unit 130, and the sound collection unit 140 within the on-site 100 by remotely controlling the mobile robot 400.

[0065] As described above, according to this embodiment, the video of the online participant UL is displayed on the display screen 120-l within the on-site 100, and the inside of the on-site 100 including the display screen 120-l within the on-site 100 is displayed on the online participant terminal 201 of the online participant UL. Thereby, the effect of improving the sense of presence of the online participant UL in the on-site 100 can be obtained.

[0066] As described above, the embodiments of the present invention have been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and design changes and the like within the scope not departing from the gist of the present invention are also included.

[0067] Further, a computer program for realizing the functions of each of the above-described devices may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed. Here, the "computer system" may include hardware such as an OS and peripheral devices. Also, the "computer system" shall include a homepage providing environment (or display environment) if the WWW system is used. The "computer-readable recording medium" refers to a writable non-volatile memory such as a flexible disk, a magneto-optical disk, a ROM, a flash memory, a portable medium such as a DVD (Digital Versatile Disc), or a storage device such as a hard disk built into a computer system.

[0068] Furthermore, the "computer-readable recording medium" also includes a volatile memory (for example, DRAM (Dynamic Random Access Memory)) inside a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and that holds the program for a certain period of time. Also, the above program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the "transmission medium" for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) like the Internet or a communication line (communication wire) like a telephone line. Also, the above program may be for realizing a part of the functions described above. Further, it may be a so-called difference file (difference program) that can realize the functions described above in combination with a program already recorded in the computer system.

Explanation of Signs

[0069] 1... On-site participation system, 100... On-site, 110... Imaging unit, 120... Display unit, 130... Speaker unit, 140... Microphone unit, 201-1, 201-2,..., 201-n... Online participant terminals, 300... Control device

Claims

1. An imaging unit capable of photographing in all directions, A display unit that can photograph a display screen by the imaging unit and is arranged on each side of a polygon or on the circumference so that a plurality of the display screens face each other, A plurality of speakers provided close to each of the plurality of display screens, A sound collection unit capable of collecting sound in all directions, The terminals of each online participant, A control device, and comprises: The imaging unit, the display unit, the plurality of speakers, and the sound collection unit are provided on-site, The control device, for each online participant, based on the online participant line-of-sight information indicating the viewing direction of the online participant within the on-site, displays an online participant video showing the online participant on the display screen that makes it appear as if the online participant is looking in the viewing direction, The control device, for each online participant, outputs the voice of the online participant from the speaker close to the display screen on which the online participant video is displayed, The control device, for each online participant, based on the online participant line-of-sight information, extracts the video in the viewing direction from the all-round on-site video photographed by the imaging unit, and transmits the on-site video for each online participant, which is the extracted video, together with the on-site sound collected by the sound collection unit to the terminal of the online participant, The terminal of the online participant displays the on-site video for each online participant and outputs the on-site sound. An on-site participation system.

2. The control device, When there are a plurality of online participants in the same viewing direction, Displays the online participant videos of the plurality of online participants separately on a plurality of adjacent display screens, Rotates the online participant video and displays it on the display screen so that the appearance of the online participant reflected on the display screen is adjusted to the same viewing direction according to the orientation of each of the plurality of adjacent display screens. The on-site participation system according to Claim 1.

3. The display unit further includes a plurality of display screens arranged on each side of a polygon or on the circumference outside the plurality of display screens arranged on each side of the polygon or on the circumference, The control device, When there are a plurality of online participants in the same viewing direction, The online participant videos of the plurality of online participants in the same viewing direction are separately displayed on a plurality of display screens that make the online participants appear to be looking in the same viewing direction. The on-site participation system according to claim 1.

4. The control device transmits, to each online participant's terminal, image data to be displayed on a display screen provided on-site separately from the display unit and within the on-site area that can be photographed by the imaging unit. The online participant's terminal displays the image data in a semi-transparent display area that is superimposed on the display area of the on-site display screen within the on-site video for each online participant. The on-site participation system according to claim 1.

5. The control device receives an online participant's line-of-sight information change instruction that forcibly changes the online participant's line-of-sight information, and changes the online participant's line-of-sight information held by the control device according to the received online participant's line-of-sight information change instruction. The on-site participation system according to claim 1.

6. The display unit includes a separate rear-side display screen on the rear side for each of the plurality of display screens arranged on each side of the polygon or on the circumference. The control device displays an online participant's rear image showing the back of the online participant on the rear-side display screen of the display screen on which the online participant video is displayed. The on-site participation system according to claim 1.

7. An imaging unit, a display unit, a plurality of speakers, and a sound collection unit are provided on a movable body that can be remotely controlled. The on-site participation system according to claim 1.

8. An imaging unit capable of photographing in all directions, A display unit provided on each side of a polygon or on the circumference such that the imaging unit can photograph the display screen and the plurality of display screens face each other, A plurality of speakers provided close to each of the plurality of display screens, A sound collection unit capable of collecting sound from all directions, Terminals of each online participant, And a control device, The imaging unit, the display unit, the plurality of speakers, and the sound collection unit are provided on-site. The step in which the control device, for each online participant, causes an online participant video showing the online participant to be displayed on the display screen so as to appear as if the online participant is looking in the viewing direction, based on the online participant line-of-sight information indicating the viewing direction of the online participant within the on-site area for each online participant; The step in which the control device, for each online participant, causes the speaker proximate to the display screen on which the online participant video is displayed to output the voice of the online participant; The step in which the control device, for each online participant, extracts the video in the viewing direction from the omnidirectional on-site video captured by the imaging unit based on the online participant line-of-sight information, and transmits the on-site video for each online participant, which is the extracted video, together with the on-site voice collected by the sound collection unit to the terminal of the online participant; The step in which the terminal of the online participant displays the on-site video for each online participant and outputs the on-site voice; An on-site participation method including the above.

Images