Conference system, server, information processing device and program

Abstract

To allow participants participating in a remote conference to grasp participants indirectly participating in the remote conference at different sites.SOLUTION: A conference system 1 connects to a plurality of information processing apparatuses 2a, 2b, 2c via a network 5 and provides a screen for a conference to the plurality of information processing apparatuses 2a, 2b, 2c, and the conference system comprises: a first participant information acquisition unit 15 that acquires first participant information corresponding to first participants who use the plurality of information processing apparatuses 2a, 2b, 2c to participate in a remote conference; a second participant information acquisition unit 16 that acquires second participant information on second participants who indirectly participate in the remote conference together with the first participants; and a display control unit 17 that displays the first participant information in a predetermined display area on the screen for a conference and displays the second participant information in a predetermined display area on the screen for a conference.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a conference system, a server, an information processing device, and a program, and more particularly to a technology for supporting a remote conference with multiple participants. [Background technology]

[0002] Videoconferencing systems are known that directly connect multiple locations via a network and hold videoconferences between those locations. This type of videoconferencing system is configured to install a camera and a screen in the conference room of each location, broadcast images of the scenery inside the conference room captured by the camera to other locations, and display images of scenery captured in other conference rooms on the screen. Therefore, conference participants can understand what is happening in other conference rooms by viewing the images displayed on the screen.

[0003] The problem with such video conference systems is that when a participant in another conference room speaks, it is difficult to tell who is speaking from the image displayed on the screen. To address this problem, a conference system has been proposed in which, when a microphone installed in the conference room detects sound, the speaker is identified from the directionality of the sound, and a marking is added to the image of the participant identified as the speaker among multiple participants included in an image captured by a camera and the image is displayed (see, for example, Patent Document 1).

[0004] Meanwhile, in recent years, with the spread of telecommuting and teleworking, an environment is being developed in which multiple information processing devices, such as personal computers and tablet devices, are interconnected via a remote conference server, allowing multiple users to hold remote conferences using these information processing devices. Currently, participants in a remote conference log in to the server using their own information processing device to connect to a remote conference platform provided by the server. When the information processing device connects to the remote conference platform, it activates the camera built into the information processing device and sends images of the participants to the server. The server distributes the images to each information processing device, allowing images of all participants who have logged in to the server and are participating in the remote conference to be displayed on each information processing device. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-165565 Summary of the Invention [Problem to be solved by the invention]

[0006] In a remote conference held by connecting multiple information processing devices to a remote conference server as described above, a conference room may be used at at least one location, with multiple participants gathering in that conference room. In this case, if at least one of the participants brings an information processing device into the conference room, the other participants can share and view the screen displayed on that information processing device, allowing them to understand the other participants participating in the remote conference. Therefore, it is not necessary for all participants participating in the remote conference in the conference room to bring an information processing device into the conference room.

[0007] However, the information processing devices of participants participating in a remote conference at a location other than the conference room do not display all of the participants participating in the conference room. Therefore, participants participating in a remote conference at a location other than the conference room have a problem in that they cannot know who is participating in the conference room. Also, when one participant among multiple participants participating in the conference room speaks, participants participating in the remote conference at a location other than the conference room have a problem in that they cannot know who is speaking.

[0008] However, the conventional video conferencing system described above does not identify each of the multiple participants in the conference room. Furthermore, the conventional video conferencing system cannot connect to a remote conference platform provided by a server. Therefore, the conventional video conferencing system does not solve the above problem.

[0009] Therefore, the present invention has been made to solve the above-mentioned problems, and aims to provide a conference system, server, information processing device, and program that allows participants participating in a remote conference in a location other than the conference room to understand the participants participating in the conference room, thereby enabling the remote conference to be conducted smoothly. [Means for solving the problem]

[0010] In order to achieve the above object, the invention according to claim 1 provides a method for controlling a plurality of information processing devices via a network. to the server Connect, The server A conference system that provides a conference screen to each of the plurality of information processing devices, Each of the plurality of information processing devices comprises a first participant information generation unit that generates a first participant image representing a first participant participating in the remote conference, and transmits the first participant information, in which first identification information that can identify the first participant is added to the first participant image, to the server; and a first audio acquisition unit that acquires first audio information corresponding to audio uttered by the first participant, adds the first identification information to the first audio information, and transmits the resulting information to the server. At least one of the plurality of information processing devices further comprises a second participant information generation unit that generates a second participant image representing a second participant who indirectly participates in the remote conference together with the first participant, and transmits second participant information, in which second identification information that can identify the second participant is added to the second participant image, to the server; and a second audio acquisition unit that acquires second audio information corresponding to audio uttered by the second participant, adds the second identification information to the second audio information, and transmits the resulting information to the server. Each of the plurality of information processing devices The above sent from a first participant information acquisition means for acquiring first participant information; The information transmitted from at least one of the plurality of information processing devices a second participant information acquiring means for acquiring second participant information; and a first participant information acquiring means for acquiring the first participant information. the first participant image included in in a predetermined display area of ​​the conference screen, and the second participant image included in and a display control means for displaying the image in the display area of ​​the conference screen. When the display control means receives audio information from an information processing device that is a sender of both the first participant information and the second participant information, the display control means highlights the first participant image or the second participant image displayed in the display area based on identification information added to the audio information. The configuration is characterized by the above.

[0012] Claim 2 The invention according to claim 1 In the conference system, Each of the plurality of information processing devices The system further includes a first photographing means for photographing the first participant, and the first participant information Generation part the first participant image is acquired based on a first photographed image acquired by the first photographing means. Generate This is a configuration characterized by the above.

[0013] Claim 3 The invention according to claim 1 or 2 In the conference system, the display control means is configured to add the name of the first participant to the image of the first participant when displaying the image of the first participant in the display area.

[0015] Claim 4 The invention according to claim Any of 1 to 4 In the conference system, The information processing device that transmits both the first participant information and the second participant information includes: The second participant information is further provided with a second photographing means for photographing the second participant. Generation part the second participant image is captured based on a second captured image captured by the second image capturing means. Generate This is a configuration characterized by the above.

[0016] Claim 5 The invention according to claim Any of 1 to 4 In the conference system, the display control means is configured to add the name of the second participant to the image of the second participant when displaying the image of the second participant in the display area.

[0021] Claim 6 The invention according to claim 4 In the conference system, The information processing device that transmits both the first participant information and the second participant information is configured to: The second participant What was said byThe configuration is characterized by further comprising a speaker detection means for detecting a speaker.

[0022] Claim 7 The invention according to claim 6 In the conference system, the display control means detects a speaker by the speaker detection means. Statement by the second participant is detected, the second participant corresponding to the second participant detected as the speaker image is highlighted in the display area.

[0023] Claim 8 The invention according to claim 6 or 7 In the conference system, the speaker detection means Record number 2. By analyzing the facial image of the second participant based on the captured image of What the second participant said This is a configuration characterized by detecting the above.

[0024] Claim 9 The invention according to claim 6 ~ 8 In one of the conference systems, The information processing device that transmits both the first participant information and the second participant information includes: The present invention further comprises a voice input means installed near the second participant, and a voice switching means for controlling the on / off of the voice input means, and the voice switching means is configured to Statement by the second participant If detected, the statement but The configuration is characterized in that the voice input means installed near the detected second participant is switched on.

[0025] Claim 9 The invention according to claim 4 In the conference system, The information processing device that transmits both the first participant information and the second participant information is configured to: The configuration further comprises a gesture detection means for detecting that the second participant has made a hand raising gesture.

[0026] Claim 11 The invention according to claim 10In the conference system, when a hand-raising gesture is detected by the gesture detection means, the display control means displays a message of the second participant corresponding to the second participant whose hand-raising gesture is detected. image is highlighted in the display area.

[0027] Claim 12 The invention according to claim 10 or 11 In the conference system, when a hand-raising gesture is detected by the gesture detection means, the display control means displays a message of the second participant corresponding to the second participant whose hand-raising gesture is detected. image The configuration is characterized in that a raised hand icon is added to the displayed message.

[0029] The invention according to claim 13 is a server that supports a remote conference by connecting to a plurality of information processing devices via a network and providing a conference screen to each of the plurality of information processing devices, wherein a first participant who participates in the remote conference using the information processing device is notified from each of the plurality of information processing devices. and first identification information that can identify the first participant. a first participant information acquisition means for acquiring first participant information; From at least one of the plurality of information processing devices, a second participant indirectly participating in the remote conference together with the first participant; and second identification information that can identify the second participant. a second participant information acquiring means for acquiring second participant information; and a first participant information acquiring means for acquiring the first participant information. image in a predetermined display area of ​​the conference screen, and image and a display control means for displaying the image in the display area. When the display control means receives voice information from an information processing device that is a sender of both the first participant information and the second participant information, the display control means identifies whether the speaker of the voice information is the first participant or the second participant based on identification information added to the voice information, and highlights the first participant image or the second participant image displayed in the display area based on the identification result. The configuration is characterized by 。

[0039] Claim 14The present invention relates to an information processing device that is connected to a server via a network and that conducts a remote conference using a platform for remote conferences provided by the server, and a first participant who participates in the remote conference using the information processing device. and adding first identification information that can identify the first participant to the first participant image. a first participant information generating means for generating first participant information and transmitting the first participant information to the server; and a second participant who indirectly participates in the remote conference together with the first participant. and adding second identification information capable of identifying the second participant to the second participant image. a second participant information generating means for generating second participant information and transmitting the second participant information to the server; image and the second participant image a conference screen acquisition means for acquiring from the server a conference screen in which the above-mentioned screens are arranged in a predetermined display area; and a display means for displaying the conference screen acquired by the conference screen acquisition means. a voice acquisition unit that acquires voice information corresponding to a voice uttered by the first participant or the second participant and transmits the voice information to the server; Equipped with When the voice acquisition unit acquires voice information corresponding to the voice uttered by the first participant, it adds the first identification information to the voice information and transmits the voice information to the server, and when the voice acquisition unit acquires voice information corresponding to the voice uttered by the second participant, it adds the second identification information to the voice information and transmits the voice information to the server. This is a configuration characterized by the above.

[0040] Claim 15 The invention according to claim 14 The information processing device further comprises a first photographing control means for controlling a first photographing means for photographing the first participant, and a second photographing control means for controlling a second photographing means for photographing the second participant, and the first participant information generating means generates information about the first participant based on a first photographed image acquired by the first photographing control means controlling the first photographing means. image The second participant information generating means generates the second participant information based on a second photographed image acquired by the second photographing control means controlling the second photographing means. image The configuration is characterized by generating:

[0041] Claim 16 The invention according to claim 15 In the information processing device, the second participant What was said by and a speaker detection means for detecting the second participant and notifying the server of the detected speaker, the speaker detection means detecting the second participant by analyzing a face image of the second participant included in the second captured image. The second participant comment What I didThis is a configuration characterized by detecting the above.

[0042] Claim 17 The invention according to claim 15 or 16 The information processing device further comprises a gesture detection means for detecting that the second participant has made a hand-raising gesture and notifying the server, wherein the gesture detection means detects the hand-raising gesture by analyzing an image of the second participant included in the second captured image.

[0043] Claim 18 The present invention relates to a program executed on a server that supports a remote conference by connecting to a plurality of information processing devices via a network and providing a conference screen to each of the plurality of information processing devices, the program including: a program for transmitting to the server, from each of the plurality of information processing devices, a first participant who will participate in the remote conference using the information processing device; and first identification information that can identify the first participant. a first step of obtaining first participant information; From at least one of the plurality of information processing devices, a second participant indirectly participating in the remote conference together with the first participant; and second identification information that can identify the second participant. a second step of acquiring second participant information; and image in a predetermined display area of ​​the conference screen, and image a third step of displaying the image in the display area; In the third step, when audio information is acquired from an information processing device that is a sender of both the first participant information and the second participant information, the speaker of the audio information is identified as either the first participant or the second participant based on identification information added to the audio information, and the first participant image or the second participant image displayed in the display area is highlighted based on the identification result. The configuration is characterized by the above. [Effects of the Invention]

[0044] According to the present invention, even participants who are participating in a remote conference in a location other than the conference room can understand the participants who are participating in the conference room, allowing the remote conference to be conducted smoothly. [Brief explanation of the drawings]

[0045] [Figure 1]FIG. 1 is a diagram illustrating an example of a configuration of a conference system. [Figure 2] FIG. 2 is a diagram illustrating an example of information transmitted and received in the conference system. [Figure 3] FIG. 2 illustrates an example of the configuration of a server. [Figure 4] FIG. 1 illustrates an example of a configuration of an information processing device. [Figure 5] 1 is a first flowchart showing the operation of the conference system. [Figure 6] FIG. 10 is a diagram showing an example of a conference screen on which a first participant is displayed. [Figure 7] 10 is a second flowchart showing the operation of the conference system. [Figure 8] FIG. 10 is a diagram illustrating an example of face detection processing. [Figure 9] FIG. 10 is a diagram showing an example of a conference screen on which a second participant is additionally displayed. [Figure 10] 10 is a third flowchart showing the operation of the conference system. [Figure 11] FIG. 10 is a diagram showing an example of a conference screen in which a speaker is highlighted. [Figure 12] 10 is a fourth flowchart showing the operation of the conference system. [Figure 13] FIG. 10 is a diagram showing an example of a conference screen in which a second participant who has made a hand-raising gesture is highlighted. [Figure 14] 10 is a flowchart for generating a second participant image on the server. [Figure 15] 10 is a flowchart illustrating a process for detecting a speaker in a server. [Figure 16] 10 is a flowchart of detecting a raise hand gesture in a server. DETAILED DESCRIPTION OF THE INVENTION

[0046] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings. Elements common to the embodiments described below are designated by the same reference numerals, and redundant description thereof will be omitted.

[0047] (One embodiment of the invention) FIG. 1 is a diagram illustrating an example configuration of a conference system 1 according to an embodiment of the present invention. The conference system 1 connects multiple locations Pa, Pb, and Pc via a network 5, and supports participants at each location Pa, Pb, and Pc in holding a remote conference. The conference system 1 includes multiple information processing devices 2 (2a, 2b, and 2c) used by participants participating in the remote conference at each location Pa, Pb, and Pc, and a server 3. The server 3 is, for example, a server installed in a cloud on the Internet, and is communicatively connected to the information processing devices 2 at each location Pa, Pb, and Pc via the network 5, including the Internet. The information processing devices 2 at each location Pa, Pb, and Pc are indirectly connected to other information processing devices 2 via the server 3, enabling them to exchange information with the other information processing devices 2. While the present embodiment illustrates a case in which the number of locations connected to the server 3 is three, the number of locations connected to the server 3 is not limited to three and may be two, four, or more. Furthermore, the number of information processing devices 2 connected to the server 3 is not limited to three and may be two, four, or more.

[0048] The server 3 interconnects the plurality of information processing devices 2 via the network 5 and provides a service for remote conferences to each of the plurality of information processing devices 2. For example, the server 3 has a web server function that provides a remote conference platform, and provides a conference screen and audio during the conference to each of the plurality of information processing devices 2 connected to the remote conference platform. The conference screen has a function to display, for example, images showing the participants participating in the remote conference at each of the locations Pa, Pb, and Pc side by side, and a function to display document images designated for sharing by at least one participant.

[0049] The server 3 also includes a database 4. The database 4 is a storage means for storing remote conference reservation information, information for authenticating remote conference participants, etc. The database 4 may be provided at a location separate from the server 3 and connected to the server 3 via a network 5.

[0050] The information processing device 2 is a portable terminal device used by participants in a remote conference, and is configured, for example, as a personal computer, tablet terminal, or smartphone. This type of information processing device 2 generally includes a display unit, a camera, a microphone, and a speaker. The information processing device 2 connects to the server 3 via the network 5 and performs various processes to support the remote conference by using a remote conference platform provided by the server 3. For example, by connecting to the server 3 and activating the display unit, camera, microphone, and speaker, the information processing device 2 can display a conference screen provided by the server 3 and make calls with participants at other locations.

[0051] In this embodiment, as an example, at location Pa, participant A participates in the remote conference using his / her own information processing device 2a. At location Pb, participant B participates in the remote conference using his / her own information processing device 2b. At location Pc, participant C brings his / her own information processing device 2c into the conference room and participates in the remote conference using that information processing device 2c. Furthermore, in this embodiment, in addition to participant C, other participants D, E, and F also participate in the remote conference in the conference room at location Pc. However, participants D, E, and F do not bring their own information processing device 2 into the conference room, but instead participate in the remote conference indirectly by sharing the information processing device 2 used by participant C.

[0052] Furthermore, a camera 6 is installed in the conference room of the base PC. This camera 6 is a camera for capturing images of participants D, E, and F who indirectly participate in the remote conference in the conference room of the base PC. For example, the camera mounted on the information processing device 2c used by participant C is positioned to capture image of participant C, making it difficult to capture images of all of the other participants D, E, and F who are in the conference room. For this reason, in this embodiment, a camera 6 is installed in the conference room separately from the information processing device 2c to capture images of the other participants D, E, and F. This camera 6 is connected to, for example, the information processing device 2c, starts a capturing operation based on an instruction from the information processing device 2c, and outputs the captured image to the information processing device 2c.

[0053] Furthermore, a screen 7 and multiple microphones 8d, 8e, and 8f are installed in the conference room of the base PC. The screen 7 is a screen for displaying the same screen as the conference screen displayed on the information processing device 2c. For example, the screen 7 may be a display device such as a large screen display connected to the information processing device 2c, or may be a device on which the conference screen is projected from a projector such as a projector connected to the information processing device 2c. The microphones 8d, 8e, and 8f are individually assigned to each of participants D, E, and F who participate in the remote conference without bringing their own information processing device 2, and are audio input means installed near each of participants D, E, and F.

[0054] The conference system 1 having the above configuration recognizes participants A, B, and C who directly participate in the remote conference using information processing devices 2a, 2b, and 2c as first participants (direct participants) and supports the first participants to conduct the remote conference smoothly.The conference system 1 also recognizes participants D, E, and F who indirectly participate in the remote conference without using information processing devices 2a, 2b, and 2c as second participants (indirect participants) and treats these second participants in the same way as first participants, thereby supporting each of the first and second participants to conduct the remote conference smoothly.

[0055] For example, as shown in FIG. 2(a), when each of the information processing devices 2a, 2b, and 2c connects to the server 3, it generates first participant information D1 regarding participants A, B, and C (first participants) who will directly participate in the remote conference, and transmits it to the server 3. This first participant information D1 includes identification information that can identify the first participant. The identification information also includes the name of the first participant. The name may be a full name, initials, a nickname, or the like.

[0056] Furthermore, each of the information processing devices 2a, 2b, and 2c can include a first participant image representing participants A, B, and C (first participants) in the first participant information D1 and transmit it to the server 3. For example, if the photography function of the camera installed in each of the information processing devices 2a, 2b, and 2c is set to on, the information processing devices 2a, 2b, and 2c can acquire a first captured image (e.g., a moving image) of participants A, B, and C using the camera, generate a first participant image from the first captured image, and transmit the first participant information D1 including the first participant image to the server 3. However, the first participant image is not necessarily limited to an image generated from the first captured image.

[0057] The server 3 creates a conference screen to be displayed on each of the information processing devices 2a, 2b, and 2c based on the first participant information D1 acquired from each of the information processing devices 2a, 2b, and 2c, and provides screen information D3 related to the conference screen to each of the information processing devices 2a, 2b, and 2c. At this time, the server 3 arranges the first participant information D1 acquired from each of the information processing devices 2a, 2b, and 2c in a predetermined display area of ​​the conference screen. For example, the server 3 creates a conference screen in which first participant images representing participants A, B, and C, identification information of participants A, B, and C, etc. are arranged in the predetermined display area.

[0058] Furthermore, if the photographing function of the camera installed in each of the information processing devices 2a, 2b, and 2c is set to on, the first participant information D1 acquired from each of the information processing devices 2a, 2b, and 2c includes a first participant image based on a first photographed image of participants A, B, and C. Therefore, the server 3 creates a conference screen on which the first participant images are arranged and provides it to each of the information processing devices 2a, 2b, and 2c.

[0059] Furthermore, if the photography function of the camera installed in each information processing device 2a, 2b, 2c is turned off, the first participant information D1 acquired from each information processing device 2a, 2b, 2c does not include a first participant image based on the first photographed image of participants A, B, C. In this case, the server 3 uses a predetermined image (e.g., a still image) for each participant A, B, C as the first participant image representing participants A, B, C and arranges it on the conference screen. The server 3 then provides the conference screen on which the first participant image, such as a still image, is arranged to each information processing device 2a, 2b, 2c.

[0060] The information processing devices 2a, 2b, and 2c acquire screen information D3 related to the conference screen from the server 3 and display the conference screen based on the screen information D3. The conference screen displayed at this time includes first participant information D1 related to participants A, B, and C (first participants) who are directly participating in the remote conference using each of the information processing devices 2a, 2b, and 2c. Therefore, participants A, B, and C who are directly participating in the remote conference using their own information processing devices 2 can visually check the conference screen to know who is directly participating in the remote conference at another location.

[0061] As shown in FIG. 2(b), the information processing device 2c brought into the conference room by participant C generates second participant information D2 for participants D, E, and F (second participants) who indirectly participate in the remote conference and transmits the information processing device 2c to the server 3. This second participant information D2 includes identification information that can identify the second participant. For example, the identification information includes the name of the second participant. In this case, the name may be a full name, initials, a nickname, or the like. For example, the identification information may be manually entered into the information processing device 2c by participant C using the information processing device 2c. In this case, participant C manually enters the identification information for multiple participants D, E, and F individually. However, if the image capture function of the camera 6 connected to the information processing device 2c is turned on, the second participant information D2 does not necessarily include the identification information for participants D, E, and F, and participant C does not need to manually enter the identification information.

[0062] Furthermore, the information processing device 2c can include second participant images representing participants D, E, and F (second participants) in the second participant information D2 and transmit them to the server 3. For example, if the photographing function of the camera 6 connected to the information processing device 2c is set to on, the information processing device 2c can acquire second captured images (e.g., moving images) of participants D, E, and F captured by the camera 6, generate second participant images from the second captured images, and transmit the second participant information D2 including the second participant images to the server 3. However, the second participant images are not necessarily limited to images generated from the second captured images.

[0063] When the server 3 acquires the second participant information D2 for participants D, E, and F from the information processing device 2c, the server 3 creates a conference screen in which the second participant information D2 is arranged in a predetermined display area in the same manner as the first participant information D1, and updates the screen information D3. The server 3 then provides the updated screen information D3 to each of the information processing devices 2a, 2b, and 2c. At this time, the server 3 generates screen information D3 by adding second participant images representing participants D, E, and F (second participants) to the conference screen, and provides the screen information D3 to each of the information processing devices 2a, 2b, and 2c. For example, if the photographing function of the camera 6 connected to the information processing device 2c is turned on, the second participant information D2 acquired from the information processing device 2c includes second participant images based on second photographed images of participants D, E, and F. Therefore, the server 3 creates a conference screen in which the second participant images are arranged, and provides the conference screen to each of the information processing devices 2a, 2b, and 2c.

[0064] Furthermore, if the photographing function of the camera installed in the information processing device 2c is turned off, the second participant information D2 acquired from the information processing device 2c does not include a second participant image based on a second photographed image of the participants D, E, and F. In this case, the server 3 uses a predetermined image (e.g., a still image) for each of the participants D, E, and F as the second participant image representing the participants D, E, and F, and arranges it on the conference screen. Then, the server 3 provides the conference screen on which the second participant image, such as a still image, is arranged to each of the information processing devices 2a, 2b, and 2c.

[0065] Therefore, participants A and B participating in the remote conference at locations Pa and Pb different from the conference room at location Pc can see participants D, E, and F indirectly participating in the remote conference at location Pc by looking at the conference screens displayed on information processing devices 2a and 2b. Such a conference system 1 will be described in more detail below. Note that, hereinafter, when there is no need to distinguish between the information processing devices 2a, 2b, and 2c, they may be collectively referred to as information processing devices 2.

[0066] FIG. 3 is a diagram showing an example configuration of the server 3. The server 3 includes a remote conference platform 10 and a communication interface 11. The remote conference platform 10 is a software function realized by a hardware processor, such as a CPU, implemented in the server 3 executing a predetermined program. The remote conference platform 10 communicates with each of the multiple information processing devices 2a, 2b, and 2c, and performs processing to support the remote conference. The communication interface 11 connects the server 3 to the network 5 and is an interface for communicating with each of the multiple information processing devices 2a, 2b, and 2c.

[0067] The remote conference platform 10 sets an address for the remote conference (e.g., a URL (Uniform Resource Locator)) based on reservation information for the remote conference registered in advance in a database 4, and notifies each of the information processing devices 2a, 2b, and 2c registered in the reservation information of the address in advance. When the start time of the remote conference approaches, the remote conference platform 10 accepts access from each of the information processing devices 2a, 2b, and 2c, establishes a connection with each of the information processing devices 2a, 2b, and 2c, and starts processing to support the remote conference. The remote conference platform 10 includes a conference screen providing unit 12, an audio processing unit 13, and an identifying unit 14.

[0068] The conference screen providing unit 12 generates a conference screen to be shared by each of the multiple information processing devices 2 and provides the conference screen to each information processing device 2, thereby supporting information sharing between each location. The conference screen providing unit 12 includes a first participant information acquiring unit 15, a second participant information acquiring unit 16, and a display control unit 17.

[0069] The first participant information acquiring unit 15 acquires the first participant information D1 transmitted from each of the information processing devices 2 a, 2 b, and 2 c. Upon acquiring the first participant information D1, the first participant information acquiring unit 15 outputs the first participant information D1 to the display control unit 17.

[0070] When the photographing function of the camera mounted on the information processing device 2a, 2b, 2c is turned on, the first participant information acquisition unit 15 continuously acquires the first participant image generated based on the first photographed image from the information processing device 2a, 2b, 2c. Therefore, the first participant information acquisition unit 15 continuously outputs the first participant image acquired from the information processing device 2a, 2b, 2c to the display control unit 17.

[0071] The second participant information acquisition unit 16 acquires second participant information D2 corresponding to a second participant who indirectly participates in the remote conference together with the first participant from at least one information processing device 2c among the multiple information processing devices 2a, 2b, and 2c. Upon acquiring the second participant information D2, the second participant information acquisition unit 16 outputs the second participant information D2 to the display control unit 17. The second participant information acquisition unit 16 also outputs the second participant information D2 to the identification unit 14.

[0072] When the photographing function of the camera 6 connected to the information processing device 2c is turned on, the second participant information acquisition unit 16 continuously acquires the second participant image generated based on the second photographed image from the information processing device 2c. Therefore, the second participant information acquisition unit 16 continuously outputs the second participant image acquired from the information processing device 2c to the display control unit 17.

[0073] When the display control unit 17 acquires the first participant information D1 output from the first participant information acquisition unit 15, it creates a conference screen on which the first participant information D1 is arranged in a predetermined display area. Furthermore, when the display control unit 17 acquires the second participant information D2 output from the second participant information acquisition unit 16, it creates a conference screen on which the second participant information D2 is arranged in a predetermined display area. Then, the display control unit 17 provides the conference screen on which the first participant information D1 and the second participant information D2 are arranged in their predetermined display areas to each of the multiple information processing devices 2a, 2b, and 2c, for display.

[0074] 1, the display control unit 17 creates a conference screen in which first participant images representing participants A, B, and C are arranged in a predetermined display area based on first participant information D1 acquired from each of the information processing devices 2a, 2b, and 2c, and transmits the created conference screen to the information processing devices 2a, 2b, and 2c. As a result, a conference screen including multiple images representing participants A, B, and C is displayed on each of the information processing devices 2a, 2b, and 2c. At this time, if the imaging function of the camera installed in each of the information processing devices 2a, 2b, and 2c is turned on, real-time images (moving images) of participants A, B, and C are displayed on the conference screen.

[0075] Furthermore, when the display control unit 17 acquires the second participant information D2 corresponding to each of the participants D, E, and F from the information processing device 2c, it creates a conference screen in which second participant images representing each of the participants D, E, and F are arranged in a predetermined display area based on the second participant information D2, and transmits the created conference screen to the information processing devices 2a, 2b, and 2c. As a result, a conference screen including multiple images representing not only the participants A, B, and C who directly participate in the remote conference, but also the participants D, E, and F who indirectly participate in the remote conference is displayed on each of the information processing devices 2a, 2b, and 2c. At this time, if the imaging function of the camera 6 connected to the information processing device 2c is turned on, real-time images (moving images) of the participants D, E, and F are displayed on the conference screen.

[0076] The audio processing unit 13 processes audio for the conference. The audio processing unit 13 supports calls between the respective locations by distributing audio collected at one location to information processing devices 2 at other locations. For example, when the audio processing unit 13 acquires audio information from one information processing device 2 among a plurality of information processing devices 2, it transmits the audio information to the other information processing devices 2 excluding the one information processing device 2, causing audio output. This allows participants participating in the remote conference at other locations to hear the audio emitted at the one location.

[0077] When the identification unit 14 acquires second participant information D2 about a second participant from at least one information processing device 2 among the information processing devices 2a, 2b, and 2c, the identification unit 14 searches the database 4 based on the second participant information D2 to identify the second participant. For example, if the second participant information D2 includes a second participant image based on a second captured image, the identification unit 14 searches the database 4 based on the features of the facial images included in the second participant image to identify participants D, E, and F who are indirectly participating in the remote conference. That is, the identification unit 14 extracts facial images of participants D, E, and F from the second participant image and compares the features of the facial images extracted from the second captured image with feature information pre-registered in the database 4 to identify participants D, E, and F. At this time, the identification unit 14 can also identify the names of participants D, E, and F who are indirectly participating in the remote conference as second participants. Therefore, even if the second participant information D2 does not include identification information that can identify the second participant, the identification unit 14 can identify the name of the second participant.

[0078] Furthermore, if the second participant information D2 includes identification information of participants D, E, and F, the identification unit 14 identifies the names of participants D, E, and F based on the identification information included in the second participant information D2. Therefore, the identification unit 14 does not need to search the database 4 to identify the names of participants D, E, and F.

[0079] When the identification unit 14 identifies the second participant, it transmits the identification result to the information processing device 2c. Furthermore, when the identification unit 14 identifies the second participant, it also outputs the identification result to the conference screen providing unit 12.

[0080] 4 is a diagram showing an example of the configuration of the information processing device 2. The information processing devices 2a, 2b, and 2c used at the respective locations Pa, Pb, and Pc all have the same configuration. The information processing device 2 includes a control unit 20, a camera 21, a microphone 22, a speaker 23, a display unit 24, an operation input unit 25, a device connection unit 26, and a communication interface 27.

[0081] The control unit 20 is configured to include a hardware processor such as a CPU and a memory. The control unit 20 functions as a browser 30 and a cooperative application 40 by the hardware processor executing a predetermined program. The browser 30 is an application for browsing web pages, and by accessing the server 3, it acquires a conference screen provided as a web page from the server 3 and displays it on the display unit 24. The cooperative application 40 is an application that is started while the browser 30 is accessing the server 3, and cooperates with the server 3 and the browser 30 to execute processing for allowing a second participant to participate in the remote conference.

[0082] Camera 21 is an imaging means (first imaging means) mounted around display unit 24 of information processing device 2. For example, camera 21 can capture an image of a first participant who is trying to view the conference screen displayed on display unit 24.

[0083] The microphone 22 is a voice input means installed in the information processing device 2. For example, the microphone 22 inputs a voice uttered by a first participant who directly participates in a remote conference using the information processing device 2, and generates voice information.

[0084] The speaker 23 is an audio output means installed in the information processing device 2. For example, the speaker 23 outputs audio based on audio information provided from the server 3. Note that the speaker 23 may be headphones worn by the participants A, B, and C.

[0085] The display unit 24 is a display means configured with a color liquid crystal display, etc. For example, the display unit 24 can display a conference screen provided by the server 3 on the display unit 24.

[0086] The operation input unit 25 is an operation input means that accepts operation inputs from participants who use the information processing device 2. For example, the operation input unit 25 includes a keyboard, a mouse, a touch panel, and the like.

[0087] The device connection unit 26 is an interface for connecting an external device to the information processing device 2. For example, the device connection unit 26 is configured with a USB interface, a video output terminal, a video input terminal, an audio input / output terminal, etc. In the case of the information processing device 2c brought into the conference room of the base PC, the camera 6, microphones 8d, 8e, 8f, and a display device 9 are connected to the device connection unit 26. The display device 9 is a device for displaying a conference screen on a screen 7 installed in the conference room of the base PC.

[0088] The communication interface 27 is an interface for connecting the information processing device 2 to the network 5 and for communicating with the server 3.

[0089] The browser 30 establishes a connection with the server 3 and executes a program module provided by the server 3, thereby functioning as a camera control unit 31, a first participant information generation unit 32, an audio acquisition unit 33, a screen acquisition unit 34, a display processing unit 35, and an audio output unit 36.

[0090] The camera control unit 31 controls the photographing operation of the camera 21 installed in the information processing device 2. For example, when the photographing function of the camera 21 is set to on, the camera control unit 31 starts the photographing operation of the camera 21. Because the camera 21 is capable of photographing the first participant, the camera control unit 31 can acquire a first photographed image of the first participant by starting the photographing operation of the camera 21. For example, the camera control unit 31 acquires the first photographed image as a moving image. Then, the camera control unit 31 outputs the first photographed image to the first participant information generation unit 32. Note that when the photographing function of the camera 21 is off, the camera control unit 31 does not cause the camera 21 to photograph. In this case, the first photographed image is not acquired in the information processing device 2.

[0091] The first participant information generation unit 32 generates first participant information D1 regarding a first participant using the information processing device 2 and transmits the first participant information D1 to the server 3. For example, the first participant information generation unit 32 first identifies the first participant. That is, the first participant information generation unit 32 identifies the first participant based on user information pre-registered in the information processing device 2. Then, the first participant information generation unit 32 generates identification information with which the first participant can be identified. This identification information includes the name of the first participant. Then, the first participant information generation unit 32 generates first participant information D1 including the identification information regarding the first participant.

[0092] The first participant information generation unit 32 also generates a first participant image representing a first participant who directly participates in the remote conference using the information processing device 2, and generates first participant information D1 including the first participant image. For example, when the imaging function of the camera 21 is on, the first participant information generation unit 32 acquires a first captured image output from the camera control unit 31. Therefore, the first participant information generation unit 32 generates a first participant image representing the first participant based on the first captured image of the first participant. In this case, the first participant information generation unit 32 may generate a first participant image representing the first participant by directly using the first captured image captured by the camera 21, or may generate a first participant image representing the first participant by cutting out a facial image of the first participant from the first captured image. The first participant information generation unit 32 may also generate an image in which the background of the first participant is replaced with another image or blurred.

[0093] On the other hand, if the photographing function of the camera 21 is off, the first participant information generating unit 32 cannot acquire the first photographed image of the first participant. In this case, the first participant information generating unit 32 may read an image (e.g., a still image) predetermined by the user and use that image as the first participant image representing the first participant.

[0094] The first participant information generation unit 32 then transmits the first participant information D1 generated as described above to the server 3. At this time, the first participant information generation unit 32 adds a command to activate the conference screen provision unit 12 of the server 3.

[0095] The voice acquisition unit 33 acquires voice information corresponding to the voice uttered by the first participant via the microphone 22. When the voice acquisition unit 33 acquires the voice information of the first participant, it transmits the voice information to the server 3. At this time, the voice acquisition unit 33 transmits the voice information together with a command that activates the voice processing unit 13 of the server 3. This command includes identification information that can identify the first participant.

[0096] When accessing the browser 30, the screen acquisition unit 34 acquires a conference screen provided by the server 3. When the screen acquisition unit 34 acquires the conference screen, it outputs the conference screen to the display processing unit 35.

[0097] The display processing unit 35 is a processing unit that displays the conference screen acquired by the screen acquisition unit 34 on the display unit 24. Furthermore, when a display device 9 is connected to the device connection unit 26, the display processing unit 35 also displays the conference screen on the display device 9. This allows participants D, E, and F who indirectly participate in the remote conference in, for example, a conference room at the base PC to view the conference screen displayed on the screen 7 by the display device 9.

[0098] When the audio output unit 36 ​​acquires audio information from the server 3, it drives the speaker 23 based on the audio information to output the audio uttered by the participants at other locations.

[0099] The cooperative application 40 is started, for example, by an instruction from a first participant using the information processing device 2. The cooperative application 40 functions as a camera control unit 41, a second participant information generation unit 42, an image analysis unit 43, and an audio acquisition unit 46.

[0100] The camera control unit 41 controls the shooting operation of the camera 6 connected to the device connection unit 26. For example, when the shooting function of the camera 6 is set to on, the camera control unit 41 starts the shooting operation of the camera 6. Since the camera 6 is installed in a state where it can shoot the second participant, the camera control unit 41 can acquire a second captured image of the second participant by starting the shooting operation of the camera 6. For example, in the case of an information processing device 2c brought into the conference room of the base PC, the camera control unit 41 acquires a second captured image of participants D, E, and F who indirectly participate in the remote conference. The camera control unit 41 also acquires the second captured image as a moving image. Then, the camera control unit 41 outputs the second captured image to the second participant information generation unit 42.

[0101] The second participant information generation unit 42 generates second participant information D2 about a second participant who indirectly participates in the remote conference and transmits the second participant information D2 to the server 3. At this time, the second participant information generation unit 42 uses the information processing device 2 to generate a second participant image representing the second participant who indirectly participates in the remote conference, and generates second participant information D2 including the second participant image. For example, when the photographing function of the camera 6 is on, the second participant information generation unit 42 acquires a second captured image output from the camera control unit 41. Therefore, the second participant information generation unit 42 generates a second participant image representing the second participant based on the second captured image of the second participant. At this time, the second participant information generation unit 42 performs a process of extracting a face image from the second captured image and a process of identifying participants D, E, and F based on the face image. Once participants D, E, and F have been identified, the second participant information generation unit 42 generates second participant images representing each of participants D, E, and F based on the second captured images acquired by the camera control unit 41. The second participant information generation unit 42 may use the second captured images acquired by the camera 6 as the second participant images representing the second participants, or may extract a facial image of the second participant from the second captured image and use that facial image as the second participant image representing the second participant. The second participant information generation unit 42 may also replace the background portion of the second participant extracted from the second captured image with another image or blur it to generate an image.

[0102] On the other hand, if the photographing function of the camera 6 is off, the second participant information generating unit 42 cannot acquire a second photographed image of the second participant. In this case, the second participant information generating unit 42 may read an image (e.g., a still image) specified by a user (e.g., participant C) using the information processing device 2, and use that image as a second participant image representing the second participant.

[0103] The second participant information generation unit 42 may also display a screen on the display unit 24 for inputting identification information for identifying the second participant, accept an input operation of the identification information performed on the operation input unit 25, and generate identification information corresponding to the second participant. In the example of FIG. 1, participant C, who is using the information processing device 2c in the conference room at the base PC, manually inputs identification information for the other participants D, E, and F. The second participant information generation unit 42 then acquires the identification information input by participant C and generates second participant information D2 including the identification information. In this case, it is preferable that the second participant information generation unit 42 generates second participant information D2 in which a second participant image representing the second participant is associated with the identification information. Note that if the photographing function of the camera 6 is on, the second participant can be identified from the second photographed image as described above, so the second participant information generation unit 42 may omit participant C's input operation of the identification information.

[0104] For example, when identifying the second participant based on the second captured image, the second participant information generation unit 42 transmits, for example, a facial image of the second participant extracted from the second captured image to the server 3 as identification target information. At this time, the second participant information generation unit 42 transmits the information together with a command that activates the identification unit 14 of the server 3. Based on this command, the server 3 activates the identification unit 14 and searches the database 4 based on the facial image to identify the second participant included in the second captured image. This identifies the names of participants D, E, and F who will indirectly participate in the remote conference in the conference room of the base PC. The server 3 then returns the identification results of the second participant to the information processing device 2. The second participant information generation unit 42 identifies the second participant included in the second captured image based on the identification results received from the server 3.

[0105] Thereafter, the second participant information generation unit 42 transmits second participant information D2, which associates a second participant image representing the second participant with identification information including the name of the second participant, to the server 3. At this time, the second participant information generation unit 42 transmits the second participant information D2 together with a command to activate the conference screen provision unit 12 of the server 3. Furthermore, if the photographing function of the camera 6 is on, the second participant information generation unit 42 outputs the second participant image generated from the second photographed image to the image analysis unit 43.

[0106] The image analysis unit 43 is a processing unit that analyzes the second participant image output from the second participant information generation unit 42. The image analysis unit 43 includes a speaker detection unit 44 and a gesture detection unit 45.

[0107] The speaker detection unit 44 is a processing unit that detects a speaker from among the second participants identified by the second participant information generation unit 42. The speaker detection unit 44 detects a speaker by extracting a facial image from the second participant image generated based on the second captured image and analyzing the facial image. For example, the speaker detection unit 44 determines the movement of the mouth in the facial image of the second participant and detects the speaker based on the determination result. If the second participant image includes facial images of multiple second participants, the speaker detection unit 44 repeatedly performs a process of determining the movement of the mouth for each of the multiple facial images. Therefore, the speaker detection unit 44 can detect who is speaking in real time if at least one participant among the multiple second participants speaks.

[0108] When the speaker detection unit 44 detects a speaker, it notifies the server 3 of the speaker. At this time, the speaker detection unit 44 adds and transmits a command to activate the conference screen providing unit 12 of the server 3. This command includes identification information that can identify the second participant detected as the speaker.

[0109] The gesture detection unit 45 is a processing unit that detects a hand-raising gesture made by the second participant identified by the second participant information generation unit 42. The gesture detection unit 45 detects a hand-raising gesture made by the second participant based on the second participant image generated from the second captured image. For example, the gesture detection unit 45 detects a hand-raising gesture by detecting hand movements around the face image of the second participant. If the second participant image includes face images of multiple second participants, the gesture detection unit 45 targets each of the multiple face images as a detection target and repeatedly performs a process of detecting hand movements around the face image. Therefore, if at least one participant among the multiple second participants makes a hand-raising gesture, the gesture detection unit 45 can detect in real time who is making the hand-raising gesture.

[0110] When the gesture detection unit 45 detects a hand-raising gesture, it notifies the server 3 that a hand-raising gesture has been made. At this time, the gesture detection unit 45 adds and transmits a command to activate the conference screen providing unit 12 of the server 3. This command includes identification information that can identify the second participant who made the hand-raising gesture.

[0111] The voice acquisition unit 46 acquires voice information corresponding to the voice uttered by the second participant via at least one of the multiple microphones 8d, 8e, and 8f connected to the device connection unit 26. When the voice acquisition unit 46 acquires the voice information of the second participant, it transmits the voice information to the server 3. At this time, the voice acquisition unit 46 transmits the voice information together with a command that activates the voice processing unit 13 of the server 3. This command includes identification information that can identify the second participant detected as the speaker by the speaker detection unit 44.

[0112] The audio acquisition unit 46 also includes an audio switching unit 47. The audio switching unit 47 controls the on / off of the multiple microphones 8d, 8e, and 8f. For example, the audio switching unit 47 stores in advance which of the multiple second participants each of the multiple microphones 8d, 8e, and 8f is located near. To achieve this, for example, a user (participant C) using the information processing device 2c may register the installation locations of the multiple microphones 8d, 8e, and 8f in association with participants D, E, and F when the cooperative application 40 is started. Then, the audio switching unit 47 turns on the power of the microphone located near the second participant detected as the speaker by the speaker detection unit 44 among the multiple microphones 8d, 8e, and 8f, and turns off the power of the other microphones. This allows the speaker's voice to be acquired by the microphone closest to the speaker, resulting in clear, easy-to-listen audio.

[0113] Next, the operation of the conference system 1 configured as described above will be described. FIG. 5 is a first flowchart showing the operation of the conference system 1 in this embodiment. This first flowchart illustrates a flow in which participants A, B, and C who will directly participate in the remote conference log in to the server 3 using their own information processing devices 2a, 2b, and 2c. First, as the start time of the remote conference approaches, participants A and B who will directly participate in the remote conference at locations Pa and Pb operate the information processing devices 2a and 2b to launch the browser 30 and access an address notified in advance by the server 3 (step S1). At this time, the browser 30 transmits first participant information D1, including identification information for identifying participants A and B, to the server 3. When the server 3 detects access from the information processing devices 2a and 2b, it authenticates whether the participants are registered in the reservation information based on the identification information of participants A and B. If the authentication is successful, the server 3 permits participation in the remote conference and transmits a program module to the information processing devices 2a and 2b (step S2).

[0114] When the server 3 permits the information processing devices 2a and 2b to participate in the remote conference, the information processing devices 2a and 2b activate the camera 21 to capture images of participants A and B (step S3) and further activate the microphone 22 (step S4). Then, the browser 30 of the information processing devices 2a and 2b activates the camera control unit 31 to start capturing images with the camera 21 (step S5). Accordingly, the browser 30 of the information processing devices 2a and 2b activates the first participant information generation unit 32 to generate first participant images representing participants A and B based on the first captured images of participants A and B, and transmits first participant information D1 including the first participant images to the server 3 (step S6). Note that if the capture function of the camera 21 of the information processing devices 2a and 2b is turned off, participants A and B will not be captured, and the browser 30 uses a predetermined image as the first participant image and transmits first participant information D1 including the image and identification information of participants A and B to the server 3.

[0115] When the server 3 acquires the first participant information D1 regarding participants A and B from the information processing devices 2a and 2b, it activates the display control unit 17 to generate a conference screen in which the first participant information D1 regarding participants A and B is arranged in a predetermined display area, and transmits the conference screen to the information processing devices 2a and 2b (step S7). At this time, the server 3 extracts first participant images representing participants A and B from the first participant information D1 and arranges these first participant images in the predetermined display area. When the information processing devices 2a and 2b acquire the conference screen from the server 3, they display the conference screen on the display unit 24. This enables participants A and B to hold a conference while viewing the same conference screen.

[0116] Meanwhile, at the base PC, participant C, who will be directly participating in the remote conference, operates his / her own information processing device 2c to launch the browser 30 and access the address notified in advance by the server 3 (step S9). At this time, the browser 30 transmits first participant information D1, including identification information for identifying participant C, to the server 3. When the server 3 detects the access from the information processing device 2c, it authenticates whether or not the participant C is a participant registered in the reservation information based on the identification information of participant C. As a result, if the authentication is successful, the server 3 permits participation in the remote conference and transmits a program module to the information processing device 2c (step S10).

[0117] When the information processing device 2c is permitted to participate in the remote conference by the server 3, the information processing device 2c activates the camera 21 for capturing an image of participant C (step S11) and further activates the microphone 22 (step S12). Then, the browser 30 of the information processing device 2c operates the camera control unit 31 to start capturing an image with the camera 21 (step S13). Accordingly, the browser 30 of the information processing device 2c activates the first participant information generation unit 32 to generate a first participant image representing participant C based on the first captured image of participant C, and transmits first participant information D1 including the first participant image to the server 3 (step S14). Note that if the capturing function of the camera 21 in the information processing device 2c is turned off, participant C will not be captured, and therefore the browser 30 uses a predetermined image as the first participant image and transmits first participant information D1 including the image and identification information of participant C to the server 3.

[0118] When the server 3 acquires the first participant information D1 about participant C from the information processing device 2c, it activates the display control unit 17 to generate a conference screen in which the first participant information D1 about participant C is arranged in a predetermined display area, and transmits the conference screen to the information processing devices 2a, 2b, and 2c (step S15). At this time, the server 3 extracts a first participant image representing participant C from the first participant information D1 and arranges the first participant image in a predetermined display area. When the information processing devices 2a and 2b acquire the conference screen to which information about participant C has been added from the server 3, they display the conference screen on the display unit 24 (step S16). Furthermore, when the information processing device 2c acquires the conference screen from the server 3, they display the conference screen on the display unit 24 (step S17). This allows participants A, B, and C to hold a conference while viewing the same conference screen.

[0119] 6 is a diagram showing an example of a conference screen G1 displayed on the display unit 24 of the information processing devices 2a, 2b, and 2c as described above. The conference screen G1 has a material display area R1 that displays materials to be used in the conference, an icon display area R2 that can be operated by participants A, B, and C using the information processing devices 2a, 2b, and 2c, and a participant display area R3 that displays information based on the first participant information D1 and the second participant information D2.

[0120] The document image designated for shared viewing on at least one of the information processing devices 2a, 2b, and 2c is displayed in the document display area R1. For example, when participant A operates his / her own information processing device 2a to designate the sharing of the document image, the document image is transmitted from the information processing device 2a to the server 3. When the server 3 acquires the document image from the information processing device 2a, it activates the conference screen providing unit 12 to generate a conference screen G1 in which the document image is arranged in the document display area R1 and distributes the generated conference screen G1 to each of the information processing devices 2a, 2b, and 2c. As a result, the document image designated by participant A is displayed in the document display area R1 of the conference screen G1 displayed on each of the information processing devices 2a, 2b, and 2c.

[0121] In the icon display area R2, a plurality of icons 51 to 55 that can be operated by the participants A, B, and C using the information processing devices 2a, 2b, and 2c are displayed. For example, by operating these icons 51 to 55, the participants A, B, and C can turn on and off the camera 21, turn on and off the microphone 22, make a hand-raising gesture, and the like.

[0122] The participant display area R3 displays a list of information about the participants participating in the remote conference. In the example of FIG. 6, the participant display area R3 displays first participant images 61 representing participants A, B, and C who will directly participate in the remote conference. The display control unit 17 of the server 3 adds the names of participants A, B, and C when placing the first participant images acquired from the information processing devices 2a, 2b, and 2c in the participant display area R3. As a result, the names of participants A, B, and C are displayed along with the first participant images 61 corresponding to participants A, B, and C in the participant display area R3 of the conference screen G1 displayed on each of the information processing devices 2a, 2b, and 2c. Therefore, participants A, B, and C participating in the remote conference at each of the locations Pa, Pb, and Pc can understand who is participating in the remote conference.

[0123] Next, FIG. 7 is a second flowchart illustrating the operation of the conference system 1 according to this embodiment. This second flowchart illustrates a flow in which participants D, E, and F, who are indirectly participating in the remote conference, are additionally displayed as participants while participants A, B, and C are logged in to the server 3. Participant C, who is directly participating in the remote conference in the conference room of the base PC, operates his / her information processing device 2c to launch the cooperation application 40 (step S20). The cooperation application 40 then launches the camera 6 connected to the device connection unit 26 (step S21) and further launches the microphones 8d, 8e, and 8f (step S22). The cooperation application 40 then activates the camera control unit 41 to start capturing images with the camera 6 (step S23). As a result, the information processing device 2c acquires second captured images of participants D, E, and F, who are indirectly participating in the remote conference.

[0124] When image capture by the camera 6 begins, the cooperative application 40 activates the second participant information generation unit 42. The second participant information generation unit 42 performs face detection processing on the second captured image obtained by the camera 6 and extracts face images from the second captured image (step S24). FIG. 8 is a diagram illustrating an example of the face detection processing. For example, as shown in FIG. 8(a), a captured image G20 captured by the camera 6 at the site PC includes images of three participants D, E, and F. The second participant information generation unit 42 performs face detection processing on the captured image G20, thereby extracting face images G21, G22, and G23 of participants D, E, and F, respectively, from the captured image G20, as shown in FIG. 8(b). When the second participant information generation unit 42 extracts the facial images G21, G22, and G23 through face detection processing, it transmits these facial images G21, G22, and G23 to the server 3 as identification target information, and causes the identification unit 14 of the server 3 to perform processing to identify the second participant.

[0125] When the server 3 acquires the facial image from the information processing device 2c, it activates the identification unit 14 and executes a participant identification process to search the database 4 and identify the second participant (step S25). This participant identification process identifies that the facial images G21, G22, and G23 correspond to participants D, E, and F, respectively. When the identification unit 14 identifies the second participant, the server 3 transmits the identification result to the information processing device 2c. This allows the information processing device 2c to automatically identify participants D, E, and F who are present in the conference room with participant C. In this case, the information processing device 2c does not need to have participant C input identification information related to participants D, E, and F.

[0126] It is preferable that, after identifying participants D, E, and F based on the facial images acquired from the information processing device 2c, the server 3 authenticates whether or not participants D, E, and F are participants registered in the reservation information for the remote conference, and transmits the identification result to the information processing device 2c on the condition that the authentication is successful. Furthermore, if the authentication fails, it is preferable that the server 3 notify the information processing device 2c that the participants are not registered as participants in the remote conference.

[0127] Furthermore, when the imaging function of the camera 6 in the information processing device 2c is turned off, participants D, E, and F are not captured, and therefore the second participant information generation unit 42 uses, for example, an image specified by participant C as the second participant image representing each of participants D, E, and F. For example, the second participant information generation unit 42 has participant C input identification information for participants D, E, and F, and transmits first participant information D1 including the second participant image and the identification information input by participant C to the server 3. This allows the server 3 to identify participants D, E, and F based on the identification information input by participant C. The server 3 can then authenticate whether participants D, E, and F are participants registered in the remote conference reservation information.

[0128] If the photographing function of the camera 6 is on, the information processing device 2c can identify the participants D, E, and F. Then, the information processing device 2c generates second participant images representing each of the participants D, E, and F based on the second photographed image captured by the camera 6, and transmits second participant information D2 including the second participant image and identification information to the server 3 (step S26). At this time, the second participant information generation unit 42 uses, for example, facial images G21, G22, and G23 of the participants D, E, and F, respectively, extracted from the second photographed image G20, as the second participant images. Furthermore, when transmitting the second participant images representing each of the participants D, E, and F to the server 3, the second participant information generation unit 42 adds the names of the participants D, E, and F to the second participant images. This allows the server 3 to associate the second participant images received from the information processing device 2c with the names of the participants.

[0129] When the server 3 acquires the second participant images corresponding to participants D, E, and F from the information processing device 2c, the server 3 updates the conference screen by arranging the second participant images in the participant display area R3 together with the first participant images corresponding to participants A, B, and C, and transmits the updated conference screen to the information processing devices 2a, 2b, and 2c (step S27). When the information processing devices 2a and 2b acquire the updated conference screen from the server 3, the server 3 displays the conference screen on the display unit 24 (step S28). When the information processing device 2c acquires the conference screen from the server 3, the server 3 displays the conference screen on the display unit 24 (step S29). This allows participants A and B, who are participating in the remote conference at locations Pa and Pb different from location Pc, to know that participants D, E, and F have joined the remote conference.

[0130] FIG. 9 illustrates an example of a conference screen G2 in which a second participant is additionally displayed. The participant display area R3 of the conference screen G2 displays a first participant image 61 representing each of participants A, B, and C who will directly participate in the remote conference, and a second participant image 62 representing each of participants D, E, and F who will indirectly participate in the remote conference. The display control unit 17 of the server 3 adds the names of participants D, E, and F when placing the second participant image acquired from the information processing device 2c in the participant display area R3. As a result, the names of participants D, E, and F are displayed along with images corresponding to participants D, E, and F in the participant display area R3 of the conference screen G1 displayed on each of the information processing devices 2a, 2b, and 2c. Therefore, participants A and B who are participating in the remote conference at locations Pa and Pb, which are different from location Pc, can recognize that participants D, E, and F have also participated in the remote conference.

[0131] In this way, the conference system 1 of this embodiment can import images corresponding to participants D, E, and F who indirectly participate in the remote conference into the conference screen using the information processing device 2, in the same way as images of participants A, B, and C who directly participate in the remote conference, and display them on the information processing device 2 of each of the locations Pa, Pb, and Pc. Therefore, participants A and B who directly participate in the remote conference at locations Pa and Pb different from the conference room at location Pc can know that not only participant C who directly participates in the remote conference at another location Pc, but also participants D, E, and F are participating in the remote conference, allowing all participants to smoothly conduct the remote conference.

[0132] Next, FIG. 10 is a third flowchart illustrating the operation of the conference system 1 according to this embodiment. The third flowchart illustrates the process of detecting participants D, E, and F, who are indirectly participating in a remote conference in a conference room at a base PC, as speakers. For example, when the server 3 acquires audio information from the information processing device 2a, it can identify participant A as the speaker because the audio information was input from the information processing device 2a. When the server 3 acquires audio information from the information processing device 2b, it can identify participant B as the speaker because the audio information was input from the information processing device 2b. In contrast, when the server 3 acquires audio information from the information processing device 2c, even if it detects that the audio information was input from the information processing device 2c, it cannot accurately identify which of participants C, D, E, and F is the speaker. Therefore, the conference system 1 according to this embodiment performs processing based on the third flowchart illustrated in FIG. 10 to enable the server 3 to identify the speaker. The third flowchart illustrated in FIG. 10 assumes that the camera 6's imaging function is turned on because the speaker is detected by analyzing the second captured image acquired by the camera 6.

[0133] When the remote conference starts with participants A, B, C, D, E, and F joining as shown in the second flowchart, the information processing device 2c activates the image analysis unit 43 of the cooperative application 40. When the cooperative application 40 is launched, the image analysis unit 43 activates the speaker detection unit 44 and continuously performs image analysis on the second participant images that are sequentially generated as moving images (step S30). That is, as participants D, E, and F join the remote conference, the speaker detection unit 44 continuously analyzes the mouth movements of participants D, E, and F based on the second participant images to constantly determine whether or not they are the speaker. When the speaker detection unit 44 detects a speaker, the information processing device 2c notifies the server 3 that the speaker has been detected (step S31). At this time, the information processing device 2c notifies the server 3 by adding identification information that identifies which of participants D, E, and F is the speaker.

[0134] Furthermore, when the speaker detection unit 44 detects a speaker, the information processing device 2c activates the audio switching unit 47 of the audio acquisition unit 46. The audio switching unit 47 then turns on the power of the microphone among the multiple microphones 8d, 8e, and 8f that is installed near the second participant detected as the speaker by the speaker detection unit 44, and switches off the power of the other microphones (step S32). At this time, the information processing device 2c may also switch off the power of the microphone 22 of the device body.

[0135] When the server 3 is notified by the information processing device 2c that one of participants D, E, or F is the speaker while acquiring the audio information from the information processing device 2c, the server 3 identifies the speaker based on the notification. Then, the server 3 activates the display control unit 17 of the conference screen providing unit 12 to update the conference screen (step S33). That is, when a speaker is detected from among the second participants, the display control unit 17 highlights the second participant image 62 corresponding to the second participant detected as the speaker in the participant display area R3.

[0136] FIG. 11 is a diagram illustrating an example of a conference screen G2 in which a speaker is highlighted. FIG. 11 illustrates a case in which participant D is detected as the speaker. For example, when the information processing device 2c notifies the server 3 that participant D is the speaker, the server 3 highlights the second participant image 62 corresponding to participant D displayed in the participant display area R3, as shown in FIG. 11. In the illustrated example, the second participant image 62 corresponding to participant D is highlighted by displaying an image frame 65 of a predetermined color and a predetermined thickness around the periphery of the second participant image 62 corresponding to participant D. However, the highlighting method is not limited to this. In this way, when the server 3 acquires audio information from the information processing device 2c, the server 3 identifies which of participants D, E, and F is the speaker based on the notification from the information processing device 2c, and highlights the second participant image 62 corresponding to the second participant who is the speaker.

[0137] Note that when participant C speaks in the conference room at the base PC, the speaker detection unit 44 does not detect the speaker from among participants D, E, and F. In this case, the information processing device 2c does not notify the server 3 that a speaker has been detected. Therefore, when the server 3 acquires audio information from the information processing device 2c but does not receive a notification from the information processing device 2c that a speaker has been detected, it can identify that the speaker is participant B. Therefore, in this case, the server 3 updates the conference screen to highlight the first participant image 61 corresponding to participant C.

[0138] When the server 3 updates the conference screen G2 based on the notification from the information processing device 2c, it transmits the updated conference screen G2 to the information processing devices 2a, 2b, and 2c. Then, when the information processing devices 2a and 2b acquire the updated conference screen G2 from the server 3, they display the conference screen on the display unit 24 (step S34). Similarly, when the information processing device 2c acquires the conference screen G2 from the server 3, they display the conference screen G2 on the display unit 24 (step S35). This allows participants A and B, who are participating in the remote conference at locations Pa and Pb different from location Pc, to know that participant D, who is indirectly participating in the remote conference, is speaking.

[0139] Next, FIG. 12 is a fourth flowchart illustrating the operation of the conference system 1 according to this embodiment. The fourth flowchart illustrates the flow of detecting hand-raising gestures by participants D, E, and F who are indirectly participating in the remote conference in the conference room of the base PC. For example, participants A, B, and C who are directly participating in the remote conference using their own information processing devices 2a, 2b, and 2c can raise their hand to other participants by operating the icon 54 displayed in the icon display area R2 when they wish to speak. That is, when the server 3 detects that the icon 54 has been operated on one of the information processing devices 2a, 2b, and 2c, it can identify which of participants A, B, and C has raised their hand. In contrast, participants D, E, and F who are indirectly participating in the remote conference cannot perform such icon operations. Therefore, the conference system 1 according to this embodiment performs processing based on the fourth flowchart shown in FIG. 12, enabling the server 3 to detect hand-raising gestures by participants D, E, and F. This fourth flowchart also assumes that the camera 6's imaging function is turned on.

[0140] When the remote conference starts with participants A, B, C, D, E, and F joining as shown in the second flowchart, the information processing device 2c activates the image analysis unit 43 of the cooperative application 40. When the cooperative application 40 is launched, the image analysis unit 43 activates the gesture detection unit 45 and continuously performs image analysis on the second participant images, which are sequentially generated as moving images (step S40). That is, as participants D, E, and F join the remote conference, the gesture detection unit 45 continuously analyzes hand movements around the facial images of participants D, E, and F based on the second participant images, and constantly determines whether a hand-raising gesture has been made. When the gesture detection unit 45 detects a hand-raising gesture, the information processing device 2c notifies the server 3 that a hand-raising gesture has been detected (step S41). At this time, the information processing device 2c notifies the server 3 by adding identification information that identifies which of participants D, E, and F made the hand-raising gesture.

[0141] When the server 3 is notified by the information processing device 2c that a hand-raising gesture has been made by one of the participants D, E, or F, the server 3 identifies the participant who made the hand-raising gesture based on the notification. The server 3 then activates the display control unit 17 of the conference screen providing unit 12 to update the conference screen (step S42). That is, the display control unit 17 highlights the second participant image 62 corresponding to the second participant whose hand-raising gesture has been detected in the participant display area R3.

[0142] FIG. 13 is a diagram illustrating an example of a conference screen G2 in which a second participant who has made a hand-raising gesture is highlighted. FIG. 13 illustrates a case in which a hand-raising gesture by participant D has been detected. For example, upon receiving a notification from the information processing device 2c that a hand-raising gesture by participant D has been detected, the server 3 highlights the second participant image 62 corresponding to participant D displayed in the participant display area R3, as shown in FIG. 13. In the illustrated example, the second participant image 62 corresponding to participant D is highlighted by adding a hand-raising icon 66 to the second participant image 62. However, the highlighting method is not limited to this. In this way, during the progress of the remote conference, the server 3 identifies which of participants D, E, and F has made a hand-raising gesture based on a notification from the information processing device 2c, and highlights the second participant image 62 corresponding to the second participant who made the hand-raising gesture.

[0143] When the server 3 updates the conference screen G2 based on the notification from the information processing device 2c, it transmits the updated conference screen G2 to the information processing devices 2a, 2b, and 2c. Then, when the information processing devices 2a and 2b acquire the updated conference screen G2 from the server 3, they display the conference screen on the display unit 24 (step S43). Similarly, when the information processing device 2c acquires the conference screen G2 from the server 3, they display the conference screen G2 on the display unit 24 (step S44). This allows participants A and B, who are participating in the remote conference at locations Pa and Pb different from location Pc, to know that participant D, who is indirectly participating in the remote conference, is making a hand-raising gesture.

[0144] As described above, when a remote conference is held by connecting multiple information processing devices 2a, 2b, and 2c to the server 3, the conference system 1 of this embodiment generates first participant information D1 about participants A, B, and C who will directly participate in the remote conference using each of the multiple information processing devices 2a, 2b, and 2c, and arranges the first participant information D1 in the participant display area R3 of the conference screen to display it on each of the information processing devices 2a, 2b, and 2c. Furthermore, when there are participants D, E, and F who will indirectly participate in the remote conference along with participant C who will directly participate in the remote conference using the information processing device 2, the conference system 1 generates second participant information D2 about participants D, E, and F, and arranges the second participant information D2 in the participant display area R3 of the conference screen to display it on each of the information processing devices 2a, 2b, and 2c, similar to the first participant information D1.

[0145] Therefore, the conference system 1 of this embodiment can display not only participants A, B, and C who directly participate in the remote conference using the information processing devices 2a, 2b, and 2c, but also participants D, E, and F who indirectly participate in the remote conference without using the information processing device 2 as participants on the conference screen, allowing each participant participating in the remote conference at a different location to recognize all of the other participants, thereby enabling the remote conference to proceed smoothly.In other words, the conference system 1 of this embodiment is configured to incorporate participants D, E, and F who do not use the information processing devices 2a, 2b, and 2c into the remote conference platform of the server 3, and to treat participants A, B, and C who participate in the remote conference using the information processing devices 2a, 2b, and 2c and participants D, E, and F who indirectly participate in the remote conference without using the information processing devices 2a, 2b, and 2c in the same way.

[0146] Furthermore, when participants D, E, and F who are indirectly participating in the remote conference speak, the conference system 1 of this embodiment automatically detects which of participants D, E, and F is the speaker, and highlights the image of the second participant corresponding to the second participant detected as the speaker. Therefore, even if participants D, E, and F who are indirectly participating in the remote conference speak, participants A, B, and C who are directly participating in the remote conference can know who is speaking by checking the conference screen.

[0147] In the conference system 1 of this embodiment, when participants D, E, and F who are indirectly participating in the remote conference make a hand-raising gesture, the conference system 1 automatically detects which of the participants D, E, and F made the hand-raising gesture and highlights the image of the second participant corresponding to the second participant who made the hand-raising gesture. Therefore, when participants D, E, and F who are indirectly participating in the remote conference make a hand-raising gesture, participants A, B, and C who are directly participating in the remote conference can see who is about to speak by visually checking the conference screen.

[0148] In the above description, an example has been described in which, when the photographing function of the camera 6 connected to the information processing device 2c is on, the information processing device 2c acquires a second photographed image from the camera 6 and generates second participant information D2 corresponding to each of the participants D, E, and F who indirectly participate in the remote conference in the information processing device 2c. However, the process of generating the second participant information D2 can also be performed by the server 3. That is, the camera control unit 41 of the information processing device 2c transmits the second photographed image taken by the camera 6 to the server 3 as is, and the server 3 generates second participant information D2 corresponding to each of the participants D, E, and F based on the second photographed image. In this case, the above-described second participant information generation unit 42 is installed in the server 3.

[0149] Fig. 14 is a flowchart for generating second participant information D2 for participants D, E, and F in the server 3. In other words, the flowchart shown in Fig. 14 is a flowchart that replaces the second flowchart shown in Fig. 7, and shows the flow for, for example, adding and displaying participants D, E, and F who indirectly participate in the remote conference as participants when participants A, B, and C are logged in to the server 3.

[0150] First, participant C, who will be directly participating in the remote conference in the conference room of the base PC, operates his / her information processing device 2c to launch the cooperative application 40 (step S50). The cooperative application 40 then launches the camera 6 connected to the device connection unit 26 (step S51) and also launches the microphones 8d, 8e, and 8f (step S52). The cooperative application 40 then activates the camera control unit 41 to start capturing images with the camera 6 (step S53). As a result, the information processing device 2c acquires second captured images of participants D, E, and F, who are indirectly participating in the remote conference. The cooperative application 40 then transmits the second captured images obtained from the camera 6 to the server 3.

[0151] When the server 3 acquires the second captured image from the information processing device 2c, it performs a face detection process on the second captured image and extracts facial images of participants D, E, and F from the second captured image (step S54). The server 3 then activates the identification unit 14 to execute a participant identification process for searching the database 4 to identify the second participants (step S55). This participant identification process allows each of participants D, E, and F appearing in the second captured image to be identified. Once the identification unit 14 has identified the second participants, the server 3 activates the second participant information generation unit 42 to generate second participant information D2 corresponding to each of participants D, E, and F based on the second captured image acquired from the information processing device 2c (step S56). At this time, the second participant information generation unit 42 uses, for example, the facial images of participants D, E, and F extracted from the second captured image G20 as the second participant images. In addition, the second participant information generating unit 42 adds the names of participants D, E, and F to the facial images of participants D, E, and F, respectively.

[0152] Next, the server 3 activates the display control unit 17 of the conference screen providing unit 12 to update the conference screen by arranging the second participant images 62 of participants D, E, and F in the participant display area R3 together with the first participant images 61 of participants A, B, and C, and transmits the updated conference screen to the information processing devices 2a, 2b, and 2c (step S57). Upon receiving the updated conference screen from the server 3, the information processing devices 2a and 2b display the conference screen on the display unit 24 (step S58). Similarly, upon receiving the conference screen from the server 3, the information processing device 2c displays the conference screen on the display unit 24 (step S59). At this time, the conference screens displayed on the information processing devices 2a, 2b, and 2c are the same as those shown in FIG. 9. Therefore, participants A and B, who are participating in the remote conference at locations Pa and Pb, which are different from location Pc, can recognize that participants D, E, and F have joined the remote conference.

[0153] In the above description, the case where the information processing device 2c detects speech and hand-raising gestures by participants D, E, and F has been mainly exemplified. However, the functions of the image analysis unit 43, including the speaker detection unit 44 and gesture detection unit 45, can also be installed in the server 3. In this case, the camera control unit 41 of the information processing device 2c transmits the second captured image taken by the camera 6 to the server 3 as is. The server 3 then performs image analysis on the second captured image acquired from the information processing device 2. That is, the server 3 performs image analysis using the speaker detection unit 44 and gesture detection unit 45 to detect whether any of the participants D, E, and F who indirectly participate in the remote conference is speaking or making a hand-raising gesture.

[0154] Fig. 15 is a flowchart for detecting a speaker from among participants D, E, and F in the server 3. That is, the flowchart shown in Fig. 15 is a flowchart replacing the third flowchart shown in Fig. 10, and shows the flow for detecting one of participants D, E, and F as the speaker in the server 3.

[0155] The information processing device 2c activates the camera 6 to start capturing images, and then transmits the second captured images of participants D, E, and F to the server 3 (step S60). Upon receiving the second captured images from the information processing device 2c, the server 3 activates the speaker detection unit 44 of the image analysis unit 43 to start image analysis of the second captured images sequentially received from the information processing device 2c (step S61). That is, as participants D, E, and F join the remote conference, the server 3 continuously analyzes the mouth movements of participants D, E, and F based on the second captured images, constantly determining whether or not they are the speaker. Upon detecting a speaker, the server 3 notifies the information processing device 2c that the speaker has been detected (step S62). At this time, the server 3 notifies the information processing device 2c by adding identification information that identifies which of participants D, E, and F is the speaker.

[0156] When the information processing device 2c is notified of the speaker by the server 3, the audio switching unit 47 of the audio acquisition unit 46 is activated. The audio switching unit 47 then turns on the power of the microphone among the multiple microphones 8d, 8e, and 8f that is installed near the second participant detected by the server 3 as the speaker, and switches off the power of the other microphones (step S63). At this time, the information processing device 2c may also switch off the power of the microphone 22 of the device body.

[0157] Furthermore, when the server 3 detects that one of participants D, E, or F is the speaker, it activates the display control unit 17 of the conference screen providing unit 12 to update the conference screen (step S64). That is, when a speaker is detected among the second participants, the display control unit 17 highlights the second participant image corresponding to the second participant detected as the speaker in the participant display area R3. The server 3 then transmits the updated conference screen G2 to each of the information processing devices 2a, 2b, and 2c. When the information processing devices 2a and 2b acquire the updated conference screen G2 from the server 3, they display the conference screen on the display unit 24 (step S65). When the information processing device 2c acquires the conference screen G2 from the server 3, they also display the conference screen G2 on the display unit 24 (step S66). This allows participants A and B, who are participating in the remote conference at locations Pa and Pb different from location Pc, to know which of participants D, E, or F, who are indirectly participating in the remote conference, is speaking.

[0158] Next, Fig. 16 is a flowchart for detecting that any one of participants D, E, or F has made a hand-raising gesture in the server 3. That is, the flowchart shown in Fig. 16 is a flowchart replacing the fourth flowchart shown in Fig. 12, and shows the flow for detecting that any one of participants D, E, or F has made a hand-raising gesture in the server 3.

[0159] The information processing device 2c activates the camera 6 to start capturing images and transmits second captured images of participants D, E, and F to the server 3 (step S70). Upon receiving the second captured images from the information processing device 2c, the server 3 activates the speaker detection unit 44 of the image analysis unit 43 to start image analysis of the second captured images sequentially received from the information processing device 2c (step S71). That is, as participants D, E, and F join the remote conference, the server 3 continuously analyzes hand movements around the facial images of participants D, E, and F based on the second captured images, constantly determining whether a hand-raising gesture has been made. Upon detecting a hand-raising gesture, the server 3 identifies the participant who made the hand-raising gesture (step S72). The server 3 then activates the display control unit 17 of the conference screen providing unit 12 to update the conference screen (step S73). That is, the display control unit 17 highlights the second participant image corresponding to the second participant whose raising hand gesture was detected in the participant display area R3. Then, the server 3 transmits the updated conference screen G2 to each of the information processing devices 2a, 2b, and 2c. When the information processing devices 2a and 2b acquire the updated conference screen G2 from the server 3, they display the conference screen on the display unit 24 (step S74). When the information processing device 2c acquires the conference screen G2 from the server 3, they display the conference screen G2 on the display unit 24 (step S75). This allows participants A and B, who are participating in the remote conference at locations Pa and Pb different from location Pc, to know that one of participants D, E, and F, who are indirectly participating in the remote conference, is making a raising hand gesture.

[0160] In this way, the conference system 1 of this embodiment may employ a configuration in which the server 3 detects speakers and hand-raising gestures.

[0161] (Variation) Although the preferred embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications can be applied.

[0162] For example, in the above embodiment, a camera 6 installed in a conference room at point Pc is connected to information processing device 2c, and server 3 acquires images captured by camera 6 via information processing device 2c. However, camera 6 may be connected to a network-connected device installed in the conference room, for example, and transmit the captured images directly to server 3 via network 5. In this case, the above-mentioned camera control unit 41 is installed in server 3. Then, server 3 can acquire second captured images of participants D, E, and F by directly controlling camera 6 via network 5.

[0163] Furthermore, the multiple microphones 8d, 8e, and 8f installed in the conference room of the base PC may also be directly connected to the server 3 via the network 5. In this case, the above-described voice acquisition unit 46 is installed in the server 3.

[0164] Furthermore, the display device 9 installed in the conference room of the base PC may also be directly connected to the server 3 via the network 5. In this case, the conference screen providing unit 12 of the server 3 can display the conference screen on the screen 7 installed in the conference room by directly transmitting the conference screen to the display device 9 via the network 5.

[0165] Furthermore, in the above embodiment, when the photographing functions of the cameras 21 and 6 are on, names are assigned to the facial images of the participants A, B, C, D, E, and F extracted from the first photographed image and the second photographed image, respectively, and the names are displayed in the participant display area R3 of the conference screen. However, as long as the facial images of the participants A, B, C, D, E, and F are displayed, it is not necessary to assign and display the names of each participant. This is because, even if names are not displayed, each participant can identify the other participants by visually checking the facial images displayed on the conference screen.

[0166] On the other hand, if the photographing function of camera 21 or camera 6 is off and facial images of each participant are not displayed in participant display area R3, it is preferable to display the first participant image 61 and the second participant image 62 with the name of each participant attached, as described above. That is, the first participant image 61 and the second participant image 62 may simply be silhouette images of a person, and it may be difficult to identify the participants from such images alone, so it is preferable to display them with their names attached. Note that, if the photographing function of camera 21 or camera 6 is off, it may be possible to not display the first participant image 61 or the second participant image 62 in participant display area R3, and to display only the name of each participant in participant display area R3.

[0167] In the above embodiment, as an example, a conference screen is first created in which a first participant image is displayed in the participant display area R3, and then a conference screen is created in which a second participant image is displayed in the participant display area R3. However, when creating a conference screen including both the first and second participant images, the above-described two-stage processing is not necessarily required. For example, if the information processing device 2c simultaneously transmits the first participant information D1 about participant C and the second participant information D2 about participants D, E, and F to the server 3, the server 3 creates a conference screen including both the first and second participants and transmits the conference screen to the information processing devices 2a, 2b, and 2c. This allows the information processing device 2c to display a conference screen including both the first and second participants in a single processing stage.

[0168] In the above embodiment, a case where different processing is performed when a first participant speaks and when a second participant speaks has been exemplified. However, this is not limited to this, and a configuration is also possible in which the same processing is performed when a first participant speaks and when a second participant speaks. For example, when the information processing devices 2a, 2b, and 2c detect a speech by the first participant using the microphone 22, the information processing devices 2a, 2b, and 2c add information indicating that the first participant has spoken to the first participant information D1 and transmit the information to the server 3. The server 3 can detect that the first participant has spoken based on the information included in the first participant information D1. As a result, the server 3 can highlight the first participant image 61 corresponding to the first participant who spoke.

[0169] On the other hand, when the information processing device 2c detects that the second participant has spoken by analyzing the second captured image, the information processing device 2c adds information indicating that the second participant has spoken to the second participant information D2 and transmits the information to the server 3. The server 3 can detect that the second participant has spoken based on the information included in the second participant information D2. As a result, the server 3 can highlight the second participant image 62 corresponding to the second participant who spoke.

[0170] By configuring in this way that common processing is performed when the first participant speaks and when the second participant speaks, the server 3 can apply common processing and reflect it on the conference screen regardless of whether the first participant or the second participant speaks, which has the advantage of being able to apply a common processing program to the first participant and the second participant.

[0171] In the above embodiment, a case where different processing is performed when a first participant makes a hand-raising gesture and when a second participant makes a hand-raising gesture has been exemplified. However, this is not limited to this, and a configuration is also possible in which the same processing is performed when a first participant makes a hand-raising gesture and when a second participant makes a hand-raising gesture. For example, when the information processing devices 2a, 2b, and 2c detect a hand-raising operation on the conference screen, the information processing devices 2a, 2b, and 2c add information indicating that the first participant made a hand-raising gesture to the first participant information D1 and transmit the information to the server 3. The server 3 can detect that the first participant made a hand-raising gesture based on the information included in the first participant information D1. As a result, the server 3 can highlight the first participant image 61 corresponding to the first participant who made the hand-raising gesture.

[0172] On the other hand, when the information processing device 2c detects that the second participant has made a hand-raising gesture by analyzing the second captured image, the information processing device 2c adds information indicating that the second participant has made a hand-raising gesture to the second participant information D2 and transmits the information to the server 3. The server 3 can detect that the second participant has made a hand-raising gesture based on the information included in the second participant information D2. As a result, the server 3 can highlight the second participant image 62 corresponding to the second participant who made the hand-raising gesture.

[0173] In this way, if the configuration is such that common processing is performed when the first participant makes a hand-raising gesture and when the second participant makes a hand-raising gesture, the server 3 can apply common processing and reflect it on the conference screen regardless of whether the first participant or the second participant makes a hand-raising gesture, which has the advantage of being able to apply a common processing program to the first participant and the second participant.

[0174] In the above embodiment, the speaker is detected by determining the facial images of participants D, E, and F included in the second captured image. However, the method for detecting the speaker is not necessarily limited to image analysis. For example, when a voice input of a predetermined level or higher is detected in any one of the microphones 8d, 8e, and 8f, the participant located near that microphone may be identified as the speaker.

[0175] In the above embodiment, the conference system 1 is illustrated as including a plurality of microphones 8d, 8e, and 8f assigned to each of the participants D, E, and F who indirectly participate in the remote conference. However, the plurality of microphones 8d, 8e, and 8f are not essential for the conference system 1 of this embodiment. [Explanation of symbols]

[0176] 1. Conference System 2(2a, 2b, 2c) Information processing device 3 Server 4 Database 5. Network 6. Camera (secondary photography means) 8a, 8b, 8c Microphone (voice input means) 12 Conference Screen Provision Department 13 Audio processing unit 14 Specification part (specification means) 15 First participant information acquisition unit (first participant information acquisition means) 16 Second participant information acquisition unit (second participant information acquisition means) 17 Display control unit (display control means) 21 Camera (first photographing means) 24 Display section (display means) 30 browsers 31 camera control unit (first photography control means) 41 camera control unit (second photography control means) 32 First participant image generation unit (first participant image generation means) 42 Second participant image generation unit (second participant image generation means) 43 Image analysis unit 44 Speaker detection unit (speaker detection means) 45 Gesture detection unit (gesture detection means) 46 Voice Acquisition Unit 47 Audio switching unit (audio switching means)

Claims

1. A conference system in which a plurality of information processing devices are connected to a server via a network, and the server provides a conference screen to each of the plurality of information processing devices, Each of the plurality of information processing devices a first participant information generation unit that generates a first participant image representing a first participant participating in the remote conference, and transmits first participant information to the server, the first participant image being generated by adding first identification information that can identify the first participant; a first voice acquisition unit that acquires first voice information corresponding to a voice uttered by the first participant, adds the first identification information to the first voice information, and transmits the first voice information to the server; Equipped with At least one of the plurality of information processing devices a second participant information generation unit that generates a second participant image representing a second participant who indirectly participates in the remote conference together with the first participant, and transmits second participant information to the server, the second participant image being generated by adding second identification information that can identify the second participant; a second voice acquisition unit that acquires second voice information corresponding to a voice uttered by the second participant, adds the second identification information to the second voice information, and transmits the second voice information to the server; Further provided with The server a first participant information acquiring means for acquiring the first participant information transmitted from each of the plurality of information processing devices; a second participant information acquiring means for acquiring the second participant information transmitted from at least one of the plurality of information processing devices; a display control means for displaying the first participant image included in the first participant information acquired by the first participant information acquisition means in a predetermined display area of ​​the conference screen, and for displaying the second participant image included in the second participant information acquired by the second participant information acquisition means in the display area of ​​the conference screen; Equipped with A conference system characterized in that, when the display control means acquires audio information from an information processing device that is the sender of both the first participant information and the second participant information, it highlights the first participant image or the second participant image displayed in the display area based on identification information attached to the audio information.

2. Each of the plurality of information processing devices a first photographing means for photographing the first participant; Further provided with 2. The conference system according to claim 1, wherein the first participant information generating unit generates the first participant image based on a first photographed image taken by the first photographing unit.

3. The conference system according to claim 1 or 2, characterized in that the display control means displays the first participant image in the display area by adding the name of the first participant to the first participant image.

4. An information processing device that transmits both the first participant information and the second participant information, a second photographing means for photographing the second participant; Further provided with 4. The conference system according to claim 1, wherein the second participant information generating unit generates the second participant image based on a second captured image taken by the second image capturing unit.

5. The conference system according to any one of claims 1 to 4, characterized in that the display control means displays the second participant image in the display area by adding the name of the second participant to the second participant image.

6. An information processing device that transmits both the first participant information and the second participant information, a speaker detection means for detecting that the second participant has spoken based on the second photographed image photographed by the second photographing means; 5. The conference system according to claim 4, further comprising:

7. The conference system described in claim 6, characterized in that when the speaker detection means detects a statement by the second participant, the display control means highlights the second participant image corresponding to the second participant whose statement was detected in the display area.

8. The conference system according to claim 6 or 7, characterized in that the speaker detection means detects that the second participant has spoken by analyzing a facial image of the second participant based on the second captured image.

9. An information processing device that transmits both the first participant information and the second participant information, a voice input means installed near the second participant; an audio switching means for controlling the audio input means on and off; Further provided with The conference system according to any one of claims 6 to 8, characterized in that when the speaker detection means detects a speech by the second participant, the audio switching means switches on the audio input means installed near the second participant whose speech is detected.

10. An information processing device that transmits both the first participant information and the second participant information, a gesture detection means for detecting that the second participant has made a hand-raising gesture based on the second photographed image photographed by the second photographing means; 5. The conference system according to claim 4, further comprising:

11. The conference system according to claim 10, characterized in that, when a raising hand gesture is detected by the gesture detection means, the display control means highlights in the display area the image of the second participant corresponding to the second participant whose raising hand gesture was detected.

12. The conference system according to claim 10 or 11, characterized in that, when a hand-raising gesture is detected by the gesture detection means, the display control means adds a hand-raising icon to the second participant image corresponding to the second participant whose hand-raising gesture was detected and displays it.

13. A server that supports a remote conference by connecting to a plurality of information processing devices via a network and providing a conference screen to each of the plurality of information processing devices, a first participant information acquiring means for acquiring, from each of the plurality of information processing devices, a first participant image representing a first participant participating in the remote conference using the information processing device and first participant information including first identification information capable of identifying the first participant; a second participant information acquisition means for acquiring, from at least one of the plurality of information processing devices, second participant information including a second participant image representing a second participant who indirectly participates in the remote conference together with the first participant and second identification information capable of identifying the second participant; a display control means for displaying the first participant image acquired by the first participant information acquisition means in a predetermined display area of ​​the conference screen, and for displaying the second participant image acquired by the second participant information acquisition means in the display area; Equipped with The server is characterized in that, when the display control means acquires audio information from an information processing device that is the sender of both the first participant information and the second participant information, it identifies whether the speaker of the audio information is the first participant or the second participant based on identification information attached to the audio information, and highlights the first participant image or the second participant image displayed in the display area based on the identification result.

14. An information processing device that is connected to a server via a network and that holds a remote conference using a platform for remote conferences provided by the server, a first participant information generating means for generating a first participant image representing a first participant participating in the remote conference using the information processing device, generating first participant information by adding first identification information capable of identifying the first participant to the first participant image, and transmitting the first participant information to the server; a second participant information generating means for generating a second participant image representing a second participant who indirectly participates in the remote conference together with the first participant, generating second participant information by adding second identification information capable of identifying the second participant to the second participant image, and transmitting the second participant information to the server; a conference screen acquisition means for acquiring from the server a conference screen on which the first participant image and the second participant image are arranged in a predetermined display area; a display means for displaying the conference screen acquired by the conference screen acquisition means; a voice acquisition unit that acquires voice information corresponding to a voice uttered by the first participant or the second participant and transmits the voice information to the server; Equipped with The information processing device is characterized in that, when the audio acquisition unit acquires audio information corresponding to the audio spoken by the first participant, it adds the first identification information to the audio information and sends it to the server, and when it acquires audio information corresponding to the audio spoken by the second participant, it adds the second identification information to the audio information and sends it to the server.

15. a first photographing control means for controlling a first photographing means for photographing the first participant; a second photographing control means for controlling a second photographing means for photographing the second participant; Further provided with the first participant information generating means generates the first participant image based on a first photographed image acquired by the first photographing control means controlling the first photographing means; 15. The information processing device according to claim 14, wherein the second participant information generating means generates the second participant image based on a second captured image acquired by the second photographing control means controlling the second photographing means.

16. a speaker detection means for detecting that the second participant has spoken and notifying the server of the detection; Further provided with 16. The information processing apparatus according to claim 15, wherein the speaker detection means detects that the second participant has spoken by analyzing a face image of the second participant included in the second captured image.

17. a gesture detection means for detecting that the second participant has made a hand raising gesture and notifying the server; Further provided with 17. The information processing apparatus according to claim 15, wherein the gesture detection means detects a hand-raising gesture by analyzing an image of the second participant included in the second captured image.

18. A program executed on a server that supports a remote conference by connecting to a plurality of information processing devices via a network and providing a conference screen to each of the plurality of information processing devices, the program comprising: a first step of acquiring, from each of the plurality of information processing devices, a first participant image representing a first participant participating in a remote conference using the information processing device and first participant information including first identification information capable of identifying the first participant; a second step of acquiring, from at least one of the plurality of information processing devices, second participant information including a second participant image representing a second participant who indirectly participates in the remote conference together with the first participant and second identification information capable of identifying the second participant; a third step of displaying the first participant image acquired in the first step in a predetermined display area of ​​the conference screen and displaying the second participant image acquired in the second step in the display area; Execute The third step is a program characterized in that, when audio information is obtained from an information processing device that is the sender of both the first participant information and the second participant information, the program identifies whether the speaker of the audio information is the first participant or the second participant based on identification information attached to the audio information, and highlights the first participant image or the second participant image displayed in the display area based on the identification result.

Images