Remote conference method and remote conference system

Abstract

To provide a technology for allowing some of participants to talk to each other without interfering with a progress of a teleconference.SOLUTION: A teleconference method includes a first generation step, a second generation step, a display step, a determination step, and a transmission step. The first generation step acquires voice of participants for each of conference terminals of at least three participants participating in a teleconference, and generates voice data including volume information from the acquired voice. The second generation step acquires video of the participants during the teleconference for each conference terminal and generates video data. The display step displays seating information indicating seating for each of the participants in a virtual space for each of the conference terminals of the participants. The determination step determines a destination of the voice data of the participants for each of the conference terminals of the participants on the basis of the volume information of the voice data and the video data of the participant as well as the seating information. The transmission step transmits the voice data of the participants, for each of the conference terminals of the participants, on the basis of the destination determined for the voice data of the participants.SELECTED DRAWING: Figure 5A

Description

[Technical field]

[0001] The present invention relates to a remote conference method and a remote conference system. [Background technology]

[0002] In recent years, the use of video conference systems via communication lines such as the Internet and dedicated lines has increased. The following Patent Document 1 discloses a conversation control method for a conversation in which a plurality of people (three or more) at remote locations hold an audio conference using telephone lines or the like. This conversation control method generates stereophonic audio data for each participant based on the positions of the participants in the conference in a virtual layout and the orientations of the participants, making it easier for each participant to grasp the situation, such as who is speaking to whom. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2001-274912 A Summary of the Invention [Problem to be solved by the invention]

[0004] In face-to-face conferences, participants sitting next to each other may talk in a low voice. In remote conferences such as video conferences, there is also a need for some participants to talk with each other while participating in the conference.

[0005] An object of the present invention is to provide a technique that allows some participants to have a conversation with each other without interrupting the progress of a remote conference. [Means for solving the problem]

[0006] The remote conference method according to the present invention includes a first generating step, a second generating step, a displaying step, a determining step, and a transmitting step. The first generating step acquires the voices of the participants for each of the conference terminals of at least three or more participants participating in the remote conference, and generates voice data including volume information from the acquired voices. The second generating step acquires video of the participants during the conference for each of the conference terminals of the participants, and generates video data. The displaying step displays seat information indicating the seat of each participant in the virtual space for each of the conference terminals of the participants. The determining step determines, for each of the conference terminals of the participants, a destination of the voice data of the participant based on the seat information, the volume information of the voice data of the participant, and the video data. The transmitting step transmits, for each of the conference terminals of the participants, the voice data of the participant based on the destination determined for the voice data of the participant.

[0007] The remote conference system according to the present invention includes a conference terminal and a server. The conference terminal is a conference terminal for each of at least three or more participants participating in the remote conference. The server is connected to the conference terminal for each of the participants via a communication line. The conference terminal includes a storage unit, a display unit, an audio data generating unit, a video data generating unit, a determination unit, and a first transmission unit. The storage unit stores seat information indicating a seat in a virtual space for each participant. The display unit displays the seat information. The audio data generating unit acquires the voice of the participant and generates audio data including volume information from the acquired voice. The video data generating unit acquires a video of the participant during the conference and generates video data. The determination unit determines a destination of the audio data based on the seat information, the volume information of the audio data, and the video data. The first transmission unit transmits audio information including audio data associated with destination information indicating the determined destination and video information including video data to the server together with source information indicating the terminal. The server includes an acquisition unit and a second transmission unit. The acquisition unit acquires the audio information, the video information, and the source information from each conference terminal. The second transmission unit transmits the audio information to each conference terminal based on the destination information of the audio information from each conference terminal. Effect of the Invention

[0008] According to the remote conference method and the remote conference system of the present invention, some of the participants can talk with each other without interrupting the progress of the remote conference. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a schematic diagram showing the configuration of a video conference system according to an embodiment. [Diagram 2] FIG. 2 is a block diagram showing a schematic configuration of a conference terminal according to an embodiment. [Figure 3A] FIG. 3A is a diagram showing an example of a seating chart. [Figure 3B] FIG. 3B is a diagram showing an example of seat information. [Figure 4] FIG. 4 is a block diagram showing a schematic configuration of a server according to an embodiment. [Figure 5A] FIG. 5A is an operational flow showing a process of transmitting and receiving audio data and video data in a conference terminal and a server. [Figure 5B] FIG. 5B is an operational flow showing a process of playing back video data and audio data transmitted from the server in the conference terminal. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, a remote conference system and a remote conference method according to an embodiment will be described with reference to the drawings. In the drawings, the same or corresponding parts are designated by the same reference characters and description thereof will not be repeated.

[0011] Fig. 1 is a schematic diagram showing the configuration of a video conference system 1 (an example of a remote conference system) in this embodiment. As shown in Fig. 1, the video conference system 1 includes a conference terminal 10 for each of a plurality of participants who participate in the video conference, and a server 20. In this embodiment, it is assumed that four participants A to D participate in the video conference. When distinguishing between the conference terminals 10 of participants A to D, they are referred to as conference terminals 10A to 10D. Note that the number of participants in a video conference is not limited to four, and may be at least three or more.

[0012] The conference terminal 10 and the server 20 are connected to a communication line N such as a public line or a dedicated line. Participants use the conference terminal 10 to communicate with other participants in remote locations and hold a video conference. Each participant participates in the video conference as if he or she were seated in a seat designated on a seating chart in a preset virtual space. The configuration of the video conference system 1 will be specifically described below.

[0013] (Conference terminal 10 (10A to 10D)) 2 is a block diagram showing a schematic configuration of the conference terminal 10. In this embodiment, the conference terminal 10 may be a device such as a PC (Personal Computer), a tablet terminal, or a smartphone. The configuration of the conference terminal 10 will be described below using the conference terminal 10A of participant A as an example.

[0014] As shown in FIG. 2, the conference terminal 10A includes a microphone 11, a camera 12, a speaker 13, a communication unit 14 (an example of a first transmission unit and a reception unit), an operation unit 15 (an example of a selection reception unit), a memory unit 16, a display unit 17, and a control unit 18.

[0015] The microphone 11 collects the voice of the participant A and outputs an audio signal of the collected voice to the control unit 18.

[0016] The camera 12 captures an image of the participant A as a subject, and outputs an imaging signal to the control unit 18.

[0017] The speaker 13 performs D / A conversion on the audio signal output from the control unit 18, amplifies it, and emits the sound.

[0018] The communication unit 14 is a communication interface for communicating with the server 20 via the communication line N. Under the control of the control unit 18, the communication unit 14 establishes communication with the server 20 using a communication protocol such as RTP (Real-time Transport Protocol), and transmits and receives video data and audio data. Specifically, the communication unit 14 outputs the video data and audio data received from the server 20 to the control unit 18. The communication unit 14 also transmits the video data and audio data input from the control unit 18 to the server 20.

[0019] The operation unit 15 includes a mouse, a keyboard, a touch panel, etc. The operation unit 15 accepts an operation by the participant A, and outputs to the control unit 18 an operation signal indicating the accepted operation.

[0020] The storage unit 16 includes a non-volatile storage medium such as a hard disk. The storage unit 16 stores a seating chart 100a and seat information 100b. FIG. 3A is a diagram showing an example of the seating chart 100a. The seating chart 100a shows the seats of participants A to D in a virtual space. Specifically, in the seating chart 100a, identification information (e.g., names) for identifying the participants is written on seats S1 to S4 surrounding a circular virtual table T. The seats of participants A to D are given the same alphabet as those of participants A to D. The seating chart 100a is displayed on the display unit 17 during the video conference.

[0021] Fig. 3B is a diagram showing an example of the seating information 100b. The seating information 100b is information indicating the positions of the participants A to D in the virtual space, and corresponds to the seating chart 100a shown in Fig. 3A. As shown in Fig. 3B, the seating information 100b stores, for each participant, information (such as IP addresses) indicating the conference terminals 10 of the participants seated to the left and right of the participant.

[0022] The display unit 17 includes a display panel and a drive circuit for driving the display panel (both not shown). Under the control of the control unit 18, the drive circuit supplies drive signals to the display panel for displaying various images such as the seating chart 100a and images of the participants A to D.

[0023] The control unit 18 includes a CPU (Central Processing Unit) and memory (ROM (Read Only Memory) and RAM (Random Access Memory)). The control unit 18 functions as an audio / video processing unit 181 (an example of an audio data generating unit and a video data generating unit), a determination unit 182, a display control unit 183, and a volume adjustment unit 184 by the CPU executing a control program stored in the ROM.

[0024] The audio / video processor 181 includes a CODEC. The audio / video processor 181 sequentially transmits and receives packets of video data and audio data (an example of video information and audio information) during a video conference to and from the server 20 via the communication unit 21.

[0025] Specifically, the audio / video processor 181 converts the audio signal input from the microphone 11 at regular intervals and the video signal input from the camera 12 at regular intervals into digital data in accordance with the standard of the video conference system (e.g., H.323). The audio / video processor 181 then encodes the digital data to generate audio data and video data, and outputs the data to the determination unit 182.

[0026] Furthermore, the audio / video processor 181 decodes the video data and audio data from the server 20, which are sequentially input from the communication unit 14. The video data and audio data from the server 20 are multiplexed with the video data and audio data of the other conference terminals B to D. The audio / video processor 181 decodes the video data and audio data from the server 20 and separates them into packets of video data and audio data for each of the conference terminals B to D. The audio data of the conference terminals B to D includes volume information, and information such as destination information, source information, and timestamp is added to the audio data. The source information is the IP address of the conference terminal 10 that is the source of the audio data. The destination information includes either first destination information or second destination information. The first destination information is the IP addresses of all the conference terminals 10 except the own terminal, and the second destination information is the IP address of the other conference terminal 10.

[0027] Hereinafter, the voice data including the first destination information may be referred to as first voice data, and the voice data including the second destination information may be referred to as second voice data.

[0028] The audio / video processor 181 outputs the decoded and separated audio data for each conference terminal 10 to the volume adjuster 184 in order of the time stamps.

[0029] The volume adjustment unit 184 mixes each audio signal that has been gain-adjusted for each audio data based on the destination information and volume information included in the audio data for each conference terminal 10, and outputs the mixed audio signal from the speaker 13. Specifically, when each audio data from the conference terminals B to D (hereinafter, audio data B to D) is the first audio data, the volume adjustment unit 184 operates in a normal mode. The normal mode is a mode in which audio signals in which the overall audio has been adjusted to a uniform volume based on the volume information of each audio data are mixed and input to the speaker 13. In addition, when any of the audio data B to D is the second audio data, that is, when the first audio data and the second audio data are included, the volume adjustment unit 184 operates in a specific mode. The specific mode is a mode in which audio signals in which gain adjustment has been performed so that the volume of the first audio data is smaller than the volume of the second audio data are mixed and input to the speaker 13. In this embodiment, the audio / video processing unit 181, the volume adjustment unit 184, and the speaker 13 are examples of a reproduction unit.

[0030] The determination unit 182 determines the destination of the audio data based on the video data and audio data input from the audio / video processing unit 181 and the seating information 100b. Then, the determination unit 182 generates a packet of audio data to which header information including destination information indicating the determined destination is added, and a packet of video data to which header information including destination information indicating all other conference terminals 10 is added, and outputs the packets to the communication unit 14.

[0031] Specifically, when the volume of the voice data of participant A (hereinafter, voice data A) input from the voice / video processing unit 181 is equal to or greater than a threshold value, the determination unit 182 sets the first destination information indicating the IP addresses of all other conference terminals 10 as the destination information for the voice data A.

[0032] Furthermore, when the volume of the voice data A is less than the threshold, the determination unit 182 performs image analysis of the video data input at approximately the same timing as the voice data A, and sets the second destination information indicating the IP address of the one conference terminal 10 as the destination information of the voice data A. The one conference terminal 10 is determined based on the face direction of the participant A shown in the video data and the seat information 100b. That is, the determination unit 182 determines the conference terminal 10B or 10D of the participant who is seated in the direction (left or right) in which the face of the participant A is facing, among the participant B or participant D next to the participant A, as the destination. For example, when the participant A turns his face toward the participant D (see FIG. 3A) next to him on the right and speaks to him in a low voice so that the volume is less than the threshold, the IP address of the conference terminal 10D is set as the second destination information.

[0033] The display control unit 183 outputs each video data of participants A to D input from the audio / video processing unit 181 to the display unit 17, and causes the display unit 17 to display the video of participants A to D. Furthermore, the display control unit 183 causes the display unit 17 to display the seating chart 100a shown in FIG. 3A during the video conference. Note that the video of participants A to D may be displayed in the same arrangement as the arrangement of participants A to D in the seating chart 100a. In this embodiment, the display control unit 183 and the display unit 17 are an example of a playback unit.

[0034] (Server 20) Fig. 4 is a block diagram showing a schematic configuration of the server 20. As shown in Fig. 4, the server 20 includes a communication unit 21 (an example of an acquisition unit and a second transmission unit), a control unit 22, and a storage unit .

[0035] The communication unit 21 is a communication interface that communicates with the conference terminals 10A to 10D via the communication line N. Under the control of the control unit 22, the communication unit 21 establishes communication with the conference terminals 10A to 10D using a predetermined communication protocol such as RTP, and transmits and receives video data and audio data.

[0036] The storage unit 23 includes a non-volatile storage medium such as a hard disk, etc. The storage unit 23 stores conference terminal information (not shown) including identification information (such as IP addresses) of the conference terminals 10A to 10D.

[0037] The control unit 22 includes a CPU and memory (ROM and RAM). The CPU executes a control program stored in the ROM, whereby the control unit 22 communicates with each conference terminal 10 via the communication unit 21. Specifically, the control unit 22 acquires packets of audio data (first audio data or second audio data) and video data transmitted from each conference terminal 10, and transmits the packets to another conference terminal 10 that is different from the source of the packets of audio data and video data.

[0038] That is, the video data transmitted to the conference terminal 10A is video data in which the video data B to D transmitted from the conference terminals 10B to 10D are multiplexed. In addition, when the audio data A to D acquired from the conference terminals 10A to 10D are the first audio data, audio data in which the audio data B to D are multiplexed is transmitted to the conference terminal 10A. In addition, when, for example, the audio data A among the audio data A to D is the second audio data and the audio data A includes the second destination information indicating the conference terminal 10D, audio data in which the audio data B and C (first audio data) and the audio data A (second audio data) are multiplexed is transmitted to the conference terminal 10D. In this case, audio data in which the audio data B to D (first audio data) are multiplexed is transmitted to the conference terminal 10A. Audio data in which the audio data C and D (first audio data) are multiplexed is transmitted to the conference terminal 10B. Audio data in which the audio data B and D (first audio data) are multiplexed is transmitted to the conference terminal 10C.

[0039] (operation) Fig. 5A is an operational flow showing the transmission and reception process of audio data and video data in the conference terminal 10A and the server 20. Fig. 5B is an operational flow showing the reproduction process of video data and audio data from the server 20 in the conference terminal 10D. Note that in Figs. 5A and 5B, it is assumed that the seating chart 100a shown in Fig. 3A is displayed on the display unit 17 of each conference terminal 10.

[0040] 5A, during a video conference, conference terminal 10A collects the voice of participant A with microphone 11 and captures an image of participant A with camera 12. Control unit 18 in conference terminal 10A sequentially acquires audio data A and video data A by A / D converting and encoding the audio signal collected by microphone 11 and the video signal captured by camera 12 in audio / video processing unit 181 (step S11).

[0041] If the volume of the audio data A is equal to or greater than the threshold (step S12: Yes), the control unit 18 causes the decision unit 182 to generate packets of the audio data A (first audio data) and the video data A to which first destination information indicating the conference terminals 10B to 10D and source information indicating the own terminal, etc. are added, and transmits the packets to the server 20 via the communication unit 14 (step S13).

[0042] Furthermore, if the volume of the audio data A is not greater than or equal to the threshold (step S12: No), the control unit 18 causes the decision unit 182 to generate packets of the second audio data and video data A and transmit them to the server 20 via the communication unit 14 (step S14).

[0043] Specifically, the determination unit 182 refers to the seating information 100b (FIG. 3B), performs image analysis of the acquired video data, and determines the conference terminal 10 corresponding to the face direction of the participant A as the destination. Then, the determination unit 182 generates a packet of audio data A (second audio data) to which second destination information indicating the IP address of the determined conference terminal 10 and source information indicating the terminal itself are added. The determination unit 182 also generates a packet of video data A to which destination information indicating the IP addresses of all other conference terminals 10 and source information indicating the terminal itself are added. The determination unit 182 inputs the packets of audio data A and video data A to the communication unit 14. The communication unit 14 transmits the packets of audio data A (second audio data) and video data A input from the determination unit 182 to the server 20.

[0044] That is, for example, during a video conference, when participant A wants to talk to participant D, participant A turns his face to the right and speaks in a low voice. In this case, the conference terminal 10D is determined as the destination of the voice data (voice data A) of participant A, and a packet of the voice data A (second voice data) to which second destination information indicating the IP address of the conference terminal 10D and the like is added is transmitted to the server 20 together with the packet of the video data A.

[0045] The control unit 18 repeats the processes from step S11 onwards until the video conference is ended (step S15: No), and ends the processes when the video conference is ended (step S15: Yes).

[0046] The server 20 sequentially acquires the voice data (voice data A to D) and the video data (video data A to D) from the conference terminals 10A to 10D via the communication unit 21 (step S21).

[0047] If the acquired voice data A to D includes a packet of the second voice data (step S22: Yes), the control unit 22 multiplexes and transmits packets of the second voice data and the first voice data to the conference terminal 10 indicated by the second destination information in the packet of the second voice data, and multiplexes and transmits packets of the first voice data to other conference terminals 10 (step S23).

[0048] That is, when the packet of the audio data A includes the second destination information indicating the IP address of the conference terminal 10D, audio data in which packets of the audio data A (second audio data) and packets of the audio data B to D (first audio data) are multiplexed, and video data in which packets of the video data A to C are multiplexed are transmitted to the conference terminal 10D. The audio data and video data in which packets of the audio data B to D and packets of the video data B to D are multiplexed, respectively, are transmitted to the conference terminal 10A. The audio data and video data in which packets of the audio data C and D and packets of the video data A, C, and D are multiplexed, respectively, are transmitted to the conference terminal 10B. The audio data and video data in which packets of the audio data B and D and packets of the video data A, B, and D are multiplexed, respectively, are transmitted to the conference terminal 10C. In this way, only packets of the first audio data from the conference terminals 10 other than the own terminal are multiplexed and transmitted to the conference terminal 10 that is not the destination of the second audio data.

[0049] Then, the control unit 22 transmits to each conference terminal 10, via the communication unit 21, the video data obtained by multiplexing the packets of the video data acquired from the other conference terminals 10 (step S24). That is, the video data obtained by multiplexing the packets of the video data B to D is transmitted to the conference terminal 10A, and the video data obtained by multiplexing the packets of the video data A, C, and D is transmitted to the conference terminal 10B. Moreover, the video data obtained by multiplexing the packets of the video data A, B, and D is transmitted to the conference terminal 10C, and the video data obtained by multiplexing the packets of the video data A to C is transmitted to the conference terminal 10D.

[0050] Furthermore, if the acquired packets of the voice data A to D do not include packets of the second voice data (step S22: No), that is, if only packets of the first voice data are acquired, the control unit 22 transmits to each conference terminal 10, via the communication unit 21, voice data obtained by multiplexing the packets of the voice data acquired from the other conference terminals 10 (step S25). That is, voice data obtained by multiplexing the packets of the voice data B to D is transmitted to the conference terminal 10A, and voice data obtained by multiplexing the packets of the voice data A, C, and D is transmitted to the conference terminal 10B. Moreover, voice data obtained by multiplexing the packets of the voice data A, B, and D is transmitted to the conference terminal 10C, and voice data obtained by multiplexing the packets of the voice data A to C is transmitted to the conference terminal 10D.

[0051] The control unit 22 repeats the processes from step S21 onwards until the video conference ends (step S26: No), and ends the processes when the video conference ends (step S26: Yes).

[0052] For convenience of explanation, the conference terminal 10A has been described as an example in FIG. 5A, but the conference terminals 10B to 10D also perform the same process as the conference terminal 10A in response to the speech of the participants B to D.

[0053] Next, the audio and video playback process in the conference terminal 10D will be described with reference to Fig. 5B. The conference terminal 10D sequentially acquires multiplexed audio data and video data from the server 20 via the communication unit 14 (step S31).

[0054] The control unit 18 of the conference terminal 10D decodes the acquired audio data and video data in the audio / video processing unit 181, and separates the packets of the audio data A to C and the video data A to C, respectively (step S32).

[0055] If the packets of the audio data A to C include only packets of the first audio data (step S33: Yes), the control unit 18 operates the volume adjustment unit 184 in normal mode (step S33). That is, the volume adjustment unit 184 mixes audio signals in which the gains of the audio data are adjusted so that the volumes of the audio data A to C are uniform, and outputs the mixed audio signals from the speaker 13.

[0056] In step S33, if the packets of the audio data A to C include packets of the second audio data (step S33: No), the control unit 18 operates the volume adjustment unit 184 in a specific mode (step S35). The audio data A is the second audio data whose destination is the conference terminal D. The volume adjustment unit 184 mixes audio signals obtained by adjusting the gains of the audio data A to C so that the volumes of the audio data B and C are smaller than the volume of the audio data A, and inputs the mixed audio signals to the speaker 13. The speaker 13 amplifies and outputs the audio signal input from the volume adjustment unit 184. This makes it easier for participant D to hear the voice of participant A, and can also hear the voices of participants B and C other than participant A.

[0057] Then, the control unit 18 outputs each of the video data A to C to the display control unit 183 via the audio / video processing unit 181, and causes the display unit 17 to display the videos of the participants A to C, respectively (step S36).

[0058] The control unit 18 repeats the processes from step S31 onwards until the video conference ends (step S37: No), and when the video conference ends (step S37: Yes), ends the playback process.

[0059] For convenience of explanation, the conference terminal 10D has been described as an example in FIG. 5B. However, the conference terminals 10A to 10C also perform the same process as the conference terminal 10D in response to the voice data acquired from the server 20.

[0060] In this embodiment, during a video conference, each participant can turn his / her face toward the participant next to him / her in the virtual space and speak in a low voice, so that the voice can be sent only to the specific participant. Therefore, some conference participants can talk to each other without interrupting the progress of the video conference. In addition, the voices of other participants are reproduced quieter than the voices of the conversations between some participants, so that the conversations between some participants are easier to hear. Since some conference participants can talk to each other while listening to the voices during the video conference, a sense of realism similar to that of an actual face-to-face conference can be obtained compared to when the voices during the video conference cannot be heard at all.

[0061] The above describes the embodiments of the remote conference system and the remote conference method according to the present invention. However, the remote conference system and the remote conference method are not limited to the above embodiments, and can be implemented in various forms without departing from the gist of the present invention. The drawings are mainly schematic illustrations of the respective components for ease of understanding, and the thickness, length, number, etc. of each component shown in the drawings are different from the actual ones due to the convenience of drawing. Furthermore, the shape, dimensions, etc. of each component shown in the above embodiments are merely examples, and are not particularly limited, and various modifications are possible within a range that does not substantially deviate from the effects of the present invention. Below, modified examples of the above embodiments are described.

[0062] [Variations] (1) When playing back the second voice data in the conference terminal 10, the display mode of the video of each participant displayed on the display unit 17 may be changed so that the sender of the second voice data can be recognized. That is, for example, the video of the participant corresponding to the sender of the second voice data may be displayed larger than the videos of the other participants, or a mark indicating that the second voice data is being played back may be superimposed on the video of the participant corresponding to the sender of the second voice data.

[0063] (2) When the volume of the audio data is equal to or lower than a threshold, the determination unit 182 may determine the destination of the audio data based on the participant's operation via the operation unit 15 in addition to the orientation of the participant's face shown in the video data. The participant's operation may be, for example, an operation of designating one of seats S1 to S4 in the seating chart 100a displayed on the display unit 17 with a mouse or the like. By configuring in this way, the destination of the audio data can be determined more reliably and flexibly.

[0064] (3) The voice data transmitted from the server 20 to the conference terminal 10 may include second voice data transmitted from a plurality of different conference terminals 10. For example, when the participant A and the participant C each speak softly to the participant D, the server 20 transmits voice data obtained by multiplexing packets of the voice data A and the voice data C (second voice data) and packets of the voice data B (first voice data) to the conference terminal 10D. The conference terminal 10D adjusts the gain in the volume adjustment unit 184 so that the volume of the voice data B is smaller than the volumes of the voice data A and the voice data C, mixes the gain-adjusted voice signals of the voice data A to C, and outputs them from the speaker 13. Note that the volume of either the voice data A or the voice data C may be made larger than the other according to a predetermined priority.

[0065] (4) The volume adjustment at the conference terminals 10 may be performed by the server 20, and the audio data after the volume adjustment may be multiplexed and transmitted to each conference terminal 10.

[0066] (5) The video conference system 1 may be applied to a WEB conference system (an example of a remote conference system) in which the conference terminals 10 and the server 20 are connected via the Internet. In this case, each conference terminal 10 establishes an RTP session with the other conference terminals 10 that are holding the video conference, and transmits RTP packets containing the video data and audio data that have been encoded and subjected to a predetermined encryption process to the server 20. Each conference terminal 10 decodes and separates the multiplexed audio data and video data from the server 20, and performs a decryption process to play back the audio data and video data.

[0067] (6) In the conference terminal 10, when generating packets of voice data, if the voice of the voice data is silent, silence information indicating silence may be added to the voice data. In this case, for example, after receiving voice data including the second voice data from the conference terminal 10A from the server 20 in step S33, the conference terminal 10D may operate in the specific mode until a predetermined number of packets of the second voice data to which silence information has been added are consecutively received. When the conference terminal 10A has consecutively received the predetermined number of packets of the second voice data to which silence information has been added, the conference terminal 10A ends the specific mode.

[0068] (7) In the conference terminal 10, for example, when a participant performs a video-off operation to not transmit video to the server 20, the conference terminal 10 may transmit to the server 20 a predetermined image signal including identification information identifying the participant or the conference terminal 10, or a video-off signal indicating video-off. The server 20 may transmit the predetermined image signal or video-off signal received from the conference terminal 10 to another conference terminal 10, and cause the other conference terminal 10 to display an image corresponding to the predetermined image signal or video-off signal. [Industrial Applicability]

[0069] The present invention can be used in remote conferences such as video conferences and web conferences using terminals of at least three or more participants. [Explanation of symbols]

[0070] 1. Video conferencing system 10, 10A~10D Conference terminals 11. Mike 12 Camera 13 Speaker 14, 21 Communications Department 15 Control section 16, 23 Storage section 17 Display section 18, 22 Control section 20 Servers 181 Audio and video processing unit 182 Decision Section 183 Display control section 184 Volume control section

Claims

1. a first generation step of acquiring voices of at least three or more participants at each conference terminal of the remote conference and generating voice data including volume information from the acquired voices; a second generation step of acquiring video of the participants during the conference and generating video data for each of the conference terminals of the participants; a display step of displaying seat information indicating a seat in a virtual space for each of the participants for each of the conference terminals of the participants; a determining step of determining, for each of the conference terminals of the participants, a destination of transmission of the voice data of the participants based on the seat information, the volume information of the voice data of the participants, and the video data; a transmitting step of transmitting the voice data of each of the participants based on the destination determined for the voice data of the participant for each of the conference terminals of the participants; A remote conferencing method comprising:

2. 2. The remote conference method according to claim 1, wherein said transmitting step transmits, to the conference terminal of each of said participants, said video data from conference terminals of participants other than said participant, based on a source of said audio data for each of said participants.

3. The determining step includes setting, as a destination of the voice data, either a first destination indicating the conference terminals of all the participants or a second destination indicating a conference terminal of one of the conference terminals of all the participants; the first destination is set when the volume information of the audio data is greater than a predetermined threshold; the second destination is set when the volume information of the audio data is equal to or less than the predetermined threshold; 3. The remote conference method according to claim 2, wherein the conference terminal of the one participant is a conference terminal that corresponds to a face direction of the one participant shown in the video data acquired together with the audio data in the seating information.

4. The method further includes a reproduction step of receiving the voice data of the other participants and reproducing the voice data for each of the conference terminals of the participants; 4. The remote conferencing method according to claim 3, wherein, when the audio data received in the playback step includes audio data including the first destination and audio data including the second destination, the playback step plays the audio data including the first destination at a lower volume than the audio data including the second destination.

5. The reproducing step includes receiving the video data of the other conference terminal, 5. The remote conference method according to claim 4, wherein the display step displays the video data generated in the second generation step and the video data of the other conference terminal received in the reproduction step based on the seat information.

6. The reproducing step includes receiving, together with the voice data of the other conference terminal, transmission source information indicating a transmission source of the voice data, and 6. The remote conferencing method according to claim 5, wherein, when the audio data received in the playback step includes the second destination, the display step displays the video data corresponding to the source information of the audio data including the second destination in a distinguishable manner from other video data.

7. The method further includes a selection receiving step of receiving a selection of the participant in the seat information displayed in the display step for each of the participant's conference terminals; 7. The remote conference method according to claim 1, wherein the determining step determines a destination of the audio data based on the selection of the participant accepted in the selection accepting step and the video data.

8. A conference terminal for each of at least three or more participants participating in a remote conference; a server connected to the conference terminal for each participant via a communication line; Equipped with The conference terminal includes: A storage unit for storing seat information indicating a seat in the virtual space for each of the participants; A display unit that displays the seat information; a voice data generating unit that acquires voices of the participants and generates voice data including volume information from the acquired voices; a video data generating unit that acquires video of the participants during the conference and generates video data; a determination unit that determines a destination of the audio data based on the seat information, the volume information of the audio data, and the video data; a first transmission unit that transmits, to the server, audio information including the audio data associated with destination information indicating the determined destination and video information including the video data together with source information indicating the terminal; Including, The server, an acquisition unit that acquires the audio information, the video information, and the transmission source information from each conference terminal; a second transmission unit that transmits the audio information to each of the conference terminals based on the acquired destination information of the audio information from each of the conference terminals; A remote conferencing system.

Images