Remote conference system and remote conference method

Abstract

To reproduce audio data with a sound image localization intended by a user while changing the sound image localization according to the amount of communication between users.SOLUTION: A remote conference system includes a storage unit, a table management unit, an audio receiving unit, a sound image localization management unit, a sound image localization processing unit, and a priority conversation management unit. When an instruction is input from a first user to designate a second user as a priority target, the priority conversation management unit calculates a priority weight by adding a first value to a weight corresponding to a pair of the first user and the second user. When the instruction is input, the sound image localization management unit determines a sound image localization of the audio data to be reproduced on a terminal device of the second user on the basis of the priority weight.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a remote conference system and a remote conference method. [Background technology]

[0002] Conventionally, a system is known that judges the strength of a relationship based on the amount of communication between users and automatically changes the sound image localization. For example, in the sound image localization control system disclosed in Patent Document 1, a communication server has a relationship graph in which weights are associated with identifiers of two user terminals, and controls the weights according to the amount of communication between the users of both terminals. A multipoint connection device receives audio data, adaptively determines the sound image localization according to the weights of the relationship graph, and performs sound image localization processing on the audio data based on the determined sound image localization. The audio data that has been subjected to the sound image localization processing is transmitted from the multipoint connection device to the user terminal and played back on the user terminal. This makes it possible to automatically change the sound image localization and volume of a certain user relative to another user in a virtual space according to the amount of communication between the users. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2013-17027 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the method disclosed in Patent Document 1, the sound image localization is automatically determined according to the amount of communication between users, so there is a risk that audio data will be played back with a sound image localization that is not intended by the user. Furthermore, in the method disclosed in Patent Document 1, the weights are controlled according to the amount of communication between users that is specified in advance. Therefore, it is difficult to reflect the sound image localization desired by the user in real time, and there is a risk that audio data will be played back with a sound image localization that is not intended by the user.

[0005] One aspect of the present disclosure aims to provide a remote conferencing system and a remote conferencing method that can change sound image localization depending on the amount of communication between users and play back audio data with sound image localization intended by the user. [Means for solving the problem]

[0006] One aspect of the present disclosure is a remote conference system for conducting a remote conference between a plurality of terminal devices, the system including: a memory unit that stores a table in which a weight is associated with each pair of any two users of a plurality of users; a table management unit that updates the weight corresponding to the pair in the table based on information transmitted and received between the terminal devices of the two users forming the pair; an audio receiving unit that receives audio data from the terminal device of a first user; a sound image localization management unit that determines a sound image localization of the audio data based on the weight corresponding to the pair in the table; a sound image localization processing unit that performs sound image localization processing on the received audio data based on the determined sound image localization and transmits the data to the terminal device of another user forming the pair with the first user; and a priority conversation management unit that, when an instruction to designate a second user as a priority target is input from the first user, adds a first value to the weight corresponding to the pair of the first user and the second user to calculate a priority weight, and when the instruction is input, the sound image localization management unit determines the sound image localization of the audio data to be played on the terminal device of the second user based on the priority weight.

[0007] Another aspect of the present disclosure is a remote conference method for conducting a remote conference between a plurality of terminal devices, the method including: updating a weight corresponding to each pair of any two users of a plurality of users in a table in which a weight is associated with the pair based on information transmitted and received between the terminal devices of the two users forming the pair; receiving voice data from the terminal device of a first user; determining a sound image localization of the voice data based on the weight corresponding to the pair in the table; applying sound image localization processing to the received voice data based on the determined sound image localization, and transmitting the received voice data to the terminal device of another user forming the pair with the first user; and, when an instruction to designate a second user as a priority target is input from the first user, adding a first value to the weight corresponding to the pair of the first user and the second user to calculate a priority weight, and determining the sound image localization includes, when the instruction is input, determining a sound image localization of the voice data to be played on the terminal device of the second user based on the priority weight. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic diagram of a remote conference system. [Diagram 2] FIG. 2 is a block diagram showing the electrical configuration of the remote conference system. [Diagram 3] FIG. 4 is a diagram showing a data structure of a relationship table. [Figure 4A] FIG. 2 is a diagram showing an example of a sound image localization position of a user in a virtual space. [Figure 4B] FIG. 4B is a diagram showing a state of sound image localization as seen by user B shown in FIG. 4A. [Diagram 5] FIG. 2 is a diagram showing the process and data flow during normal conversation in the remote conference system. [Figure 6] 11 is a flowchart of a priority conversation process according to the first embodiment. [Figure 7A] 13 is an example of a screen displayed on a terminal device of a first user. [Figure 7B] 13 is an example of a screen displayed on a terminal device of a second user. [Figure 8]13 is a diagram showing a data configuration of a relationship table during priority conversation in the first embodiment. FIG. [Figure 9] FIG. 2 is a diagram showing the process and data flow during priority conversation in the remote conference system. [Figure 10A] FIG. 13 is a diagram showing a sound image localization state as seen by user B during priority conversation in the first embodiment. [Figure 10B] 5A and 5B are diagrams illustrating changes in sound image localization positions during priority conversation in the first embodiment. [Figure 11A] 13 is a flowchart of a priority conversation process according to the second embodiment. [Figure 11B] 13 is a flowchart of a priority conversation process according to the second embodiment. [Figure 12] FIG. 11 is a diagram showing a data configuration of a relationship table during priority conversation in the second embodiment. [Figure 13A] FIG. 11 is a diagram showing a sound image localization state as seen by a user C during priority conversation in the second embodiment. [Figure 13B] 13A and 13B are diagrams illustrating changes in sound image localization positions during priority conversation in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the drawings, the same or equivalent elements are designated by the same reference numerals, and duplicated explanations will be omitted.

[0010] [First embodiment] A first embodiment of the present disclosure will be described. Fig. 1 is a schematic diagram of a remote conference system 100. As shown in Fig. 1, the remote conference system 100 is a system for executing a remote conference among a plurality of terminal devices 120. The remote conference system 100 includes a server 110, a plurality of terminal devices 120, a multipoint connection device 130, and the like, which are connected via a network 101 (e.g., the Internet). In this embodiment, a video conference (television conference) in which two-way communication by voice and image is performed among the plurality of terminal devices 120 will be exemplified as a remote conference executed in the remote conference system 100.

[0011] The server 110 is a computer for adaptively controlling the sound image localization of the sound reproduced on each terminal device 120, and includes a control unit 111, a storage unit 112, and a communication unit 113. The control unit 111 is a processor such as a CPU, for example, and operates according to a program stored in the storage unit 112. The storage unit 112 is an information recording medium such as a hard disk, ROM, or RAM, for example, and is an information recording medium that holds the program executed by the control unit 111. The communication unit 113 is a network interface, and transmits and receives information via the network 101 in response to an instruction from the control unit 111.

[0012] Each terminal device 120 is an electronic device or computer used by a user to hold a remote conference, and examples of such devices include a personal computer, a smartphone, a tablet terminal, a display device, etc. Although not shown, each terminal device 120 includes a control unit, a storage unit, a communication unit, etc., similar to the server 110. In the example of Fig. 1, three terminal devices 120 used by three users are shown as the multiple terminal devices 120.

[0013] The multipoint control unit 130 is an MCU (Multipoint Control Unit) that relays data from each terminal device 120. Like the server 110, the multipoint control unit 130 includes a control unit 131, a storage unit 132, and a communication unit 133.

[0014] [Terminal device configuration] 2 is a block diagram showing an electrical configuration of the remote conference system 100. In each terminal device 120, the control unit executes a program stored in the storage unit to realize the functional blocks of a communication unit 221, an audio transmission unit 222, and an audio reproduction unit 223. Peripheral devices (display, camera, keyboard, mouse, etc.) 224, a microphone 225, and a surround speaker 226 are connected to each terminal device 120.

[0015] The communication unit 221 provides various communication functions (e.g., a file sharing function, a message function, a chat function, etc.) to the user in the remote conference. For example, the communication unit 221 transmits a data file, a source user identifier, and a destination user identifier designated by the user to the server 110 in accordance with a file sharing instruction from the user. Upon receiving the data file and the source user identifier from the server 110, the communication unit 221 displays the data file and the source user identifier on the peripheral device 224.

[0016] Furthermore, in accordance with a user's instruction to send a message, the communication unit 221 transmits the instant message, the identifier of the source user, and the identifier of the destination user designated by the user to the server 110. Upon receiving the instant message and the identifier of the source user from the server 110, the communication unit 221 displays the instant message and the identifier of the source user on the peripheral device 224.

[0017] The audio transmitting unit 222 encodes the audio input from the microphone 225 and transmits the encoded audio data together with the identifier of the source user to the multipoint control unit 130. When the audio reproducing unit 223 receives the audio data that has been subjected to sound image localization processing from the multipoint control unit 130, it reproduces the surround audio via a plurality of surround speakers 226 so that the user can feel the sound image localization.

[0018] [Server configuration] 2, in server 110, control unit 111 executes a program stored in storage unit 112 to realize functional blocks of information management unit 211, priority conversation management unit 212, and table management unit 214. Storage unit 112 of server 110 stores relationship table 213 in which a weight is associated with each pair of any two users out of a plurality of users.

[0019] The information management unit 211 manages information exchange based on the communication function described above. For example, when the information management unit 211 receives a data file, a source user identifier, and a destination user identifier from the terminal device 120 on which the file sharing function has been executed, the information management unit 211 transmits the received data file and source user identifier to the destination user's terminal device 120. Then, the information management unit 211 transmits the source user identifier, the destination user identifier, and the current time to the table management unit 214.

[0020] Furthermore, when the information management unit 211 receives an instant message, the source user's identifier, and the destination user's identifier from the terminal device 120 on which the message function has been executed, it transmits the received instant message and the source user's identifier to the destination user's terminal device 120. Then, the information management unit 211 transmits the source user's identifier, the destination user's identifier, and the current time to the table management unit 214.

[0021] Fig. 3 is a diagram showing the data configuration of the relationship table 213. As shown in Fig. 3, in the relationship table 213, a weight and a sound image localization are associated with each pair of two user identifiers. The user identifier is information capable of uniquely identifying a user, and is an email address specific to the user in this embodiment. Alternatively, the user identifier may be a name specific to the user, or an identification number or model name of the terminal device 120 used by the user.

[0022] The initial value of the weight is a predetermined constant value (e.g., 50), but a value calculated based on a predetermined rule may be set. The sound image localization changes according to the current weight value, as described later. The weight is related to the sound image localization management and sound image localization processing, which will be described later, and is used as an index indicating the amount of recent communication between users.

[0023] Based on information transmitted and received between the terminal devices 120 of the two users forming a pair, the table management unit 214 updates the weight corresponding to the pair in the relationship table 213. For example, when the table management unit 214 receives the identifier of the source user, the identifier of the destination user, and the current time from the information management unit 211, it adds a predetermined value to the weight corresponding to the pair of the received two identifiers in the relationship table 213.

[0024] At this time, the table management unit 214 may change the weight value to be added depending on the type of communication function used between the source user and the destination user. For example, when the table management unit 214 receives the above information based on the file sharing function from the information management unit 211, it may add "30" to the corresponding weight, and when the table management unit 214 receives the above information based on the message function from the information management unit 211, it may add "10" to the corresponding weight. Furthermore, the table management unit 214 may subtract a predetermined value (e.g., "1") from all weights in the relationship table 213 every predetermined time (e.g., every 30 minutes).

[0025] When a change occurs in the weight in the relationship table 213, the table management unit 214 transmits two user identifiers associated with the changed weight to the multipoint control unit 130. Details of the priority conversation management unit 212 will be described later.

[0026] [Multipoint connection device configuration] In the multipoint control device 130, the control unit 131 executes a program stored in the memory unit 132, thereby realizing each of the functional blocks of an audio receiving unit 231, a sound image localization management unit 232, and a sound image localization processing unit 233.

[0027] The voice receiving unit 231 receives voice data from the terminal device 120 of the first user. In this example, the voice receiving unit 231 receives the voice data and the identifier of the source user from the terminal device 120, and outputs these data to the sound image localization processing unit 233.

[0028] The sound image localization management unit 232 determines the sound image localization of the audio data based on the weighting corresponding to the pair in the relationship table 213. In this example, the sound image localization management unit 232 receives from the server 110 the identifier of the source user and the identifier of the destination user whose weighting in the relationship table 213 has changed. The sound image localization management unit 232 refers to the relationship table 213, acquires the weighting corresponding to the pair of the received two identifiers, and adaptively determines the sound image localization based on the weighting. The sound image localization management unit 232 outputs the determined sound image localization to the sound image localization processing unit 233, and updates the sound image localization in the relationship table 213 to the determined sound image localization.

[0029] 4A is a diagram showing an example of a sound image localization position of a user in a virtual space. The sound image localization management unit 232 localizes the sound image of another user around each user in the virtual space based on the relationship table 213. For example, the sound image localization management unit 232 specifies a weight of another user corresponding to a first user in the relationship table 213. The sound image localization management unit 232 localizes the sound image of the other user at one of positions P1 to P6 (see FIG. 4B) around the first user based on the specified weight. In this example, the larger the corresponding weight of the other user, the closer the sound image is to the front side and the closer to the first user.

[0030] In the example of Fig. 4A, the sound images of other users A, C, and D are localized around user B. Fig. 4B is a diagram showing the state of sound image localization as seen by user B shown in Fig. 4A. As shown in Fig. 4B, positions P1 and P2 are in front of user B, and position P1 is closer to user B than position P2. Therefore, the sound image at position P1 is reproduced at a louder volume on the terminal device 120 of user B than the sound image at position P2.

[0031] Positions P3 and P5 are on the left side of user B, and position P3 is closer to user B than position P5. Therefore, the sound image at position P3 is reproduced at a louder volume on the terminal device 120 of user B than the sound image at position P5. Positions P4 and P6 are on the right side of user B, and position P4 is closer to user B than position P6. Therefore, the sound image at position P4 is reproduced at a louder volume on the terminal device 120 of user B than the sound image at position P6.

[0032] In this example, the sound volumes at positions P3 and P4 are about the same, and the sound volumes at positions P5 and P6 are about the same. However, in the terminal device 120 of user B, the sound is easier to hear from the front side of user B than from the left and right sides of user B. Therefore, the sound image at position P1 is reproduced at a louder volume on the terminal device 120 of user B than the sound images at positions P3 and P4. The sound image at position P2 is reproduced at a louder volume on the terminal device 120 of user B than the sound images at positions P5 and P6.

[0033] 4A and 4B, as seen from user B, the sound images of other users A, C, and D are localized at positions P5, P6, and P2, respectively. In this case, positions P5, P6, and P2 are all relatively far away from user B. In the terminal device 120 of user B, the voice data of users A, C, and D is reproduced at a relatively low volume. However, since user D is in front of user B, the voice data of user D is reproduced at a louder volume than the voice data of users A and C.

[0034] In the above example, six types of positions for sound image localization are illustrated, but two or more types of positions for sound image localization may be used. In addition, the position for sound image localization based on the weight is not limited to the above example, and various methods can be applied.

[0035] The sound image localization processing unit 233 performs sound image localization processing on the received audio data based on the determined sound image localization, and transmits the data to the terminal device 120 of the other user paired with the first user. In this example, the sound image localization processing unit 233 performs sound image localization processing for reproducing well-known surround sound on the audio data acquired from the audio receiving unit 231 based on the sound image localization determined by the sound image localization management unit 232. The sound image localization processing unit 233 transmits the audio data that has been subjected to sound image localization processing to the terminal device 120 connected to the multipoint connection unit 130. In the terminal device 120 that receives this audio data, the audio reproduction unit 223 reproduces the audio data, allowing the users participating in the remote conference to hear the surround sound.

[0036] [System operation during normal conversation] Fig. 5 is a diagram showing the process and data flow during a normal conversation in the remote conference system 100. In the example of Fig. 5, a user of terminal device A and a user of terminal device B participate in a remote conference at two terminal devices 120. In this remote conference, the user of terminal device A transmits data files and instant messages to the user of terminal device B. These data files and instant messages, as well as information based on other communication tools (e.g., chat data), are collectively referred to as communication data. In other words, the user of terminal device A is the source user of the communication data, and the user of terminal device B is the destination user of the communication data.

[0037] 5, in S501, the communication unit 221 of the terminal device A transmits the communication data, the source user identifier, and the destination user identifier to the server 110. In S503, the information management unit 211 of the server 110 transmits the communication data and the source user identifier received from the terminal device A to the terminal device B.

[0038] In S505, the communication unit 221 of the terminal device B displays the communication data received from the server 110 on the peripheral device 224. In S507, the table management unit 214 of the server 110 manages the relationship table 213. At this time, in the relationship table 213, as described above, the weights corresponding to the pair of identifiers of the source user and destination user of the communication data (i.e., the identifiers of the users of the terminal devices A and B) are updated.

[0039] In S509, the audio transmitting unit 222 of the terminal device A transmits the audio data and the identifier of the source user to the multipoint control unit 130. In S511, the audio transmitting unit 222 of the terminal device B transmits the audio data and the identifier of the source user to the multipoint control unit 130. In S513, the table management unit 214 of the server 110 transmits a set of user identifiers (i.e., the identifiers of the users of the terminal devices A and B) corresponding to the weights changed in the relationship table 213 to the multipoint control unit 130.

[0040] In S515, as described above, in the multipoint control device 130, the sound image localization management unit 232 determines the sound image localization of the user of terminal device A, and the sound image localization processing unit 233 performs sound image localization processing of the voice data of terminal device A received in S509. Also, the sound image localization management unit 232 determines the sound image localization of the user of terminal device B, and the sound image localization processing unit 233 performs sound image localization processing of the voice data of terminal device B received in S511.

[0041] In S517, the sound image localization processing unit 233 transmits the audio data of terminal device A that has been subjected to sound image localization processing to terminal device B. In S519, the sound image localization processing unit 233 transmits the audio data of terminal device B that has been subjected to sound image localization processing to terminal device A. In S521, the audio reproduction unit 223 of terminal device A reproduces the audio data of terminal device B in surround sound. In S523, the audio reproduction unit 223 of terminal device B reproduces the audio data of terminal device A in surround sound.

[0042] [Priority conversation processing] 6 is a flowchart of the priority conversation process according to the first embodiment. The priority conversation process is a process in which a first user allows a second user to preferentially listen to the voice of the first user in a remote conference in which multiple users including a first user and a second user participate. As described below, when a priority conversation instruction designating the second user as a priority target is input from the first user, the priority conversation management unit 212 of the server 110 calculates a priority weight by adding a first value to the weight corresponding to the pair of the first user and the second user.

[0043] As shown in Fig. 6, in S601, the priority conversation management unit 212 determines whether a priority conversation instruction has been input. In this example, each of the multiple terminal devices 120 displays a conference screen 700 that displays multiple users participating in the remote conference. The priority conversation management unit 212 determines that a priority conversation instruction has been input when a first user performs an operation on the conference screen 700 of the terminal device 120 to select a second user from the multiple users as a priority target. In the following example, of the multiple users A to D participating in the remote conference, the first user who inputs a priority conversation instruction is user A, and the second user designated as a priority target in this priority conversation instruction is user B.

[0044] Fig. 7A is an example of a screen displayed on the terminal device 120 of the first user. As shown in Fig. 7A, in the terminal device 120 of user A, who is the first user, icons 710 for designating other users B, C, and D as priority targets are provided in each of the display areas of the conference screen 700. For example, when the user operates cursor 720 to select icon 710 corresponding to user B, a priority conversation instruction in which user A designates user B as a priority target is transmitted from terminal device 120 to server 110. In server 110 that receives the priority conversation instruction, priority conversation management unit 212 determines that a priority conversation instruction has been input.

[0045] In this example, when a user selects an icon 710, the display mode (e.g., color, size, shape, etc.) of the icon 710 is changed. In the example of Fig. 7, the color of the icon 710 corresponding to user B is changed from white to black. This allows user A to visually recognize on the screen that user B has been selected as a priority target.

[0046] Furthermore, when a priority conversation instruction is input, the priority conversation management unit 212 may display an image or message indicating that the first user has designated the second user as a priority target on the conference screen 700 of the terminal device 120 used by the second user. Fig. 7B is an example of a screen displayed on the terminal device 120 of the second user. As shown in Fig. 7B, icons 710 are also provided in the display areas of the other users A, C, and D included in the conference screen 700 on the terminal device 120 of user B, who is the second user.

[0047] In response to input of a priority conversation instruction on the terminal device 120 of user A, the priority conversation management unit 212 causes the terminal device 120 of user B, designated as a priority target, to display a message 730 on the conference screen 700. The message 730 indicates that user B has been designated as a priority conversation target of user A. Note that it is also possible to indicate that user A has input a priority conversation instruction by changing the display mode of the icon 710 corresponding to user A (in the example of FIG. 7B, the border thickness of the icon 710).

[0048] If a priority conversation instruction is input in S601, then in S603 the priority conversation management unit 212 calculates a priority weight by adding a first value to the weight of the first user as seen from the second user. Figure 8 is a diagram showing the data configuration of the relationship table 213 at the time of a priority conversation in the first embodiment. In this example, as shown in Figure 8, in S601 the priority conversation management unit 212 refers to the relationship table 213 to identify records 2131 corresponding to the identifiers of users A and B, and creates a priority conversation table 800 based on these records 2131.

[0049] Priority conversation table 800 includes two records 801, 802 based on identified record 2131. In record 801, user A is set as the source user of the voice data, and user B is set as the destination user of the voice data. In record 802, user B is set as the source user of the voice data, and user A is set as the destination user of the voice data. The initial values ​​of weights and sound image localization of records 801, 802 are the same as those of record 2131.

[0050] In this way, record 2131 determines the two-way weighting between the first user and the second user, while records 801 and 802 determine the one-way weighting between the first user and the second user. The priority conversation management unit 212 calculates a priority weight of "240" by adding a first value, for example "100", to the weighting of user A from the perspective of user B in record 801. On the other hand, the priority conversation management unit 212 does not update the weighting of user B from the perspective of user A in record 802.

[0051] Next, in S605, the priority conversation management unit 212 transmits the identifier of the first user and the identifier of the second user to the multipoint control unit 130. In S607, the priority conversation management unit 212 determines whether there is an instruction to end the priority conversation. For example, the priority conversation management unit 212 determines that there is an instruction to end the priority conversation when the icon 710 is deselected on the conference screen 700 or when the remote conference itself is ended.

[0052] In this case, in S609, the priority conversation management unit 212 restores the weight of the first user as seen from the second user to the original weight. In this example, the priority conversation management unit 212 restores the weight of the first user as seen from the second user to the original weight from the priority weight by deleting the priority conversation table 800. Next, in S611, the priority conversation management unit 212 transmits the identifier of the first user and the identifier of the second user whose corresponding weights have been changed to the multipoint connection device 130, and ends the priority conversation processing. Note that if no priority conversation instruction is input in S601, the priority conversation management unit 212 returns the processing to S601. If no instruction to end the priority conversation is input in S607, the priority conversation management unit 212 returns the processing to S607.

[0053] [System behavior during priority conversation] Fig. 9 is a diagram showing the process and data flow during a priority conversation in the remote conference system 100. Only the differences from the system operation during a normal conversation shown in Fig. 5 will be described below. In the example of Fig. 9, similarly to Fig. 5, after S501 to S513 are executed, user A, who is the user of terminal device A, selects user B, who is the user of terminal device B, as a priority target on the conference screen 700.

[0054] In this case, in S901, the terminal device A transmits a priority conversation instruction, an identifier of the first user (user A), and an identifier of the second user (user B) to the server 110. The server 110 executes the above-mentioned S601 to S605 (see FIG. 6). As a result, in S903, the priority conversation management unit 212 manages the priority conversations and creates a priority conversation table 800 for the first and second users. In S905, the priority conversation management unit 212 transmits the user identifiers of the first and second users corresponding to this priority conversation table 800 (i.e., the identifiers of each of users A and B) to the multipoint control device 130.

[0055] On the other hand, in S509, the voice transmitting unit 222 of the terminal device A transmits the voice data and the identifier of the source user to the multipoint control unit 130. In S511, the voice transmitting unit 222 of the terminal device B transmits the voice data and the identifier of the source user to the multipoint control unit 130.

[0056] When a priority conversation instruction is input, the sound image localization management unit 232 of the multipoint control device 130 determines the sound image localization of the audio data to be reproduced on the terminal device 120 of the second user based on the priority weight. In this example, in S515, the sound image localization management unit 232 of the multipoint control device 130 determines the sound image localization of the first user relative to the second user based on the weight in the priority conversation table 800 corresponding to the received user identifiers of the first user and the second user. The sound image localization management unit 232 performs sound image localization processing of the audio data of the terminal device A received in S509 based on the determined sound image localization.

[0057] For example, in the relationship between user A and user B, the source user (first user) of the voice data of terminal device A received in S509 is user A, and the destination user (second user) is user B. Therefore, in S515, the sound image localization management unit 232 refers to record 801 in the priority conversation table 800 shown in Fig. 8 to obtain a priority weight of "240", and determines "P1" as the sound image localization corresponding to this weight. The sound image localization management unit 232 outputs the determined sound image localization to the sound image localization processing unit 233, and updates the sound image localization of record 801 to the determined sound image localization.

[0058] 10A is a diagram showing a sound image localization state seen from user B during priority conversation in the first embodiment. As described above, when the sound image localization of user A relative to user B is updated from "P5" to "P1", the sound image localization of user A changes to the front vicinity of user B. Therefore, in priority conversation in which user A designates user B, the volume of user A's sound reproduced on terminal device B becomes louder than in normal conversation.

[0059] On the other hand, in the relationship between user A and user B, the source user (first user) of the voice data of terminal device B received in S511 is user B, and the destination user (second user) is user A. Therefore, in S515, sound image localization management unit 232 refers to record 802 in priority conversation table 800 shown in Fig. 8 to obtain a weight of "140", and determines "P5", the same as in normal conversation, as the sound image localization corresponding to this weight. In other words, in a priority conversation in which user A designates user B, the volume of user B reproduced on terminal device A is the same as in normal conversation.

[0060] Also, in S515, the sound image localization management unit 232 determines the sound image localization of the first user with respect to users other than the second user by referring to the relationship table 213 in the same manner as in normal conversation. Therefore, in a priority conversation in which user A designates user B, the volume of user A's sound reproduced on the terminal device 120 of users other than the second user (e.g., users C and D) is the same as that in normal conversation.

[0061] In S517, the sound image localization processing unit 233 transmits the audio data of terminal device A that has been subjected to sound image localization processing to terminal device B. In S519, the sound image localization processing unit 233 transmits the audio data of terminal device B that has been subjected to sound image localization processing to terminal device A. In S521, the audio reproduction unit 223 of terminal device A reproduces the audio data of terminal device B in surround sound. In S523, the audio reproduction unit 223 of terminal device B reproduces the audio data of terminal device A in surround sound.

[0062] 10B is a diagram showing a change in the sound image localization position during priority conversation in the first embodiment. As described above, in priority conversation in which user A designates user B, the sound image localization of user A changes to the front vicinity of user B, and the volume of user A reproduced on terminal device B becomes louder. Therefore, user A can give user B priority in listening to his / her voice, so that user A can, for example, convey an important message that he / she wants to convey to user B reliably as if he / she were speaking in front of user B.

[0063] [Second embodiment] A second embodiment of the present disclosure will be described. In the following, the same reference numerals will be used to omit the description of the same or corresponding configurations or processes as those in the first embodiment, and differences from the first embodiment will be described. In the priority conversation of the second embodiment, in addition to allowing the second user to preferentially listen to the voice of the first user, it is also possible for the first user to not allow other participating users (third users) to listen to the voice of the first user.

[0064] In the second embodiment, the multiple users participating in the remote conference include at least a first user, a second user, and a third user. When a priority conversation instruction is input, the priority conversation management unit 212 calculates a non-priority weight by subtracting a second value from the weight corresponding to the pair of the first user and the third user. When a priority conversation instruction is input, the sound image localization management unit 232 determines the sound image localization of the audio data to be reproduced on the terminal device 120 of the third user based on the non-priority weight. The second embodiment will be described in more detail below.

[0065] [Priority conversation processing] 11A and 11B are flowcharts of a priority conversation process according to the second embodiment. In this example, among a plurality of users participating in a remote conference, a first user is User A, a second user is User B, and a third user is User C.

[0066] As shown in Fig. 11A, the priority conversation management unit 212 of the server 110 executes the processes of S601 to S605 in the same manner as in the first embodiment (see Fig. 6). After executing S605, in S1101 the priority conversation management unit 212 determines whether a third user is in the remote conference. If a third user is in the remote conference, in S1103 the priority conversation management unit 212 calculates a non-priority weight by subtracting a second value from the weight of the first user as seen from the third user.

[0067] 12 is a diagram showing the data configuration of the relationship table 213 at the time of a priority conversation in the second embodiment. In this example, as shown in FIG. 12, in S1103, the priority conversation management unit 212 refers to the relationship table 213 to identify records 2132 corresponding to the identifiers of user A and user C, and creates a non-priority conversation table 1200 based on these records 2132.

[0068] Non-prioritized conversation table 1200 includes two records 1201, 1202 based on identified record 2132. In record 1201, user A is set as the source user of the voice data, and user C is set as the destination user of the voice data. In record 1202, user C is set as the source user of the voice data, and user A is set as the destination user of the voice data. The initial values ​​of weights and sound image localization of records 1201, 1202 are the same as those of record 2131.

[0069] In this way, record 2132 determines the two-way weighting between the first user and the second user, while records 1201 and 1202 determine the one-way weighting between the first user and the third user. The priority conversation management unit 212 calculates a non-priority weight of "0" by subtracting the second value, for example "100", from the weighting of user A as seen from user C in record 1201. On the other hand, the priority conversation management unit 212 does not update the weighting of user C as seen from user A in record 1202.

[0070] Next, in S1105, priority conversation management section 212 transmits the identifier of the first user and the identifier of the third user to multipoint control device 130. If there are multiple third users in the remote conference, creation of non-priority conversation table 1200 and transmission of the user identifier are executed in S1103 to S1105 for each combination of the first user and each third user.

[0071] As shown in FIG. 11B, after execution of S1105, or if there is no third user in the remote conference in S1101, the priority conversation management unit 212 executes the processes of S607 to S611 in the same manner as in the first embodiment (see FIG. 6). After execution of S611, in S1107, the priority conversation management unit 212 determines whether or not a third user is in the remote conference. If there is a third user in the remote conference, in S1109, the priority conversation management unit 212 restores the weight of the first user as seen from the third user to its original weight. In this example, the priority conversation management unit 212 restores the weight of the first user as seen from the third user to its original weight from the non-priority weight by deleting the non-priority conversation table 1200. Note that if there are multiple third users in the remote conference, all of the non-priority conversation tables 1200 are deleted.

[0072] Next, in S1111, the priority conversation management unit 212 transmits the identifier of the first user and the identifier of the third user whose corresponding weights have been changed to the multipoint control unit 130. After execution of S1111, or if there is no third user in the remote conference in S1107, the priority conversation process is terminated.

[0073] [System behavior during priority conversation] In the system operation during priority conversation in the second embodiment, the same processing as in the first embodiment (see Figure 9) is executed, so that the sound image positioning of user A changes to the front vicinity of user B, and the volume of user A's sound played on terminal device B becomes louder.

[0074] Furthermore, when a priority conversation instruction is input at terminal device A, server 110 executes not only S601 to S605 described above, but also S1101 to S1105 (see FIG. 11A). As a result, in S903, priority conversation management unit 212 creates a non-priority conversation table 1200 for the first and third users when managing priority conversations. In S905, priority conversation management unit 212 transmits the user identifiers of the first and third users corresponding to this non-priority conversation table 1200 (i.e., the identifiers of each of users A and C) to multipoint control unit 130.

[0075] In S515, the sound image localization management unit 232 determines the sound image localization of the first user relative to the third user based on the weights in the non-priority conversation table 1200 corresponding to the received user identifiers of the first user and the third user. The sound image localization management unit 232 performs sound image localization processing of the voice data of the terminal device A received in S509 based on the determined sound image localization.

[0076] For example, in the relationship between user A and user C, the source user (first user) of the voice data of terminal device A received in S509 is user A, and the destination user (third user) is user C. Therefore, in S515, sound image localization management unit 232 refers to record 1201 from non-priority conversation table 1200 shown in Fig. 12 to obtain a non-priority weight of "0", and determines "P0", which is an infinitely distant position, as the sound image localization corresponding to this weight. Sound image localization management unit 232 outputs the determined sound image localization to sound image localization processing unit 233, and updates the sound image localization of record 1201 to the determined sound image localization.

[0077] 13A is a diagram showing a sound image localization state seen from user C during priority conversation in the second embodiment. When the sound image localization of user A relative to user C is updated from "P6" to "P0" as described above, the sound image localization of user A becomes infinitely far from user C. Therefore, in a priority conversation in which user A designates user B, the volume of user A's voice reproduced on user C's terminal device 120 becomes substantially zero, and user C cannot recognize the voice of user A.

[0078] 13B is a diagram showing changes in sound image localization position during priority conversation in the second embodiment. As in the case of user C, the weights of all third users participating in the remote conference are updated to non-priority weight "0" in the corresponding records of the non-priority conversation table 1200, and the sound image localization is updated to "P0". Therefore, in a priority conversation in which user A designates user B, user B can hear user A's voice at a loud volume, while other users C and D cannot hear user A's voice. Therefore, for example, user A can whisper to user B a secret conversation that he does not want other users C and D to hear.

[0079] On the other hand, in the relationship between user A and user C, the source user (first user) of the voice data of terminal device 120 of user C is user C, and the destination user (second user) is user A. Therefore, in S515, sound image localization management unit 232 refers to record 1202 in non-priority conversation table 1200 shown in Fig. 12 to obtain a weight of "100", and determines "P6" as the sound image localization corresponding to this weight, which is the same as that in normal conversation. In other words, in a priority conversation in which user A designates user B, the volume of the third user (e.g., users C and D) played back on terminal device A is the same as that in normal conversation.

[0080] [remarks] The present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. The technical scope of the present invention also includes embodiments obtained by appropriately combining the technical means disclosed in the different embodiments. Furthermore, new technical features can be formed by combining the technical means disclosed in the respective embodiments.

[0081] In the above embodiment, a video conference (television conference) is exemplified as a remote conference executed by the remote conference system 100, but the present invention is not limited thereto. For example, the remote conference system 100 may execute a remote conference in which each user interacts in a virtual space using an avatar. A virtual space screen in which the avatars interact is displayed on the terminal device 120, and a first user may select an avatar of a second user as a priority conversation target on the virtual space screen.

[0082] In the above embodiment, in a priority conversation in which a first user designates a second user, the sound image localization of the first user is changed, while the sound image localization of the second user is not changed. Alternatively, in a priority conversation in which a first user designates a second user, both the sound image localization of the first user and the sound image localization of the second user may be changed so that each user can listen to the other's voice with priority. In this priority conversation, the second user can listen to the voice of the first user with priority, and the first user can also listen to the voice of the second user with priority.

[0083] In this modified example, there is no need to distinguish between the source user and the destination user of the voice data, as in the priority conversation table 800 (see FIG. 8) of the above embodiment. For example, in S603, the priority conversation management unit 212 of the server 110 calculates priority weights by adding a first value to the weights in the relationship table 213 corresponding to the first user and the second user, instead of creating the priority conversation table 800. In S515, the sound image localization management unit 232 of the multipoint control unit 130 determines sound image localization based on the priority weights, and performs sound image localization processing on both the voice data of the first user and the second user.

[0084] Similarly, in a priority conversation in which a first user designates a second user, the sound image localization of the first user and the sound image localization of the third user may both be changed so that they cannot hear each other's voice. This makes it possible to prevent the third user from hearing the voice of the first user and the first user from hearing the voice of the third user in a priority conversation in which a first user designates a second user.

[0085] In this modified example, there is no need to distinguish between the source user and destination user of the voice data, as in the non-priority conversation table 1200 (see FIG. 12) in the above embodiment. For example, in S603, the priority conversation management unit 212 calculates a non-priority weight by subtracting a second value from the weight in the relationship table 213 corresponding to the first user and the third user, instead of creating the non-priority conversation table 1200. In S515, the sound image localization management unit 232 determines the sound image localization based on this non-priority weight, and performs sound image localization processing on the voice data of both the first user and the third user.

[0086] The first value and the priority weight may be any value as long as they can change the sound image localization to one that is easier for the second user to hear. The second value and the non-priority weight may be any value as long as they can change the sound image localization to one that is harder for the second user to hear.

[0087] The functional blocks of the information management unit 211, the priority conversation management unit 212, and the table management unit 214 are not limited to being provided in the server 110, and some or all of these functional blocks may be provided in another electronic device connected to the server 110. The functional blocks of the audio receiving unit 231, the sound image localization management unit 232, and the sound image localization processing unit 233 are not limited to being provided in the multipoint control unit 130, and some or all of these functional blocks may be provided in another electronic device connected to the multipoint control unit 130. The server 110 and the multipoint control unit 130 may be included in one electronic device. [Explanation of symbols]

[0088] 100 Remote conference system, 112 Storage unit, 120 Terminal device, 212 Priority conversation management unit, 213 Relation table, 214 Table management unit, 231 Audio receiving unit, 232 Sound image localization management unit

Claims

1. A remote conference system for conducting a remote conference among a plurality of terminal devices, A storage unit that stores a table in which a weight is associated with each pair of any two users among a plurality of users; a table management unit that updates the weights corresponding to the pair in the table based on information transmitted and received between the terminal devices of the two users forming the pair; a voice receiving unit for receiving voice data from the terminal device of a first user; a sound image localization management unit that determines a sound image localization of the audio data based on a weight corresponding to the pair in the table; a sound image localization processing unit that performs sound image localization processing on the received voice data based on the determined sound image localization and transmits the sound data to the terminal device of another user that is paired with the first user; a priority conversation management unit that calculates a priority weight by adding a first value to the weight corresponding to the pair of the first user and the second user when an instruction to designate the second user as a priority target is input from the first user, the sound image localization management unit, when the instruction is input, determines a sound image localization of the audio data to be reproduced on the terminal device of the second user based on the priority weight; Remote conferencing system.

2. each of the plurality of terminal devices displays a conference screen displaying a plurality of users participating in the remote conference; the priority conversation management unit determines that the instruction has been input when the first user performs an operation to select the second user as the priority target from among the plurality of users on the conference screen of the terminal device; The remote conference system according to claim 1 .

3. When the instruction is input, the priority conversation management unit displays, on the conference screen of the terminal device used by the second user, an image or a message indicating that the first user has designated the second user as the priority target. The remote conference system according to claim 2 .

4. the plurality of users participating in the remote conference includes at least the first user, the second user, and a third user; when the instruction is input, the priority conversation management unit calculates a non-priority weight by subtracting a second value from the weight corresponding to the pair of the first user and the third user; the sound image localization management unit, when the instruction is input, determines a sound image localization of the audio data to be reproduced on the terminal device of the third user based on the non-priority weight. The remote conference system according to claim 1 .

5. A remote conference method for conducting a remote conference among a plurality of terminal devices, comprising: In a table in which a weight is associated with each pair of any two users among a plurality of users, the weight corresponding to the pair is updated based on information transmitted and received between the terminal devices of the two users forming the pair; receiving voice data from the terminal device of a first user; determining a sound image localization of the audio data based on weights corresponding to the pairs in the table; performing a sound image localization process on the received voice data based on the determined sound image localization, and transmitting the sound data to the terminal device of the other user who is paired with the first user; when an instruction to designate a second user as a priority target is input from the first user, adding a first value to the weight corresponding to the pair of the first user and the second user to calculate a priority weight; determining the sound image localization includes, when the instruction is input, determining a sound image localization of the audio data to be reproduced on the terminal device of the second user based on the priority weight; Remote conferencing methods.

Images