Online call management apparatus and storage medium storing online call management program

Abstract

Provided are an online call management apparatus and a storage medium storing an online call management program. The online call management apparatus of an embodiment has a first acquisition section, a second acquisition section, and a control section. The first acquisition section acquires, from at least one terminal that plays a sound image via a playback device, playback environment information that is information related to a playback environment of a sound of the playback device. The second acquisition section acquires orientation information that is information of a direction of the sound image for a user of the terminal. The control section performs control of playback of the sound image for each terminal in accordance with the playback environment information and the orientation information.

Description

Technical Field

[0001] The implementation involves an online call management device and a storage medium storing an online call management program. Background Technology

[0002] Sound image localization technology is known to locate sound images in the space surrounding a user's head using various playback devices with different sound playback environments, such as dual-channel speakers positioned in front of the user, headphones worn in the user's ears, and headphones worn on the user's head. This technology can create the illusion that the user hears sound from a direction different from the actual direction of the playback device.

[0003] In recent years, there have been attempts to apply sound image localization technology to online calls. For example, in online meetings, the voices of multiple speakers can sometimes be concentrated and difficult to hear clearly. In contrast, by localizing the sound images of each speaker in different directions in the space surrounding the user's head, the user can hear each speaker clearly.

[0004] In order to locate the sound image in the space surrounding each user's head, the information about the sound playback environment of each user's playback device needs to be known. Given that the sound playback environment of each user's playback device is different, it is possible that the sound image for one user may be properly located, but not for other users. Summary of the Invention

[0005] The implementation provides an online call management device and a storage medium storing an online call management program, which can play audio images appropriately positioned for each user even when the audio playback environment of each user's audio playback device is different in the case of online calls.

[0006] The online call management device of this embodiment includes a first acquisition unit, a second acquisition unit, and a control unit. The first acquisition unit acquires, via a network, information related to the playback environment of the sound from at least one terminal playing audio / video via a playback device, i.e., playback environment information. The second acquisition unit acquires information about the positioning direction of the audio / video for the user at the terminal, i.e., orientation information. The control unit controls the playback of audio / video for each terminal based on the playback environment information and the orientation information. Attached Figure Description

[0007] Figure 1 This is a diagram showing the structure of an example of an online call system equipped with the online call management device of the first embodiment.

[0008] Figure 2 This is a diagram showing the structure of an example terminal.

[0009] Figure 3 This is a flowchart illustrating an example of the actions taken by a host terminal during an online call.

[0010] Figure 4 This is a flowchart illustrating an example of the actions taken during an online call by a guest's terminal.

[0011] Figure 5 This is an example of an input screen showing playback environment information and orientation information.

[0012] Figure 6 This is an example of an input screen showing playback environment information.

[0013] Figure 7A This is a diagram illustrating the state of hearing the voices of multiple users in a concentrated manner.

[0014] Figure 7B This is a schematic diagram illustrating the state of accurate acoustic localization.

[0015] Figure 8 This is a diagram showing the structure of an example of an online call system equipped with the online call management device of the second embodiment.

[0016] Figure 9 This is a diagram showing the structure of an example server.

[0017] Figure 10 This is a flowchart illustrating the actions of the first example during an online call on the server.

[0018] Figure 11 This is a flowchart illustrating the actions of the second example during an online call on the server.

[0019] Figure 12 This is another example of an input screen showing location information.

[0020] Figure 13 This is another example of an input screen showing location information.

[0021] Figure 14A This is another example of an input screen showing location information.

[0022] Figure 14B This is another example of an input screen showing location information.

[0023] Figure 15 This is another example of an input screen showing location information.

[0024] Figure 16 This is another example of an input screen showing location information.

[0025] Figure 17 This is another example of an input screen showing location information.

[0026] Figure 18 This is an example of a display screen shown on each terminal during an online presentation, in a variation of the second embodiment.

[0027] Figure 19 This is an example of a screen displayed on the terminal when the speaker assistance button is selected.

[0028] Figure 20 This is an example of a screen displayed on the terminal when the "Discussion among listeners" button is selected.

[0029] Figure 21 This is a diagram illustrating the structure of an example of the server in the third embodiment.

[0030] Figure 22A This is an example of a screen used to input information related to echo data.

[0031] Figure 22B This is an example of a screen used to input information related to echo data.

[0032] Figure 22C This is an example of a screen used to input information related to echo data.

[0033] Figure 22D This is an example of a screen used to input information related to echo data.

[0034] (Symbol Explanation)

[0035] 1: Processor; 2: Memory; 3: Storage device; 4: Sound playback device; 5: Sound detection device; 6: Display device; 7: Input device; 8: Communication device; 11: First acquisition unit; 12: Second acquisition unit; 13: Control unit; 14: Third acquisition unit; 31: Online call management program; 101: Processor; 102: Memory; 103: Storage device; 104: Communication device; 1031: Online call management program; 1032: Echo table; 1033: Level attenuation table Detailed Implementation

[0036] Hereinafter, the embodiments will be described with reference to the accompanying drawings.

[0037] [First Implementation]

[0038] Figure 1 This is a diagram illustrating the structure of an example of an online call system equipped with the online call management device of the first embodiment. Figure 1In the online calling system shown, multiple terminals, Figure 1 Four terminals HT, GT1, GT2, and GT3 are connected to each other via a network NW. Users HU, GU1, GU2, and GU3 of each terminal conduct calls through terminals HT, GT1, GT2, and GT3. In the first embodiment, terminal HT is the host terminal operated by user HU, who is in charge of the online call. Terminals GT1, GT2, and GT3 are the guest terminals operated by users GU1, GU2, and GU3, who are participating in the online call as guests. Terminal HT centrally controls the positioning of audio and video in the space around the heads of each user HU, GU1, GU2, and GU3 during calls using terminals HT, GT1, GT2, and GT3. Here, in Figure 1 The number of terminals is four, but not limited to this. Two or more terminals are acceptable. With two terminals, these two terminals can be used for online calls. Alternatively, with two terminals, one terminal may not play sound, but can be used for controlling the space around the user's head on the other terminal to position the audio image.

[0039] Figure 2 It is shown Figure 1 The diagram shows the structure of an example terminal. Hereinafter, terminals HT, GT1, GT2, and GT3 will be described as having substantially the same elements. Figure 2 As shown, the terminal includes a processor 1, a memory 2, a storage device 3, a sound playback device 4, a sound detection device 5, a display device 6, an input device 7, and a communication device 8. Regarding the terminal, for example, consider various terminals capable of communication, such as personal computers (PCs), tablet computers, and smartphones. Furthermore, each terminal may not necessarily need to have the same... Figure 2 The elements shown are the same. Each terminal may or may not have these elements. Figure 2 The elements shown can also have Figure 2 Elements other than those shown.

[0040] Processor 1 is a processor that controls the overall operation of the terminal. For example, the processor 1 of the host terminal HT operates as the first acquisition unit 11, the second acquisition unit 12, and the control unit 13 by executing programs stored in the storage device 3. In the first embodiment, the processor 1 of the visitor terminals GT1, GT2, and GT3 may not necessarily be able to operate as the first acquisition unit 11, the second acquisition unit 12, and the control unit 13. Processor 1 is, for example, a CPU. Processor 1 can also be an MPU, GPU, ASIC, FPGA, etc. Processor 1 can be a single CPU or multiple CPUs.

[0041] The first acquisition unit 11 acquires playback environment information input from each of the terminals HT, GT1, GT2, and GT3 participating in the online call. The playback environment information is information related to the playback environment of the sound playback device 4 used in each of the terminals HT, GT1, GT2, and GT3. The information related to the sound playback environment includes information indicating what device is used as the sound playback device 4. This information indicates, for example, whether a stereo speaker, headphones, or earphones are used as the sound playback device 4. Furthermore, when a stereo speaker is used as the sound playback device 4, the information related to the sound playback environment also includes, for example, information indicating the spacing between the left and right speakers.

[0042] The second acquisition unit 12 acquires the location information input by the terminal HT participating in the online call. The location information is information on the positioning direction of the user's voice image for each terminal, including the user HU of the terminal HT.

[0043] The control unit 13 controls the playback of audio images in each terminal, including the terminal HT, based on playback environment information and orientation information. For example, the control unit 13 generates audio image filter coefficients suitable for each terminal based on the playback environment information and orientation information, and sends the generated audio image filter coefficients to each terminal. The audio image filter coefficients are coefficients convolved with the left and right audio signals input to the audio playback device 4, and are generated, for example, based on the head transfer function C, which is the sound transmission characteristic between the audio playback device 4 and the user's head (binaural), and the head transfer function d, which is the sound transmission characteristic between an imaginary sound source determined according to the orientation information and the user's head (binaural). For example, the storage device 3 stores a table of head transfer functions C for each playback environment information and a table of head transfer functions d for each orientation information. The control unit 13 acquires the head transfer function C and the head transfer function d based on the playback environment information of each terminal acquired by the first acquisition unit 11 and the orientation information of each terminal acquired by the second acquisition unit 12, and generates audio image filter coefficients for each terminal.

[0044] Memory 2 includes ROM and RAM. ROM is non-volatile memory. ROM stores terminal boot programs, etc. RAM is volatile memory. RAM is used, for example, as working memory during processing in processor 1.

[0045] Storage device 3 is, for example, a hard disk drive or a solid-state drive. Storage device 3 stores various programs executed by processor 1, such as online call management program 31. Online call management program 31 is, for example, an application downloaded from a predetermined download server, and is a program used to perform various processes related to online calls in the online call system. Here, the storage device 3 of the visitor's terminals GT1, GT2, and GT3 may not store the online call management program 31.

[0046] The sound playback device 4 is a device for playing sound. In this embodiment, the sound playback device 4 is a device capable of playing stereo sound, such as a stereo speaker, headphones, or earphones. By playing the sound signal (i.e., the sound image signal) convolved with the aforementioned sound image filter coefficients by the sound playback device 4, the sound image is positioned in the space surrounding the user's head. In this embodiment, the sound playback devices 4 of each terminal can be the same or different. Furthermore, the sound playback device 4 can be a device built into the terminal or an external device capable of communicating with the terminal.

[0047] The sound detection device 5 detects the voice input of the user operating the terminal. The sound detection device 5 is, for example, a microphone. The microphone of the sound detection device 5 can be a stereo microphone or a mono microphone. Furthermore, the sound detection device 5 can be a device built into the terminal or an external device capable of communicating with the terminal.

[0048] Display device 6 is a display device such as a liquid crystal display (LCD) or an organic EL display. Display device 6 displays various screens, including input screens and explanation screens. Furthermore, display device 6 can be a display device built into the terminal or an external display device capable of communicating with the terminal.

[0049] Input device 7 is an input device such as a touch panel, keyboard, or mouse. When input device 7 is operated, a signal corresponding to the operation is input to processor 1. Processor 1 performs various processes based on this signal.

[0050] Communication device 8 is a communication device used for terminals to communicate with each other via network NW. Communication device 8 can be used for wired communication or wireless communication.

[0051] Next, the operation of the online call system in the first embodiment will be explained. Figure 3 This is a flowchart illustrating an example of the actions of a host terminal HT during an online call. Figure 4 This is a flowchart illustrating an example of the actions taken during an online call by the visitor's terminals GT1, GT2, and GT3. Figure 3The action is executed by processor 1 of the host terminal HT. Additionally, Figure 4 The actions are executed by processor 1 of the visitor's terminals GT1, GT2, and GT3.

[0052] First, the operation of the terminal HT will be explained. In step S1, the processor 1 of the terminal HT displays the input screen containing playback environment information and location information on the display device 6. The data used to display the input screen containing playback environment information and location information can be pre-stored in the storage device 3 of the terminal HT. Figure 5 This is a diagram showing an example of an input screen displaying playback environment information and orientation information on the display device 6 of the terminal HT.

[0053] like Figure 5 As shown, the input screen for playback environment information includes a list 2601 of devices intended to be used as the sound playback device 4. The user HU of the terminal HT selects the sound playback device 4 to be used from the list 2601.

[0054] In addition, such as Figure 5 As shown, the location information input screen includes input fields 2602 that display the location of each user, including the user HU themselves. Figure 5 In this example, "A" represents user HU, "B" represents user GU1, "C" represents user GU2, and "D" represents user GU3. Furthermore, the orientation is defined by setting a predetermined reference direction, such as the direction facing each user, to 0 degrees. In the first embodiment, user HU of the host computer also inputs the orientation information of other users GU1, GU2, and GU3. Here, user HU can specify the orientation information of each user within a range of 0 to 359 degrees. However, when the orientation information is repeated, the sound images of multiple users are positioned in the same direction. Therefore, in the case where the same orientation is input for multiple users, the processor 1 may also display error messages, etc., on the display device 6.

[0055] Here, in Figure 5 In this system, the input screen for playback environment information and the input screen for orientation information consist of a single screen. However, the input screens for playback environment information and orientation information can also be composed of different screens. In this case, for example, the input screen for playback environment information is displayed initially, and the input screen for orientation information is displayed after the playback environment information input is completed.

[0056] In step S2, processor 1 determines whether there is input of playback environment information and location information by user HU, or whether playback environment information has been received from other terminals GT1, GT2, and GT3. If, in step S2, it is determined that there is input of playback environment information and location information by user HU, or that playback environment information has been received from other terminals GT1, GT2, and GT3, the process proceeds to step S3. If, in step S2, it is determined that there is no input of playback environment information and location information by user HU, or that no playback environment information has been received from other terminals GT1, GT2, and GT3, the process proceeds to step S4.

[0057] In step S3, processor 1 stores the input or received information in memory 2, for example, RAM.

[0058] In step S4, processor 1 determines whether the input of information is complete, i.e., whether the playback environment information and location information of each terminal have been stored in RAM. In step S4, if it is determined that the input of information is incomplete, the process returns to step S2. In step S4, if it is determined that the input of information is complete, the process proceeds to step S5.

[0059] In step S5, processor 1 generates audio-visual filter coefficients for each terminal, i.e., for the user of each terminal, based on the playback environment information and location information of each terminal.

[0060] For example, the image filter coefficients for user HU include image filter coefficients generated based on the playback environment information of the sound playback device 4 of terminal GT1 input by user GU1 and the orientation information of user HU specified by user HU, image filter coefficients generated based on the playback environment information of the sound playback device 4 of terminal GT2 input by user GU2 and the orientation information of user HU specified by user HU, and image filter coefficients generated based on the playback environment information of the sound playback device 4 of terminal GT3 input by user GU3 and the orientation information of user HU specified by user HU.

[0061] In addition, the audio-visual filter coefficients for user GU1 include audio-visual filter coefficients generated based on the playback environment information of the sound playback device 4 of terminal HT input by user HU and the orientation information of user GU1 specified by user HU, audio-visual filter coefficients generated based on the playback environment information of the sound playback device 4 of terminal GT2 input by user GU2 and the orientation information of user GU1 specified by user HU, and audio-visual filter coefficients generated based on the playback environment information of the sound playback device 4 of terminal GT3 input by user GU3 and the orientation information of user GU1 specified by user HU.

[0062] The image filter coefficients for user GU2 and user GU3 can also be generated in the same way. That is, the image filter coefficients for user GU2 are generated based on the playback environment information of other terminals besides the playback environment information of the sound playback device 4 of terminal GT2 input by user GU2, and the location information of user GU2 specified by user HU. Similarly, the image filter coefficients for user GU3 are generated based on the playback environment information of other terminals besides the playback environment information of the sound playback device 4 of terminal GT3 input by user GU3, and the location information of user GU3 specified by user HU.

[0063] In step S6, processor 1 stores the audio-visual filter coefficients generated for user HU in storage device 3, for example. Additionally, processor 1 uses communication device 8 to send the audio-visual filter coefficients generated for users GU1, GU2, and GU3 to their respective terminals. Thus, the initial setup for online calling is completed.

[0064] In step S7, processor 1 determines whether there is user HU's voice input via sound detection device 5. If user HU's voice input is detected in step S7, the process proceeds to step S8. If user HU's voice input is not detected in step S7, the process proceeds to step S10.

[0065] In step S8, the processor 1 convolves the audio-visual filter coefficients facing the user HU with the audio signal based on the sound of the user HU input via the sound detection device 5 to generate an audio-visual signal facing other users.

[0066] In step S9, processor 1 uses communication device 8 to send audio-visual signals to terminals GT1, GT2, and GT3, which are intended for other users. Afterward, the process proceeds to step S13.

[0067] In step S10, processor 1 determines whether it has received audio / video signals from other terminals via communication device 8. If it is determined in step S10 that audio / video signals from other terminals have been received, the process proceeds to step S11. If it is determined in step S10 that no audio / video signals from other terminals have been received, the process proceeds to step S13.

[0068] In step S11, the processor 1 separates the audio-visual signal facing the user HU from the received audio-visual signal. For example, when an audio-visual signal is received from the terminal GT1, the processor 1 separates the audio-visual signal after convolving the audio-visual filter coefficients generated based on the playback environment information of the sound playback device 4 of the terminal HT input by the user HU and the orientation information of the user GU1 specified by the user HU.

[0069] In step S12, processor 1 plays the audio-visual signal using audio playback device 4. Then, the process proceeds to step S13.

[0070] In step S13, processor 1 determines whether to end the online call. For example, if the online call is terminated by an instruction from user HU's input device 7, it is determined that the online call should be terminated. In step S13, if it is determined that the online call should not be terminated, the process returns to step S2. In this case, if there is a change in the playback environment information or location information during the online call, processor 1 reflects the change and regenerates the audio-visual filter coefficients to continue the online call. In step S13, if it is determined that the online call should be terminated, processor 1... Figure 3 The processing is now complete.

[0071] Next, the actions of terminals GT1, GT2, and GT3 will be explained. Since terminals GT1, GT2, and GT3 operate identically, the actions of terminal GT1 will be explained representatively below.

[0072] In step S101, the processor 1 of the terminal GT1 displays the input screen of playback environment information on the display device 6. The data used to display the input screen of playback environment information can also be pre-stored in the storage device 3 of the terminal GT1. Figure 6 This is a diagram illustrating an example of an input screen displaying playback environment information on display devices 6 of terminals GT1, GT2, and GT3. (See diagram for example.) Figure 6 As shown, the playback environment information input screen includes a list 2601 of devices intended to be used as sound playback device 4. That is, the playback environment information input screen of terminal HT can be the same as that of terminals GT1, GT2, and GT3. Here, the data from the playback environment information input screen of terminal GT1 can also be stored in the storage device 3 of terminal HT. In this case, in Figure 3 In step S1, the processor 1 of terminal HT sends the input screen data of the playback environment information of terminals GT1, GT2, and GT3 to terminals GT1, GT2, and GT3. In this case, the input screen data used to display the playback environment information may not be pre-stored in the storage device 3 of terminals GT1, GT2, and GT3.

[0073] In step S102, processor 1 determines whether there is input of playback environment information by user GU1. If it is determined in step S102 that there is input of playback environment information by user GU1, the process proceeds to step S103. If it is determined in step S102 that there is no input of playback environment information by user GU1, the process proceeds to step S104.

[0074] In step S103, the processor 1 uses the communication device 8 to send the input playback environment information to the terminal HT.

[0075] In step S104, processor 1 determines whether it has received the audio-visual filter coefficients for user GU1 from terminal HT. If it is determined in step S104 that no audio-visual filter coefficients for user GU1 have been received, the process returns to step S102. If it is determined in step S104 that audio-visual filter coefficients for user GU1 have been received, the process proceeds to step S105.

[0076] In step S105, the processor 1 stores the received audio-visual filter coefficients for the user GU1, for example, in the storage device 3.

[0077] In step S106, processor 1 determines whether there is voice input from user GU1 via voice detection device 5. If it is determined that voice input from user GU1 is present in step S106, the process proceeds to step S107. If it is determined that there is no voice input from user GU1 in step S106, the process proceeds to step S109.

[0078] In step S107, the processor 1 convolves the audio-visual filter coefficients facing user GU1 with the audio signal based on the sound of user GU1 input via the sound detection device 5 to generate audio-visual signals facing other users.

[0079] In step S108, processor 1 uses communication device 8 to send audio-visual signals to terminals HT, GT2, and GT3, which are intended for other users. Afterward, the process proceeds to step S112.

[0080] In step S109, processor 1 determines whether it has received audio / video signals from other terminals via communication device 8. If it is determined in step S109 that audio / video signals from other terminals have been received, the process proceeds to step S110. If it is determined in step S109 that no audio / video signals from other terminals have been received, the process proceeds to step S112.

[0081] In step S110, the processor 1 separates the audio-visual signal facing the user GU1 from the received audio-visual signal. For example, when the audio-visual signal is received from the terminal HT, the processor 1 separates the audio-visual signal after convolving the audio-visual filter coefficients generated based on the playback environment information of the sound playback device 4 of the terminal GT1 input by the user GU1 and the orientation information of the user HU specified by the user HU.

[0082] In step S111, processor 1 plays the audio-visual signal using audio playback device 4. Then, the process proceeds to step S112.

[0083] In step S112, processor 1 determines whether to end the online call. For example, if the online call is terminated by an instruction from user GU1's input device 7, it is determined that the online call should be terminated. In step S112, if it is determined that the online call should not be terminated, the process returns to step S102. In this case, if there is a change in playback environment information during the online call, processor 1 sends the playback environment information to terminal HT to continue the online call. In step S112, if it is determined that the online call should be terminated, processor 1... Figure 4 The processing is now complete.

[0084] As explained above, in the first embodiment, based on playback environment information and location information, sound image filter coefficients for each user on each terminal are generated in the host terminal HT. Therefore, it is possible to locate the sound images of other users based on the playback environment of the sound playback device 4 in each terminal. For example, in the case of multiple users speaking simultaneously during an online call between multiple terminals, the sound image filter coefficients would normally be... Figure 7A The diagram shows the concentrated hearing of multiple users' voices VA, VB, VC, and VD. In contrast, in the first embodiment, the voices VA, VB, VC, and VD of multiple users are positioned at different locations around each user's head by specifying the user HU on the host. Therefore, it is possible to... Figure 7B The method shown creates the illusion for the user by making them hear the voices of multiple users (VA, VB, VC, VD) from different directions. Therefore, the user can clearly hear the voices of multiple users (VA, VB, VC, VD).

[0085] The generation of audio-visual filter coefficients requires playback environment information and location information. However, it is not possible to directly determine the playback environment of each visitor's audio playback device from the host terminal. In contrast, in the first embodiment, playback environment information is sent from the visitor's terminal to the host terminal, and based on this information, the host terminal generates audio-visual filter coefficients for each terminal. Thus, the first embodiment is particularly preferred in online call environments where audio-visual filter coefficients are centrally managed using a single terminal.

[0086] In this implementation, the host terminal generates new audio-visual filter coefficients each time it acquires playback environment information and location information. Alternatively, it can be envisioned that multiple audio-visual filter coefficients are shared by the host terminal and the visitor terminal, and the host terminal determines the required audio-visual filter coefficients from its pre-shared coefficients each time it acquires playback environment information and location information. Then, the host terminal may not send the audio-visual filter coefficients to its respective visitor terminal, but only send information indicating the index of the determined audio-visual filter coefficients to each visitor terminal. In this case, it is unnecessary to generate audio-visual filter coefficients sequentially during the online call.

[0087] Furthermore, in the first embodiment, the transmission and reception of information other than audio during online calls is not specifically mentioned. In the first embodiment, the transmission and reception of information other than audio, such as moving images, can also be performed.

[0088] Furthermore, in the first embodiment, the host terminal generates the audio-visual filter coefficients. In contrast, the generation of the audio-visual filter coefficients does not necessarily need to be performed by the host terminal. The generation of the audio-visual filter coefficients can be performed by any visitor terminal, or by a device different from the terminals participating in the online call, such as a server. In this case, the host terminal sends the playback environment information and location information of each terminal participating in the online call, which includes playback environment information obtained from the respective visitor terminals, to the server or similar device.

[0089] [Second Implementation]

[0090] Next, the second implementation method will be described. Figure 8 This is a diagram illustrating the structure of an example of an online call system equipped with the online call management device of the second embodiment. Figure 8 In the online calling system shown, with Figure 1 Similarly, multiple terminals, in Figure 8 Four terminals, HT, GT1, GT2, and GT3, are connected to each other via a network (NW). Users HU, GU1, GU2, and GU3 on each terminal conduct calls through terminals HT, GT1, GT2, and GT3. In the second embodiment, terminal HT is also the terminal operated by user HU, who is hosting the online call. Terminals GT1, GT2, and GT3 are the client terminals operated by users GU1, GU2, and GU3, who are participating in the online call as guests.

[0091] In the second embodiment, the server Sv is further connected to the terminals HT, GT1, GT2, and GT3 in a manner capable of communication via the network NW. In this second embodiment, the server Sv centrally controls the positioning of the audio image in the space surrounding the heads of each user HU, GU1, GU2, and GU3 during calls using terminals HT, GT1, GT2, and GT3. Figure 8 The server Sv in the middle can also be used as a cloud server.

[0092] about Figure 8 The online call system of the second embodiment shown is envisioned for applications such as online meetings or online presentations.

[0093] Figure 9 This is a diagram illustrating the structure of an example server Sv. Furthermore, terminals HT, GT1, GT2, and GT3 can have... Figure 2 The structure is shown. Therefore, the structure of terminals HT, GT1, GT2, and GT3 is omitted from the description. Figure 9 As shown, the server Sv has a processor 101, a memory 102, a storage device 103, and a communication device 104. Furthermore, the server Sv may not necessarily need to have the same... Figure 9 The elements shown are the same as the elements. Server Sv may or may not have these. Figure 9 The elements shown can also have Figure 9 Elements other than those shown.

[0094] Processor 101 is a processor that controls the overall operation of server Sv. For example, server Sv's processor 101 operates as the first acquisition unit 11, the second acquisition unit 12, the third acquisition unit 14, and the control unit 13 by executing programs stored in storage device 103. In the second embodiment, the processor 1 of the host terminal HT and the visitor terminals GT1, GT2, and GT3 may not necessarily be able to operate as the first acquisition unit 11, the second acquisition unit 12, the third acquisition unit 14, and the control unit 13. Processor 101 is, for example, a CPU. Processor 101 may also be an MPU, GPU, ASIC, FPGA, etc. Processor 101 may be a single CPU or multiple CPUs.

[0095] The first acquisition unit 11 and the second acquisition unit 12 are the same as in the first embodiment. Therefore, their descriptions are omitted. Furthermore, the control unit 13, as described in the first embodiment, controls the playback of audio and video in each terminal, including the terminal HT, based on playback environment information and orientation information.

[0096] The third acquisition unit 14 acquires utilization information from each of the terminals HT, GT1, GT2, and GT3 participating in the online call. Utilization information relates to the utilization of audio and video used in each of the terminals HT, GT1, GT2, and GT3. Utilization information includes, for example, information about attributes assigned to users participating in the online call. Additionally, utilization information includes information about group settings for users participating in the online call. Utilization information may include information related to the utilization of various other audio and video.

[0097] Memory 102 includes ROM and RAM. ROM is non-volatile memory. ROM stores the boot program of server Sv, etc. RAM is volatile memory. RAM is used, for example, as working memory during processing in processor 101.

[0098] Storage device 103 is, for example, a hard disk drive or a solid-state drive. Storage device 103 stores various programs executed by processor 101, such as online call management program 1031. Online call management program 1031 is a program used to perform various processes related to online calls in the online call system.

[0099] Communication device 104 is a communication device used by server Sv to communicate with various terminals via network NW. Communication device 104 can be used for wired communication or wireless communication.

[0100] Next, the operation of the online call system in the second embodiment will be explained. Figure 10 This is a flowchart illustrating the actions of the first example during an online call on server Sv. The actions of the host terminal HT and the visitor terminals GT1, GT2, and GT3 basically follow... Figure 4 The actions shown.

[0101] In step S201, the processor 101 sends the input screen data containing playback environment information and location information to each terminal HT, GT1, GT2, and GT3. That is, in the second embodiment, the data is displayed not only on the host terminal HT but also on the visitor terminals GT1, GT2, and GT3. Figure 5 The input screen displays playback environment information and location information. Thus, visiting users GU1, GU2, and GU3 also specify the direction of the sound image. Furthermore, the processor 101 can send the data from the input screen containing the utilization information to each terminal HT, GT1, GT2, and GT3.

[0102] In step S202, the processor 101 determines whether playback environment information and location information have been received from terminals HT, GT1, GT2, and GT3. If, in step S202, it is determined that playback environment information and location information have been received from terminals HT, GT1, GT2, and GT3, the process proceeds to step S203. If, in step S202, it is determined that no playback environment information and location information have been received from terminals HT, GT1, GT2, and GT3, the process proceeds to step S207.

[0103] In step S203, the processor 101 stores the received information in memory 102, for example, RAM.

[0104] In step S204, the processor 101 determines whether the input of information is complete, i.e., whether the playback environment information and location information of each terminal have been stored in RAM. In step S204, if it is determined that the input of information is incomplete, the process returns to step S202. In step S204, if it is determined that the input of information is complete, the process proceeds to step S205.

[0105] In step S205, the processor 101 generates audio-visual filter coefficients for each terminal, i.e., for the user of each terminal, based on the playback environment information and orientation information of each terminal.

[0106] For example, the image filter coefficients for user HU include image filter coefficients generated based on the playback environment information of the sound playback device 4 of terminal GT1 input by user GU1 and the location information of user HU specified by each user HU, GU1, GU2, and GU3; image filter coefficients generated based on the playback environment information of the sound playback device 4 of terminal GT2 input by user GU2 and the location information of user HU specified by each user HU, GU1, GU2, and GU3; and image filter coefficients generated based on the playback environment information of the sound playback device 4 of terminal GT3 input by user GU3 and the location information of user HU specified by each user HU, GU1, GU2, and GU3.

[0107] In addition, the audio-visual filter coefficients for user GU1 include audio-visual filter coefficients generated based on the playback environment information of the sound playback device 4 of terminal HT input by user HU and the location information of user GU1 specified by each user of users HU, GU1, GU2, and GU3; audio-visual filter coefficients generated based on the playback environment information of the sound playback device 4 of terminal GT2 input by user GU2 and the location information of user GU1 specified by each user of users HU, GU1, GU2, and GU3; and audio-visual filter coefficients generated based on the playback environment information of the sound playback device 4 of terminal GT3 input by user GU3 and the location information of user GU1 specified by each user of users HU, GU1, GU2, and GU3.

[0108] Similarly, the image filter coefficients for user GU2 and user GU3 can also be generated. That is, the image filter coefficients for user GU2 are generated based on playback environment information other than the playback environment information of the sound playback device 4 of terminal GT2 input by user GU2, and the location information of user GU2 specified by each of users HU, GU1, GU2, and GU3. Furthermore, the image filter coefficients for user GU3 are generated based on playback environment information other than the playback environment information of the sound playback device 4 of terminal GT3 input by user GU3, and the location information of user GU3 specified by each of users HU, GU1, GU2, and GU3.

[0109] In step S206, the processor 101 uses the communication device 104 to send the audio-visual filter coefficients generated for users HU, GU1, GU2, and GU3 to each terminal. Thus, the initial setup for online calls is completed.

[0110] In step S207, the processor 101 determines whether it has received an audio / video signal from at least one of the terminals HT, GU1, GU2, and GU3 via the communication device 104. If it is determined in step S207 that an audio / video signal has been received from a terminal, the process proceeds to step S208. If it is determined in step S207 that no audio / video signal has been received from any terminal, the process proceeds to step S210.

[0111] In step S208, the processor 101 separates the audio-visual signals for each user from the received audio-visual signals. For example, when receiving an audio-visual signal from terminal HT, the processor 101 separates the audio-visual signal for user GU1 after convolving the audio-visual filter coefficients generated based on the playback environment information of the sound playback device 4 of terminal GT1 input by user GU1 and the orientation information of user HU specified by user GU1. Similarly, the processor 101 separates the audio-visual signal for user GU2 after convolving the audio-visual filter coefficients generated based on the playback environment information of the sound playback device 4 of terminal GT2 input by user GU2 and the orientation information of user HU specified by user GU2. Furthermore, the processor 101 separates the audio-visual signal for user GU3 after convolving the audio-visual filter coefficients generated based on the playback environment information of the sound playback device 4 of terminal GT3 input by user GU3 and the orientation information of user HU specified by user GU3.

[0112] In step S209, the processor 101 uses the communication device 104 to send the respective separated audio-visual signals to the corresponding terminals. Afterwards, the processing proceeds to step S210. Furthermore, in each terminal, [the signal is transmitted / transmitted / adapted]... Figure 4 Similarly, in step S12, the received audio-visual signals are played. In server Sv, the audio-visual signals are separated, so step S11 is unnecessary. Furthermore, when multiple audio signals are received at the same time, processor 101 will send the audio-visual signals to the same terminal in an overlapping manner.

[0113] In step S210, the processor 101 determines whether to terminate the online call. For example, if the online call is terminated by the operation of all users' input devices 7, it is determined that the online call should be terminated. In step S210, if it is determined that the online call should not be terminated, the process returns to step S202. In this case, if there is a change in the playback environment information or location information during the online call, the processor 101 reflects the change and regenerates the audio-visual filter coefficients to continue the online call. In step S210, if it is determined that the online call should be terminated, the processor 101... Figure 10 The processing is now complete.

[0114] Figure 11 This is a flowchart illustrating the second example of the operation during an online call on server Sv. In this second example, not only are the audio-visual filter coefficients generated on server Sv, but also the audio-visual signals for each of the individual terminals are generated. Furthermore, the operations of the host terminal HT and the visitor terminals GT1, GU2, and GU3 follow essentially the same principles. Figure 4 The actions shown.

[0115] In step S301, the processor 101 sends the data of the input screen containing the playback environment information and location information to each terminal HT, GT1, GT2, and GT3. Furthermore, the processor 101 can also send the data of the input screen containing the utilization information to each terminal HT, GT1, GT2, and GT3.

[0116] In step S302, the processor 101 determines whether it has received playback environment information and location information from terminals HT, GT1, GT2, and GT3. If it is determined in step S302 that playback environment information and location information have been received from terminals HT, GT1, GT2, and GT3, the process proceeds to step S303. If it is determined in step S302 that it has not received playback environment information and location information from terminals HT, GT1, GT2, and GT3, the process proceeds to step S307.

[0117] In step S303, the processor 101 stores the received information in memory 102, for example, RAM.

[0118] In step S304, the processor 101 determines whether the input of information is complete, i.e., whether the playback environment information and location information of each terminal have been stored in, for example, RAM. In step S304, if it is determined that the input of information is incomplete, the process returns to step S302. In step S304, if it is determined that the input of information is complete, the process proceeds to step S305.

[0119] In step S305, the processor 101 generates audio-visual filter coefficients for each terminal, i.e., for each user, based on playback environment information and location information for each terminal. The audio-visual filter coefficients generated in step S305 can be the same as the image filter coefficients generated in step S205 of the first example.

[0120] In step S306, the processor 101 stores the audio-visual filter coefficients for each user, for example, in the storage device 103.

[0121] In step S307, the processor 101 determines whether it has received an audio signal from at least one of the terminals HT, GT1, GT2, and GT3 via the communication device 104. If it is determined in step S307 that an audio signal has been received from a terminal, the process proceeds to step S308. If it is determined in step S307 that no audio signal has been received from any terminal, the process proceeds to step S310.

[0122] In step S308, the processor 101 generates audio-visual signals for each user from the received audio signals. For example, when an audio signal is received from terminal HT, the processor 101 convolves the received audio signal with audio-visual filter coefficients generated based on the playback environment information of the audio playback device 4 of terminal GT1 input by user GU1 and the orientation information of user HU specified by user GU1 to generate an audio-visual signal for user GU1. Similarly, the processor 101 convolves the received audio signal with audio-visual filter coefficients generated based on the playback environment information of the audio playback device 4 of terminal GT2 input by user GU2 and the orientation information of user HU specified by user GU2 to generate an audio-visual signal for user GU2. In addition, the processor 101 convolves the received audio signal with audio-visual filter coefficients generated based on the playback environment information of the audio playback device 4 of terminal GT3 input by user GU3 and the orientation information of user HU specified by user GU3 to generate an audio-visual signal for user GU3. Furthermore, the processor 101 can also adjust the audio-visual signal generated based on the utilization information when such information is available. This adjustment will be explained later.

[0123] In step S309, the processor 101 uses the communication device 104 to send the generated audio-visual signals to the corresponding terminals. Afterwards, the processing proceeds to step S310. Furthermore, in each terminal, [the signal is transmitted / transmitted / adapted]... Figure 4 Similarly, in step S12, the received audio-visual signals are played. In server Sv, the audio-visual signals are separated, so step S11 is unnecessary. Furthermore, when multiple audio signals are received at the same time, processor 101 will send the audio-visual signals to the same terminal in an overlapping manner.

[0124] In step S310, the processor 101 determines whether to terminate the online call. For example, if the online call is terminated by the operation of all users' input devices 7, it is determined that the online call should be terminated. In step S310, if it is determined that the online call should not be terminated, the process returns to step S302. In this case, if there is a change in the playback environment information or location information during the online call, the processor 101 reflects the change and regenerates the audio-visual filter coefficients to continue the online call. In step S310, if it is determined that the online call should be terminated, the processor 101 causes... Figure 11 The processing is now complete.

[0125] In the first example of the second embodiment, it is conceivable that multiple pre-shared audio-visual filter coefficients are used by the server, the host terminal, and the visitor terminal. Whenever the server acquires playback environment information and location information, it determines the required audio-visual filter coefficients from the pre-shared audio-visual filter coefficients. Alternatively, the server may not send the audio-visual filter coefficients to the host terminal and each visitor terminal, but only send information indicating the index of the determined audio-visual filter coefficients to the host terminal and each visitor terminal. Furthermore, in the second example of the second embodiment, the server may also determine the required audio-visual filter coefficients from the pre-shared multiple audio-visual filter coefficients whenever it acquires playback environment information and location information. The server can then convolve the determined audio-visual filter coefficients with the audio signal.

[0126] As explained above, in the second embodiment, audio-visual filter coefficients for each terminal user are generated in the server Sv based on playback environment information and location information. This allows for the localization of the audio-visual image of other users based on the playback environment of each terminal's audio playback device 4. Furthermore, in the second embodiment, audio-visual filter coefficients are generated not only in the host terminal HT but also in the server Sv. Therefore, the load on the host terminal HT during online calls can be reduced.

[0127] Furthermore, in the second embodiment, not only the host terminal HT, but also the visitor terminals GT1, GT2, and GT3 specify playback environment information and location information, and generate sound image filter coefficients based on this playback environment information and location information. Therefore, participants in the online call can individually determine the location from which they want the sound image around them to be played.

[0128] [Modification 1 of the second embodiment]

[0129] Next, a variation of the second embodiment will be described. In the aforementioned first and second embodiments, an input screen for orientation information is shown as including... Figure 5 The location input field 2602 is the input screen. In particular, it can also be used as an input screen for location information suitable for online meetings. Figure 12 The input screen is shown below.

[0130] Figure 12 The input screen for location information includes a list 2603 of participants in the online meeting. Within the participant list 2603, labels 2604 represent each participant.

[0131] and then, Figure 12The location information input screen shown includes a diagram of a conference room 2605. Diagram 2605 includes a diagram of a conference table 2606 and diagrams of chairs arranged around the conference table 2606 2607. The user configures a marker 2604 by dragging and dropping it into the chair diagram 2607. Upon accepting this configuration, the processor 101 of server Sv determines the location of other users relative to that user. That is, the processor 101 determines the location of other users based on the positional relationship between its own marker 2604 and the markers 2604 of other users. Thus, location information can be input. By following... Figure 12 The location information shown is used to locate the sound image when inputting to the input screen, allowing users to hear the voices of other users as if they were having a meeting in an actual conference room.

[0132] Here, in Figure 12 In this system, the number of chairs is limited, so each user can identify key figures in the meeting and configure a tag 2604 corresponding to that key figure. The processor 101 of server Sv can directly transmit a mono audio signal without location to each terminal regarding the voices of users not assigned to chairs. In this case, even if the voices of other users not assigned to chairs are determined to be important statements, users can hear the voices of other users in a located manner by appropriately changing the tag.

[0133] in addition, Figure 12 The input screen showing the directional information can also be displayed during online meetings. During an online meeting, users can also change the configuration of marker 2604 to determine the location of other users. This allows for solutions, for example, to handle situations where it is difficult to hear sounds from a specific direction due to changes in the user's surrounding environment. Furthermore, it can also... Figure 12 As shown, the markers of users who have spoken are illuminated, as indicated by reference symbol 2608.

[0134] Figure 12 This is an example of a user freely determining the configurations of other users. In contrast, it could also be like... Figure 13 , Figure 14A as well as Figure 14B As shown, the input screen displays orientation information, such as when a user selects the desired configuration from a set of predefined configurations.

[0135] Figure 13 This is an example of an online meeting where there are two participants, separated by a conference table (illustration 2609), and two users 2610 and 2611 are configured in an opposite manner. For example, user 2610 is "himself". (The last sentence appears to be incomplete and possibly refers to a selection process.) Figure 13In this configuration, the processor 101 sets the orientation of the user 2611 to "0 degrees".

[0136] Figure 14A This is an example of an online meeting with three participants, shown in diagram 2609, separated by a conference table. The user 2610 (representing "self") and two other users 2611 are configured in an opposite manner. (The last sentence appears to be incomplete and possibly refers to a selection process.) Figure 14A In this configuration, the processor 101 sets the orientation of the two users 2611 to "0 degrees" and "θ degrees" respectively.

[0137] Figure 14B This is an example of an online meeting with three participants, shown in diagram 2609, separated by a conference table, and two other users, 2611, positioned at ±θ degrees relative to the user 2610 representing "themselves". (The last part, "in the selection...", appears to be an unrelated fragment and is left untranslated.) Figure 14B In this configuration, the processor 101 sets the orientation of the two users 2611 to "-θ degrees" and "θ degrees" respectively.

[0138] Furthermore, when there are two or three participants in an online meeting, the configuration of each user is not limited to... Figure 13 , Figure 14A , Figure 14B The configuration shown. Additionally, for cases where there are four or more participants in the online meeting, preparations can also be made to... Figure 13 , Figure 14A , Figure 14B The same input screen.

[0139] Furthermore, the shape of the conference table in diagram 2609 is not limited to a quadrilateral. For example, it could also be as follows: Figure 15 As shown in the schematic diagram 2609 of the round-shaped conference table, user 2610, representing "himself," and other users 2611 are configured. Figure 15 It can also be with Figure 12 Similarly, there is an input screen where users can configure location information such as marker 2604.

[0140] Alternatively, you can choose not to imitate. Figure 12 The meeting room in the room, and for example, Figure 16 The image shows an input screen where other users' diagrams 2613 are arranged in a circle around the user 2612 who heard the sound, and markers 2604 are arranged relative to these other users' diagrams 2613, thus allowing for the input of location information. In this case, the markers of users who have spoken may also be illuminated.

[0141] Furthermore, it can also be non-2-dimensional, but rather... Figure 17The orientation information is input on a 3D schematic diagram as shown. For example, the input screen could be such that a schematic diagram 2615 of another user is arranged in 3D on a circle centered on the head of the user 2614 who heard the sound, and a marker 2604 is arranged relative to this schematic diagram 2615, thereby inputting the orientation information. In this case, the marker of the user who spoke could also be illuminated, as shown by reference symbol 2616. In particular, in headsets and headphones, the positioning accuracy in front is prone to deterioration. Therefore, by using vision to sense the direction of the user who spoke, the deterioration of positioning accuracy can be improved.

[0142] [Modification 2 of the second embodiment]

[0143] Next, a variation of the second embodiment will be described. Variation 2 of the second embodiment is a preferred example for online presentations and is a specific example of using information. Figure 18 In Variation 2 of the second embodiment, an example is shown on the display screen on each terminal during an online presentation. Here, the actions of the server Sv during the online presentation process can also be... Figure 10 The first example shown and Figure 11 Perform any example of the second example shown.

[0144] like Figure 18 As shown, in Variation 2 of the second embodiment, the display screen shown during an online presentation includes a dynamic image display area 2617. The dynamic image display area 2617 is an area that displays dynamic images transmitted during the online presentation. The display of the dynamic image display area 2617 can be turned on or off arbitrarily by the user.

[0145] like Figure 18 As shown, in Variation 2 of the second embodiment, the display screen shown during the online presentation also includes a schematic diagram 2618 showing the positioning direction of other users relative to the presenter, and markers 2619a, 2619b, and 2619c representing other users. Similar to Variation 1 of the second embodiment, the user configures the markers 2619a, 2619b, and 2619c by dragging and dropping them onto the schematic diagram 2618. Furthermore, in Variation 2 of the second embodiment, attributes are assigned to each marker 2619a, 2619b, and 2619c as usage information. Attributes may include, for example, the roles of each user in the online presentation; for example, the host user HU can be arbitrarily specified. When attributes are assigned, the name 2620 representing that attribute is displayed on the display screen. Figure 18In this example, the attribute of marker 2619a is "speaker," the attribute of marker 2619b is "co-speaker," and the attribute of marker 2619c is "mechanical sound" such as the sound of a bell. Thus, in variation 2 of the second embodiment, the user is not limited to a human. Furthermore, regarding attributes, besides... Figure 18 Apart from the situations shown, they can be assigned various roles such as "timekeeper".

[0146] For example, when the attributes are specified by the host user HU, the processor 101 of the server Sv can adjust the playback of the audio image for each attribute. For example, when the "speaker's" voice signal and other users' voice signals are input simultaneously, the processor 101 can also send only the "speaker's" voice to each terminal, or locate the audio image in a way that ensures the "speaker's" voice is heard clearly. In addition, the processor 101 can also send only the voices of "machine voice," "timekeeper," etc., to the "speaker's" terminal, or locate the audio image in a way that prevents other terminals from hearing it.

[0147] like Figure 18 As shown, in Variation 2 of the second embodiment, the display screen shown during an online presentation also includes a speaker assistance button 2621 and a listener discussion button 2622. The speaker assistance button 2621 is primarily selected by the speaker's assistant, such as a timer. The speaker assistance button 2621 can also be configured to not be displayed on the assistant's terminal. The listener discussion button 2622 is selected when listeners are listening to the speaker's presentation and engaging in discussion.

[0148] Figure 19 This diagram shows an example of the screen displayed on the terminal when speaker assistance button 2621 is selected. When speaker assistance button 2621 is selected, as shown... Figure 19 As shown, the new display includes a timer setting button 2623, a start button 2624, a stop button 2625, and a temporary stop / restart button 2626.

[0149] The timekeeper setting button 2623 is used to set various settings required by the timekeeper, such as the remaining time to be published and the interval of the bell. The start button 2624, for example, is selected at the beginning of publication and is used to initiate timing processes such as measuring the remaining time to be published and ringing the bell. The stop button 2625 is used to stop the timing process. The temporary stop / restart button 2626 is used to switch between temporary stop and restart of the timing process.

[0150] Figure 20This diagram shows an example of the screen displayed on the terminal when the listener discussion button 2622 is selected. When the listener discussion button 2622 is selected, the screen transitions to... Figure 20 The image shown. Figure 20 The displayed screen includes a schematic diagram 2618 showing the positioning directions of other users relative to the user's own, and markers 2627a and 2627b representing other users. Similar to Variation 1 of the second embodiment, the user configures the markers 2627a and 2627b by dragging and dropping them onto the schematic diagram 2618. Furthermore, attributes are assigned to each marker 2627a and 2627b as usage information. Each user can arbitrarily specify attributes when selecting the listener discussion button 2622. When attributes are assigned, the name representing that attribute is displayed on the display screen. Figure 20 In the example, the attribute of tag 2627a is "speaker", and the attribute of tag 2627b is "D".

[0151] In addition, such as Figure 20 As shown, in Variation 2 of the second embodiment, the display screen shown when the "Discussion among Listeners" button 2622 is selected also includes a group setting bar 2628. The group setting bar 2628 is a display bar for setting groups among listeners. The group setting bar 2628 displays a list of currently set groups. The list of groups includes the group name and the names of the users belonging to that group. The group name can be determined by the user who initially set the group or predetermined. Additionally, a "Participate" button 2629 is displayed near the name of each group in the group setting bar 2628. When the "Participate" button 2629 is selected, the processor 101 assigns the user to the corresponding group.

[0152] Additionally, the display screen shown when the "Discussion among Listeners" button 2622 is selected also includes a "Create New Group" button 2630. The "Create New Group" button 2630 is selected when creating a new group that is not displayed in the group settings bar 2628. When the "Create New Group" button 2630 is selected, the user, for example, sets the group name. Alternatively, it can be configured to allow users who do not wish to participate in the new group creation process to be specified. Regarding users who wish to be excluded from the group, the processor 101 controls this by, for example, not displaying the "Participate" button 2629 on the display screen. Figure 20 In the middle, participation in "Group 2" is set to not be allowed.

[0153] Additionally, when the listener discussion button 2622 is selected, the displayed screen includes a start button 2631 and a stop button 2632. The start button 2631 is used to start the listener discussion. The stop button 2632 is used to stop the listener discussion.

[0154] Furthermore, when the audience discussion button 2622 is selected, the display screen includes a volume balance button 2633. The volume balance button 2633 is used to balance the volume of the "speaker" user with that of other users belonging to the group.

[0155] For example, when a group is set up and the start button 2631 is selected, the processor 101 of the server Sv positions the sound image in a way that the sound can only be heard among users belonging to the group. In addition, the processor 101 adjusts the volume of the "speaker" user and the volume of other users according to the volume balance specification.

[0156] Here, the group settings bar 2628 can also be configured to allow the user who initially set the group to switch between active and inactive groups. In this case, the active and inactive groups can also be displayed in different colors in the group settings bar 2628.

[0157] [Third Implementation]

[0158] Next, the third embodiment will be described. Figure 21 This is a diagram illustrating the structure of an example of the server Sv in the third embodiment. Here, in Figure 21 In the middle, the part about and is omitted. Figure 9 The same structure is described. The difference in the third embodiment is that the storage device 103 stores an echo table 1032. The echo table 1032 is a table of echo information used to add predetermined echo effects to the audio-visual signal. The echo table 1032 contains echo data pre-measured in small meeting rooms, large meeting rooms, and semi-anechoic chambers. The processor 101 of the server Sv obtains echo data corresponding to the virtual environment intended for utilizing the audio-visual signal, which is user-specified utilization information, from the echo table 1032, and then sends the echo data based on the obtained echo data to the audio-visual signal to each terminal.

[0159] Figure 22A , Figure 22B , Figure 22C , Figure 22D This is an example of a screen used to input information related to echo data. Figure 22A - Figure 22D The user-specified virtual environment in the screen utilizes audio and video.

[0160] Figure 22A It is the initial screen displayed, 2634. Figure 22AThe screen 2634 shown includes a "Want to Select" field 2635 for the user to select the echo and a "Delegate" field 2636 for the server Sv to select the echo. For example, the host user HT selects the desired echo from both the "Want to Select" field 2635 and the "Delegate" field 2636. When the "Delegate" field 2636 is selected, the server Sv automatically selects the echo. For example, the server Sv selects any data from echo data measured in a small meeting room, echo data measured in a large meeting room, or echo data measured in a semi-anechoic chamber, based on the number of participants in the online meeting.

[0161] Figure 22B This is screen 2637, which is displayed when the "Want to Select" field 2636 is selected. Figure 22B The screen 2637 shown includes a "Select by Room Type" field 2638 for selecting the echo corresponding to the room type and a "Select by Conversation Size" field 2639 for selecting the echo corresponding to the conversation size. For example, the host user HT selects the desired field from the "Select by Room Type" field 2638 and the "Select by Conversation Size" field 2639.

[0162] Figure 22C This is screen 2640, which is displayed when the "Select by room type" option 2638 is selected. Figure 22C The screen 2640 shown includes a "Small Meeting Room" section 2641 for selecting the echo corresponding to a small meeting room (i.e., a small-scale meeting room), a "Meeting Hall" section 2642 for selecting the echo corresponding to a large meeting hall (i.e., a large-scale meeting room), and a "Low-Sound Room" section 2643 for selecting the echo corresponding to a quiet room (i.e., an anechoic chamber). For example, the host user HT selects the desired section from the "Small Meeting Room" section 2641, the "Meeting Hall" section 2642, and the "Low-Sound Room" section 2643.

[0163] When the user selects the "Small Meeting Room" option 2641, the processor 101 of server Sv retrieves pre-measured echo data from the echo table 1032 for a small meeting room. Similarly, when the user selects the "Meeting Hall" option 2642, the processor 101 retrieves pre-measured echo data from the echo table 1032 for a large meeting room. Furthermore, when the user selects the "Low-Sound Room" option 2643, the processor 101 retrieves pre-measured echo data from the echo table 1032 for an anechoic chamber.

[0164] Figure 22D This is screen 2644, which appears when the "Select by conversation size" option 2639 is selected. Figure 22DThe screen 2644 shown includes a "Internal Meeting" section 2645 for selecting the echo corresponding to a medium-sized conversation, a "Report Meeting, etc." section 2646 for selecting the echo corresponding to a larger conversation, and a "Top Secret Meeting" section 2647 for selecting the echo corresponding to a small conversation. For example, the host user HT selects the desired section from the "Internal Meeting" section 2645, the "Report Meeting, etc." section 2646, and the "Top Secret Meeting" section 2647.

[0165] When the user selects the "Internal Meeting" option 2645, the processor 101 of server Sv retrieves pre-measured echo data from the echo table 1032, which is obtained from a small meeting room. Furthermore, when the user selects the "Report Meeting, etc." option 2646, the processor 101 retrieves pre-measured echo data from the echo table 1032, which is obtained from a large meeting room. Finally, when the user selects the "Top Secret Meeting" option 2647, the processor 101 retrieves pre-measured echo data from the echo table 1032, which is obtained from an anechoic chamber.

[0166] As explained above, according to the third embodiment, echo information corresponding to the room size, purpose of use, and atmosphere of the meeting is maintained in a table on server Sv. Server Sv adds an echo selected from the echo table to the audio signal of each user. As a result, fatigue caused by hearing each user's voice at the same volume level can be reduced.

[0167] In this third embodiment, the echo table includes echo data of three categories. The echo table may include only one or two categories of echo data, or it may include four or more categories of echo data.

[0168] [Modifications of the Third Embodiment]

[0169] In the third embodiment, a level attenuation table 1033 may also be stored in the storage device 103. The level attenuation table 1033, as table data, has level attenuation data corresponding to the distance of the volume measured in advance in the anechoic chamber. In this case, the processor 101 of the server Sv can acquire the level attenuation data corresponding to the virtual distance between the imagined sound source (using the sound image) and the user, and add the level attenuation corresponding to the acquired level attenuation data to the sound image signal. This can also reduce fatigue caused by hearing each user's voice at the same volume level.

[0170] Several embodiments of the present invention have been described, but these embodiments are given by way of example and are not intended to limit the scope of the invention. These embodiments can be implemented in various other ways, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention as well as in the scope of the invention and its equivalents as described in the patent claims.

Claims

1. An online call management device, comprising: The first acquisition unit acquires playback environment information via a network from at least one terminal that plays audio or video via a playback device. This information includes playback environment information related to the sound of the playback device and indicates which device among stereo speakers, headphones, and earphones is used as the playback device. The second acquisition unit acquires information about the positioning direction of the sound image for the user of the terminal, i.e., the orientation information. as well as The control unit controls the playback of audio and video for each terminal based on the playback environment information and the orientation information.

2. The online call management device according to claim 1, wherein, The control unit receives from the terminal the audio-visual signal after convolving the audio-visual filter coefficients based on the playback environment information and the orientation information in the terminal, separates the received audio-visual signal into audio-visual signals facing each terminal, overlaps the audio-visual signals facing the same terminal, and sends the overlapped audio-visual signals to the corresponding terminal.

3. The online call management device according to claim 1, wherein, The control unit determines the audio-visual filter coefficients for playing the audio-visual image for each terminal based on the playback environment information and the orientation information. Based on the determined audio-visual filter coefficients of each terminal, an audio-visual signal for each terminal is generated from the audio signal transmitted from the terminal. The generated audio-visual signals of each terminal are sent to the corresponding terminal.

4. The online call management device according to claim 1, wherein, There are multiple terminals. One of the multiple terminals is designated as the host terminal. The first acquisition unit acquires playback environment information about each of the terminals from each of the terminals. The second acquisition unit centrally acquires the location information of each terminal from the terminals of the host.

5. The online call management device according to claim 4, wherein, The first acquisition unit causes each of the terminals to display a first input screen for inputting the playback environment information, and acquires the playback environment information about each of the terminals based on the input at the first input screen. The second acquisition unit causes the terminal of the host to display a second input screen for inputting the orientation information of each of the terminals, and acquires the orientation information of each of the terminals from the terminal of the host based on the input at the second input screen.

6. The online call management device according to claim 1, wherein, There are multiple terminals. The first acquisition unit acquires playback environment information about each of the terminals from each of the terminals. The second acquisition unit acquires the orientation information about each of the terminals from each of the terminals.

7. The online call management device according to claim 6, wherein, The first acquisition unit causes each of the terminals to display a first input screen for inputting the playback environment information, and acquires the playback environment information about each of the terminals based on the input at the first input screen. The second acquisition unit causes each of the terminals to also display a second input screen for inputting the orientation information about each of the terminals, and acquires the orientation information about each of the terminals from each of the terminals based on the input at the second input screen.

8. The online call management device according to claim 5 or 7, wherein, The first input screen includes a list of playback devices.

9. The online call management device according to claim 5 or 7, wherein, The second input screen includes an input field for locating the direction of sounds emitted by each user as the audio-visual representation.

10. The online call management device according to claim 5 or 7, wherein, The second input screen includes an input screen that locates the direction of the sound emitted by each user as the sound image by placing markers at each seat in a layout diagram that mimics a conference room.

11. The online call management device according to claim 10, wherein, The second input screen is configured to place a marker at the seat by dragging the marker.

12. The online call management device according to claim 5 or 7, wherein, The second input screen includes an input screen that locates the sound emitted from each user as an audio image by specifying the positions of other users on a circumference centered on the user's position on the terminal.

13. The online call management device according to claim 1, wherein, The online call management device further includes a third acquisition unit, which acquires information related to the utilization of the user's audio-visual information, i.e., utilization information. The control unit then controls the playback of audio and video for each terminal based on the utilization information.

14. The online call management device according to claim 13, wherein, The third acquisition unit causes each of the terminals to display a third input screen for inputting the utilization information, and acquires the utilization information about each of the terminals based on the input at the third input screen.

15. The online call management device according to claim 14, wherein, The utilization information includes information about attributes assigned to each user. The control unit then controls the playback of audio and video for each terminal based on the attribute information.

16. The online call management device according to claim 14 or 15, wherein, The utilization information includes the group settings for each user of the terminal. The control unit then controls the playback of audio and video for each terminal according to the settings of the group.

17. The online call management device according to claim 14, wherein, The third input screen includes a first input unit for accepting a setting to play the audio-visual content based on the utilization information, a second input unit for accepting an instruction to start playing the audio-visual content based on the utilization information, a third input unit for accepting an instruction to temporarily stop or restart playing the audio-visual content based on the utilization information, and a fourth input unit for accepting an instruction to stop playing the audio-visual content based on the utilization information.

18. The online call management device according to claim 13, wherein, The information to be utilized includes information about a virtual environment that envisions utilizing the audiovisuals. The control unit adds an echo corresponding to the information of the virtual environment to the audio-visual image of each terminal.

19. The online call management device according to claim 18, wherein, The control unit adds the echo to the sound image of each terminal based on the table data of the echo pre-measured in the actual environment corresponding to the virtual environment.

20. The online call management device according to claim 13, wherein, The information to be utilized includes information about the distance between the virtual sound source playing the audio and the user of the terminal. The control unit adds a level of attenuation to the acoustic image of each terminal, corresponding to the distance.

21. The online call management device according to claim 20, wherein, The control unit adds the level of attenuation to the sound image of each terminal based on the table data of the level attenuation pre-measured in the anechoic chamber.

22. A computer-readable storage medium storing an online call management program, the online call management program being used to cause the computer to perform: The playback environment information is obtained via a network from at least one terminal that plays audio or video via a playback device. This information includes playback environment information related to the sound of the playback device and indicates which device, such as a stereo speaker, headphones, or earphones, is used as the playback device. Obtain the location information, i.e., orientation information, of the acoustic image for the user of the terminal; as well as The playback of audio and video for each terminal is controlled based on the playback environment information and the orientation information.

Images