Voice transmission / reception system

Abstract

To solve the problem that another person's voice uttered by another nearby participant interferes with a distributed voice to make it hard to hear.SOLUTION: An audio transmission and reception system (100), in which an information terminal (3) includes a microphone (12), a communication unit (15), an audio output unit (14), and a processor (16), the processor connecting to a new information terminal when finding the new information terminal by near field communication in a situation in which the information terminal is already connected to another information terminal by near field communication, and transmitting information relating to the new information terminal to the other information terminal, A distribution prohibition list (62) for prohibiting spoken voice collected by all other information terminals connected by short-range communication and a new information terminal searched by the other information terminals from being included in distribution voice is created and transmitted to a server (5), and distribution voice after voice reduction processing of the spoken voice based on the distribution prohibition list is received and output from a voice output device.SELECTED DRAWING: Figure 2A

Description

[Technical Field]

[0001] The present invention relates to an audio transmission and reception system. [Background technology]

[0002] For business purposes, remote conferences are held by sending and receiving audio data between remote locations using chat systems implemented in web conferencing systems. In the past, chats were conducted using one chat terminal per location, with multiple participants at the same location sharing the screen and audio, but in recent years, chat applications running on personal computers and smartphones have become more common, and participants at the same location now each run their own chat application on their own chat terminal.

[0003] An example of audio processing for a conference between locations is described in Patent Document 1 (JP Patent Publication No. 8-237627). Patent Document 1 discloses a multipoint video conference system that aims to "prevent the speaker's own voice from being heard at the speaker's terminal (abstract excerpt)." In this multipoint video conference system, the spoken voice is not distributed to the speaker's terminal and is not output. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 8-237627 Summary of the Invention [Problem to be solved by the invention]

[0005] On the other hand, when participants at the same location participate in a conference using their own chat terminals, the voice of the participant (say, participant A) speaking for the remote conference (referred to as a conference between locations) is collected by a microphone on participant A's chat terminal, sent to a chat server, and then distributed to the chat terminals of participants at other locations and to the chat terminal of another participant at the same location (e.g., participant B), and output from the chat terminal's speaker or earphones. As a result, participant B hears both participant A's voice (other person's voice) directly and the distributed voice output from participant B's chat terminal. Hereinafter, in this specification, "other person's voice" refers to the voice of another person (a person other than the user of the chat terminal) speaking in the same location, and is directly audible. Furthermore, "distributed voice" refers to the voice output from the chat terminal.

[0006] Because the distributed audio is delayed as it passes through the chat server, when two audio streams overlap, the same audio is played twice with a time difference, making it very difficult to hear.

[0007] Although Patent Document 1 allows a speaker to prevent their own voice from being heard on their terminal, it does not mention the problem of audio interference between other people's voices and the broadcast audio that occurs when there are multiple chat terminals at the same location, and therefore the above problem cannot be solved.

[0008] The present invention has been made in consideration of the above points, and its purpose is to eliminate the problem that when multiple participants from the same location participate using their own chat terminals, the voices of other participants nearby interfere with the broadcast voice, making it difficult to hear. [Means for solving the problem]

[0009] In order to solve the above problems, the present invention has the configurations described in the claims. [Effects of the Invention]

[0010] According to the present invention, when multiple participants from the same location participate in a chat using their own chat terminals, the problem of voices from nearby participants interfering with the broadcast voice and making it difficult to hear can be eliminated. Objects, configurations, and effects other than those described above will be clarified in the following embodiments. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating the configuration of a web conference system. [Figure 2A] FIG. 2 is a hardware configuration diagram of a web conference terminal. [Figure 2B] FIG. 2 is a hardware configuration diagram of a web conference terminal. [Figure 3] FIG. 2 is a functional block diagram of a web conference terminal according to the first embodiment. [Figure 4] FIG. 3 is a functional block diagram showing details of a correlation calculation unit. [Figure 5] 10A and 10B are diagrams illustrating a first example of a process for reducing the voice of another person near the user included in the distributed voice. [Figure 6] 4 is a flowchart showing the flow of processing in the web conference system according to the first embodiment. [Figure 7] 10A and 10B are diagrams illustrating a second example of the voice reduction process for other people's voices. [Figure 8] 10 is a flowchart showing the flow of processing in a web conference system including a second sound reduction process for speech sounds (sounds spoken by other people distributed on the system). [Figure 9] This is a diagram of a mesh network configuration between web conference terminals within a base. [Figure 10] 10A and 10B are diagrams illustrating a process of reducing the volume of speech based on a distribution prohibition list. [Figure 11] 10 is a flowchart showing the flow of processing in a web conference system that supports a third voice reduction process for voices of other people. [Figure 12] FIG. 10 is a configuration diagram of a web conference system according to a third embodiment. [Figure 13]FIG. 1 is a block diagram of a web conference terminal realized by an information processing device. [Figure 14] FIG. 10 is a functional block diagram of a web conference terminal according to a third embodiment. [Figure 15] 10 is a flowchart showing the flow of processing in a Web conference system that corresponds to a serverless Web conference system. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The same components and steps are designated by the same reference numerals throughout the drawings, and redundant explanations will be omitted.

[0013] The chat system according to the present invention is a system for transmitting and receiving voice data between multiple chat terminals directly or via a chat server. The chat system can be applied to a work support system for transmitting and receiving voice data between chat terminals worn by workers working at a work site, and a terminal at a management center located away from the work site.

[0014] The chat system according to the present invention can also be applied to a voice chat system in which voice data is transmitted and received between chat terminals worn by team members directly or via a chat server when multiple people form a team to play e-sports. Furthermore, the chat system according to the present invention can also be applied to an e-sports system or game system incorporating a voice chat system.

[0015] In the following explanation, we will explain an example of a web conferencing system that incorporates the chat system according to the present invention. For example, the present invention is expected to contribute to diversification and technological improvement in labor-intensive industries, and therefore contribute to the achievement of SDG 8.2 (increasing economic productivity through diversification, technological improvement, and innovation, particularly in labor-intensive industries and industries that increase the value of goods and services) proposed by the United Nations.

[0016] [First embodiment of the present invention] A first embodiment of the present invention will be described with reference to FIGS.

[0017] Figure 1 is a diagram showing the configuration of a web conference system.

[0018] 1, the web conference system 100 is configured by connecting web conference terminals 3A to 3F (corresponding to chat terminals; hereinafter, sometimes simply referred to as "terminals") installed at web conference locations A, B, and C, respectively, and a web conference server 5 (corresponding to chat server) via a network 4. Office AO is an office room installed at web conference location A.

[0019] The following explanation will be given using base A as an example, but the explanation for base A also applies to bases B and C.

[0020] Web conference participants 2A, 2B, and 2C are present at site A. The Web conference terminals used by participants 2A, 2B, and 2C are terminals 3A, 3B, and 3C, respectively.

[0021] When participants 2A, 2B, and 2C at base A gather in the same room such as a conference room to hold a web conference, participants 2A, 2B, and 2C hold the web conference using their own terminals 3A, 3B, and 3C, respectively.

[0022] Participants 2A, 2B, and 2C are participating in the web conference from site A, and they access web conference server 5 via network 4 using terminals 3A, 3B, and 3C, respectively, to receive web conference services. For example, participant A's image and speech (hereinafter referred to as "user voice") are collected by terminal A and sent to web conference server 5.

[0023] The web conference server 5 receives images and audio from all participants connected to the web conference service, generates distribution images and audio for the web conference, and distributes them to each participant's terminal. For example, participant A's speech (user voice) is distributed to the terminals of participants at sites B and C (terminals D, E, and F) as part of the distribution audio for the web conference.

[0024] However, the speech of participant A (user voice) is not included in the distributed audio output from terminals 3B and 3C operated by other participants near participant A, in this example, participants B and C. This eliminates the problem of participants B and C hearing the speech of participant A (other people's voice) directly through the air in office AO and the speech of participant A (user voice) included in the distributed audio output from terminals 3B and 3C with a time lag. This is one of the features common to all embodiments of the present invention.

[0025] 2A and 2B are diagrams showing the hardware configuration of a web conference terminal. Web conference terminals 3A to 3F have the same configuration, so when there is no need to distinguish between the terminals, they will be referred to as terminal 3.

[0026] The terminal 3 includes a camera 11, a microphone 12, a display 13, an audio output device 14, a communication device 15, a processor 16, a first storage device (RAM) 17, a second storage device (FROM) 18, an input device 19, and a group of sensors 20, which are connected to each other by a bus 21. The camera 11 and the display 13 are not essential for the terminal 3, and in that case, a web conference can be held using only audio.

[0027] The processor 16 is configured by, for example, a CPU.

[0028] The RAM 17 is an example of a volatile memory.

[0029] The FROM 18 is an example of a nonvolatile memory and includes a basic operation program 30, a web conference application (abbreviated as "app" in the figure) program 31, and data 32.

[0030] The camera 11 may be configured integrally with the terminal 3, or may be a camera connected via a USB terminal.

[0031] The microphone 12 collects not only the voice of the user of the terminal 3 (user voice) but also the voices of other participants at the same location speaking in the web conference (other voices). If there is only one microphone 12 and it is not directional, it collects both the user voice and the other voices. When the term "microphone-collected voice" is used to refer to the voice collected by the microphone 12 without distinguishing between the user voice and the other voices, it is simply referred to as "microphone-collected voice."

[0032] FIG. 2A illustrates a single microphone 12 (user microphone) for collecting both user and other voices. However, separate microphones with directivity suitable for collecting each voice may be provided. A microphone with directivity suitable for collecting user voice from terminal 3 is called a user-dedicated microphone, and a microphone with directivity suitable for collecting surrounding voices is called a shared microphone. A user-dedicated microphone is, for example, a microphone included in a headset. A shared microphone is, for example, a microphone suitable for collecting omnidirectional voice placed on a desk in a conference room. As shown in FIG. 2B, a dedicated microphone for other voices 12a (sometimes abbreviated as "dedicated microphone") may be connected to the bus 21, or a dedicated microphone for other voices 12b may be connected via Bluetooth (registered trademark) via a short-range wireless communication device 152. Other voices are voices collected by a microphone (user microphone or microphone dedicated to collecting other voices) when the speaker is not speaking. Using a user microphone as a dedicated microphone is preferable because it eliminates the need to add a dedicated microphone for other voices 12b. The microphone 12 (user's microphone) is muted when the user is not speaking (the microphone 12 function itself is still operating, but the audio from the microphone 12 is not broadcast), so that any audio picked up during that time is not processed as the user's own voice, but as the voice of another person.

[0033] The input device 19 is a keyboard or a touch sensor. In the case of a smartphone, the flat display (display 13) and the touch sensor are integrated, and the keyboard operates according to a basic operation program 30.

[0034] The audio output device 14 is a device that outputs the distributed audio, and may be a speaker, earphones, headphones, a headset, or an audio output terminal.

[0035] The communication device 15 includes a LAN communication device 151 that exchanges data such as images and audio with the web conference server 5, and also includes multiple communication methods and communication protocols, such as a Bluetooth (registered trademark) short-range wireless communication device 152, that are executed between terminals within the base.

[0036] The sensor group 20 includes, for example, an illuminance sensor 201 and a motion sensor 202, and assists in the use of the terminal.

[0037] FIG. 3 is a functional block diagram of the web conference terminal according to the first embodiment.

[0038] The WEB conference terminal 3 has a correlation calculation unit 161 and a sound reduction unit 162. The correlation calculation unit 161 and the sound reduction unit 162 are realized by the processor 16 expanding the basic operation program 30 and the WEB conference application program 31 in the RAM 17 and executing them. The data 32 includes data necessary to execute the basic operation program 30 and the WEB conference application program 31, and is read out as appropriate when the processor 16 executes the WEB conference application program 31 and used for processing by each unit.

[0039] The image of the terminal user taken by the camera 11 is transmitted from the LAN communicator 151 to the web conference server 5 via the network 4 .

[0040] The LAN communication device 151 receives distributed images and distributed audio for a web conference from the web conference server 5. The distributed images are displayed on the display 13. The distributed audio is supplied to a correlation calculation unit 161 and an audio reduction unit 162.

[0041] In addition, the user's voice and the voice of others (voice collected by the microphone) collected by the microphone 12 are transmitted from the LAN communicator 151 to the web conference server 5 via the network 4 and are also supplied to the correlation calculation unit 161 and the voice reduction unit 162.

[0042] The correlation calculation unit 161 receives the delivered voice and the user voice and other voices from the microphone 12 as input, performs correlation calculations, obtains the amount of delay and correlation between the two voices, and sends them to the voice reduction unit 162.

[0043] The audio reduction unit 162 reduces the user's audio and other audio from the delivery audio by, for example, subtracting the user's audio and other audio from the delivery audio with reference to the delay amount and correlation amount, and generates an output audio for the terminal 3.

[0044] The audio output unit 14 outputs the output audio from the audio reduction unit 162. This reduces the microphone collected audio (user audio and other audio) collected by the microphone 12 of the terminal and output from the audio output unit 14 as a distributed audio, thereby reducing interference with other audio that can be heard directly.

[0045] FIG. 4 is a functional block diagram showing the details of the correlation calculation unit.

[0046] The correlation calculation unit 161 includes a variable delay unit 161a, a delay amount setting unit 161b, a product-sum unit 161c, and an output processing unit 161d.

[0047] The microphone-collected voice (user voice and other voices) is input to the variable delay unit 161a. The delay time in the variable delay unit 161a is set by the delay amount setting unit 161b. The "utterance voice" input to the variable delay unit 161a is the user voice or the other voice picked up while muted.

[0048] The delayed microphone-collected audio (user audio and other people's audio) and the distributed audio are input to the product-sum unit 161c, which performs a product-sum operation to obtain a correlation amount using the set delay time as a parameter. The product-sum unit 161c varies the delay time to obtain the delay time that maximizes the correlation amount, and regards this as the delay amount associated with the distribution and the correlation amount.

[0049] The output processing unit 161d outputs the delay amount and the correlation amount when the distributed audio is a superimposed audio as shown in Fig. 5, which will be described later. When the distributed audio is a packet-multiplexed audio as shown in Fig. 7, which will be described later, the output processing unit 161d compares the correlation amount for each audio packet to be separated, and outputs the packet ID corresponding to the microphone-collected audio (user audio and other people's audio).

[0050] (First example of sound reduction processing) FIG. 5 is a diagram illustrating a first example of a process for reducing the voice of another person near the user included in the distributed voice.

[0051] The web conference server 5 includes an audio distribution unit 50. The audio distribution unit 50 sends a distributed audio 53 to the audio reduction unit 162.

[0052] In FIG. 5, voice 51A from terminal A and voices collected by other terminals (51E, 51D, 51F in FIG. 5) are superimposed and added in voice multiplexing section 52 and delivered as voice delivery 53.

[0053] The collected sound subtraction unit 162a of the sound reduction unit 162 subtracts the sound of other people near the user (other people's sound) from the distributed sound 53 by referring to the delay amount and correlation amount obtained by the correlation calculation unit 161.

[0054] FIG. 6 is a flowchart showing the flow of processing in the web conference system according to the first embodiment.

[0055] When the terminal 3 starts the Web conference application program 31 (S10), the terminal 3 logs in to the Web conference service provided by the Web conference server 5 (S11) and participates in the Web conference.

[0056] The terminal 3 captures a camera image with the camera 11 (S12) and collects sound with the microphone 12 (S13).

[0057] The terminal 3 transmits the camera image and the microphone audio collected by the microphone 12 of the terminal 3 to the web conference server 5 (S14). The web conference server 5 receives the distributed image and distributed audio (S15).

[0058] If the microphone mute button of terminal 3 is pressed and terminal 3 is in mute ON state (S16: Yes), the terminal user has no intention of speaking, and therefore terminal 3 determines that the voice picked up by the microphone is the voice of another person.

[0059] By keeping the user's microphone operating even when muted, it can be used as a microphone for collecting other people's voices. Also, a microphone for collecting other people's voices may be provided separately from the user's microphone. By placing the microphone for collecting other people's voices near the conference speaker who is close to the terminal user, other people's voices can be collected more accurately, and the accuracy of correlation calculations can be improved. When a microphone for collecting other people's voices is used, the microphone for collecting other people's voices is used to collect microphone voices in S13. When terminal 3 is in the mute ON state, the microphone for collecting other people's voices is used to collect microphone voices in S13.

[0060] If the terminal 3 is in a mute ON state (S16: Yes), the correlation calculation unit 161 performs a correlation calculation between the distributed voice and the voice of another person, calculates the amount of delay and correlation, and outputs the results to the voice reduction unit 162.

[0061] Specifically, the sound reduction unit 162 subtracts the sound collected by the microphone (other people's sound) from the distributed sound (S17, S18), and the distributed sound from which the other people's sound has been subtracted is output from the sound output unit 14 (S18, S19). The sound output from the sound output unit 14 is called "amplified sound."

[0062] If the terminal 3 is in the mute OFF state (S16: No), it is assumed that the user's voice is not included in the distributed audio (the user's voice has already been removed using existing methods), so the distributed audio is output as is as amplified audio from the audio output device 14 (S19).

[0063] If the user does not want to log out (S21: NO), the process returns to step S12 and repeats. If the user wants to log out (S21: YES), the web conference application program is terminated (S22).

[0064] (Second example of sound reduction processing) FIG. 7 is a diagram illustrating a second example of the voice reduction process for other people's voices.

[0065] 7, similarly to FIG. 5, the web conference server 5 includes an audio distribution unit 50. The audio distribution unit 50 sends a distribution audio 56 to an audio reduction unit 162.

[0066] The voice of the web conference (voice 51A from terminal A in Figure 5) and the voice collected from other terminals D, E, and F (51D, 51E, and 51F in Figure 5) are multiplexed by packet multiplexing unit 55, which stores the voice of each terminal in packets with different identification numbers (hereinafter referred to as ID), and distributes them as distributed voice 56.

[0067] The packet removal unit 57 of the audio reduction unit 162 separates the speech audio (speech delivered on the system by others) from the delivered audio 56 using the packet ID obtained by the correlation calculation unit 161, and removes it in the packet removal unit 57. The terminal audio after removal is 51D, 51E, and 51F, which are multiplexed in the audio multiplexing unit 58 and sent to the audio output unit 14.

[0068] FIG. 8 is a flowchart showing the flow of processing in the web conference system including the second sound reduction processing of speech sounds (speech sounds of other people distributed on the system).

[0069] The sound reduction process conforms to the sound reduction method shown in Fig. 7. Steps with the same functions as those in the first flowchart explained in Fig. 6 are given the same numbers, and duplicate explanations will be omitted.

[0070] The flowchart in Figure 8 differs from the flowchart in Figure 6 in step S30, where the speech (speech delivered by other people on the system) is removed using the packet removal method described in Figure 7.

[0071] As described above, the WEB conference terminal, WEB conference application, and WEB conference system of the first embodiment of the present invention have the advantage that in a WEB conference in which participants use their own WEB conference terminals, there is little interference between the participants' vocalizations and the audio distributed by the WEB conference, making it possible to provide a WEB conference in which the vocalizations are easy to hear.

[0072] [Second embodiment of the present invention] A second embodiment of the present invention will be described with reference to FIGS.

[0073] Figure 9 shows a mesh network configuration between web conference terminals within a site. Figure 9 shows a state where terminal A, terminal B, and terminal C exist at site A and are connected to each other via near field communication 36, and terminal H is added to the network.

[0074] When terminal H enters site A, it searches the vicinity using near-field communication 36 and completes a connection with terminal C that is available for connection. Terminal C detects the new participation of terminal H and notifies terminals A and B of this, and also conveys information about terminals A and B to terminal H. As a result, terminals A, B, C, and H obtain information about all terminals within site A, and are able to create a distribution audio prohibition list that prohibits speech audio collected by terminals within the same site from being included in the distribution audio.

[0075] FIG. 10 is a diagram for explaining the process of reducing the volume of speech voice based on the distribution prohibition list, and shows the voice distribution unit of the web conference server.

[0076] The microphone-collected audio 51A collected by the microphone 12 of terminal A and the microphone-collected audio (51D, 51E, 51F) collected from other terminals are input to the packet removal unit 60. The data values ​​are added in the audio multiplexing unit 61, and the result is distributed as distribution audio 63.

[0077] The audio distribution unit 50 of the web conference server 5 receives a distribution prohibition list 62 from the terminals of the participants, and for example, the distribution prohibition list 62 of terminal B lists terminals A, C, and H, which are located at the same location. In this way, the distribution prohibition list 62 specifies for each terminal the audio that should be removed from the audio distributed by that terminal. The audio that should be removed is defined by the name of the terminal (terminal A, C, H, etc.) to which the microphone that collected that audio is connected.

[0078] In generating the audio to be distributed to terminal B, packet removal unit 60 removes audio packets that are included in distribution prohibition list 62 for each terminal based on distribution prohibition list 62.

[0079] The audio multiplexer 61 adds (multiplexes) the audio that has passed through the packet remover 60 and remains, to generate audio to be delivered 63, which is delivered to the terminal B.

[0080] FIG. 11 is a flowchart showing the flow of processing in a web conference system that supports the third voice reduction processing for other people's voices.

[0081] In the flowchart of FIG. 11, steps having the same functions as those in the flowchart explained in FIG. 6 are given the same numbers, and duplicated explanations will be omitted.

[0082] The flowchart in Fig. 11 differs from the first flowchart in Fig. 6 in steps S40, S41, and S42. In S40, the close proximity communication network described in Fig. 9 is newly created or updated. In S41, the distribution prohibition list 62 is newly created or updated, and in S42, the distribution prohibition list 62 is sent to the web conference server 5.

[0083] In S15, the distributed images and audio are received from the web conference server 5, but the received distributed audio does not include the audio of other participants at the same location, as explained in FIG.

[0084] As described above, the WEB conference terminal, WEB conference application, and WEB conference system of the second embodiment of the present invention have the same features as the first embodiment, and also have the feature of being able to reliably remove the voices of other participants who are at the same location.

[0085] [Third embodiment of the present invention] A third embodiment of the present invention will be described with reference to Figures 12 to 14. This embodiment is an example in which a web conference can be held without a web conference server 5.

[0086] FIG. 12 is a configuration diagram of a web conference system according to the third embodiment.

[0087] 12 differs from the web conference system in Fig. 1 in that it is a serverless system without a web conference server 5. For example, the camera image and microphone audio of participant 2A captured and collected by terminal 3A are distributed to the terminals of all participants participating in the web conference (terminals B to F).

[0088] Terminal 3A also receives images and audio from all terminals (terminals B to F) and generates images and audio for the web conference within the terminal.

[0089] Figure 13 is a block diagram of a web conference terminal realized by an information processing device, which is a web conference terminal that supports serverless web conferences. In the web conference terminal in Figure 13, blocks that have the same functions as the web conference terminal in Figure 3 are assigned the same numbers, and duplicated explanations will be omitted.

[0090] In terminal 3 of Figure 13, the WEB conference application program 31 included in FROM18 includes a server program 33 and a client program 34, and the server program 33 distributes camera images and microphone-captured audio of the terminal user to other terminals and receives images and audio from other terminals.

[0091] The client program 34 captures and collects the camera images and microphone audio of the terminal user, and shares with the server program 33 the camera images and microphone audio of the terminal user, as well as the camera images and microphone audio from other terminals.

[0092] The server program 33 generates images and audio for the web conference and outputs the video and audio to the display 13 and audio output device 14 via the client program 34. Note that the server program 33 does not need to be installed on all terminals participating in the web conference; the web conference can be held if it is installed on at least one terminal. In this case, the terminal on which the server program 33 is installed and the client program 34 of the other terminal exchange images and audio via the communication unit 24.

[0093] FIG. 14 is a functional block diagram of a web conference terminal according to the third embodiment.

[0094] The terminal 3 in FIG. 14 further comprises a participant list creation unit 163 that creates a participant list based on the results of communication from the short-range wireless communication device 152 to the short-range communication 35 in addition to the components of the terminal 3 in FIG.

[0095] FIG. 15 is a flowchart showing the flow of processing in a web conference system that supports a serverless web conference system.

[0096] The same steps as those in the flowchart showing the processing flow of the web conference system in FIG. 6 are given the same numbers.

[0097] The program starts (S10). The flowchart showing the processing flow of the web conference system consists of a client process and a server process.

[0098] In the client process, a notification is sent to the terminals of the potential participants listed in the list of potential participants that has been obtained in advance (S50).

[0099] When a camera image is captured (S12) and sound is collected by the microphone 12 (S13), the camera image and sound collected by the microphone are shared with the server process.

[0100] Furthermore, in S51, the image and sound output by the server process are shared.

[0101] In S16, it is confirmed whether the microphone-collected audio shared in S51 includes audio from other participants at the same location. If it is determined that audio from other participants is present (S16: YES), the correlation calculation unit 161 performs correlation calculation between the output audio of the server process and the audio from other participants at the same location (S17), outputs parameters indicating the amount of delay and correlation to the audio reduction unit 162, and the audio reduction unit 162 subtracts the audio from the other participants (S18) and outputs amplified audio (S19). In addition, the image shared in step S51 is displayed on the display 13 (S20).

[0102] In the server process, a report is received from each terminal (S52), and the participant list creation unit 163 creates or updates a list of participants who are actually participating in the conference from the pre-distributed list of potential participants (S53).

[0103] In S54, the camera images and collected audio are shared with the client process, and then in S55, the camera images and audio from other terminals are received. In S56, the output image of the web conference is obtained from the camera images of all terminals.

[0104] In S57, it is checked whether there is a distribution prohibition list and whether the spoken voice is included in the distribution prohibition list. If there is a distribution prohibition list and the spoken voice is included in the distribution prohibition list (S57: YES), the voice of the other person is removed (S58). The distribution prohibition list is configured by having a distribution prohibition item (flag) in the participant list. In this case, the list of distribution-prohibited participants in the participant list corresponds to the distribution prohibition list.

[0105] If there is no distribution prohibition list or if the other person's voice is not included in the distribution prohibition list (S57: NO), step S58 is skipped. Then, in step S59, output voice is created and shared with the client process.

[0106] The output images and audio of the server process correspond to the distributed images and audio of a server-based web conferencing system.

[0107] As described above, the web conference terminal, web conference application, and web conference system according to the third embodiment of the present invention have the same features as the first and second embodiments, and also enable serverless web conferences, which is advantageous in terms of cost when running a web conference with a small number of terminals.

[0108] [Fourth embodiment of the present invention] When web conference participants use noise-canceling headphones (NCH) during a web conference, it is possible to use noise-canceling technology to reduce the actual speaker's voice, without reducing the speaker's voice included in the system audio output from the NCH. However, in this case, external sounds other than the speaker's voice are also reduced, resulting in inconveniences such as not being able to hear phone ringtones or other people calling during the web conference. Therefore, by enabling the NCH's noise-canceling function only when the actual speaker's voice is being heard, the actual speaker's voice is reduced, and by disabling the noise-canceling function when there is no actual speaker's voice, the external sounds can be distinguished. Furthermore, by performing external noise cancellation only on the system audio, it is possible to reduce only the speaker's voice, making it possible to distinguish other external sounds even when there is speech.

[0109] Although each embodiment has been described using a web conference as an example, the technique of the present invention is also effective in systems that use information terminals to hold conversations between remote locations with participants in the vicinity, not limited to web conferences.

[0110] Although the embodiments of the present invention have been described above, it goes without saying that the configurations for realizing the technology of the present invention are not limited to the above-described embodiments, and various modifications are possible. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. All of these fall within the scope of the present invention. Furthermore, numerical values, messages, etc. appearing in the text and figures are merely examples, and the effects of the present invention will not be impaired even if different ones are used.

[0111] The programs described in each processing example may be independent programs, or multiple programs may constitute a single application program. The order in which each process is performed may also be changed.

[0112] Some or all of the functions of the present invention described above may be implemented in hardware, for example, by designing them as integrated circuits. They may also be implemented in software by a microprocessor unit, CPU, or the like interpreting and executing an operating program that implements each function. Furthermore, the scope of software implementation is not limited, and hardware and software may be used together. Some or all of the functions may also be implemented by a server. The server may be, for example, a local server, a cloud server, an edge server, or an online service, as long as it can cooperate with other components via communications to execute the functions. Information such as programs, tables, and files that implement each function may be stored in a memory, a recording device such as a hard disk or solid-state drive (SSD), or a recording medium such as an IC card, SD card, or DVD, or may be stored in a device on a communications network.

[0113] Furthermore, the control lines and information lines shown in the diagram are those considered necessary for explanation, and do not necessarily represent all the control lines and information lines on the product. In reality, it can be assumed that almost all components are interconnected.

[0114] The above embodiment includes the following aspects.

[0115] (Appendix 1) A chat terminal, With a microphone, a communication device for transmitting and receiving data to and from a chat server; An audio output device; a processor, The microphone collects user voices uttered by the terminal user and other voices uttered by other people who are near the terminal user, The communication device transmits the user's voice to the chat server, and receives the user's voice from the chat server. Receive the broadcast audio from The processor determines a correlation between the distributed voice and the other person's voice, reducing the other person's voice included in the distributed voice; outputting the distribution audio with the other person's audio reduced to the audio output device; Chat terminal. (Appendix 2) A chat terminal, With a microphone, a communication device for transmitting and receiving data to and from other chat terminals; An audio output device; a processor, The microphone collects user voices uttered by the terminal user and other voices uttered by other people who are near the terminal user, the communication device transmits the user's voice to the other chat terminal and receives a distributed voice from the other chat terminal; The processor determines a correlation between the distributed voice and the other person's voice, reducing the other person's voice included in the distributed voice; outputting the distribution audio with the other person's audio reduced to the audio output device; Chat terminal. (Appendix 3) A chat system configured by connecting chat terminals and a chat server for communication. , The chat terminal is With a microphone, a communication device for transmitting and receiving data to and from a chat server; An audio output device; a processor, The microphone captures the user's voice uttered by the terminal user and the voice of other people nearby the terminal user. The voice of others generated by the person is collected, The communication device transmits the user's voice to the chat server, and receives the user's voice from the chat server. Receive the broadcast audio from The processor determines a correlation between the distributed voice and the other person's voice, reducing the other person's voice included in the distributed voice; outputting the distribution audio with the other person's audio reduced to the audio output device; Chat system. (Appendix 4) A control method for a chat system configured by connecting a chat terminal and a chat server for communication It is a law, A user's voice uttered by a terminal user through a microphone connected to a chat terminal, and A step of collecting a voice of another person who is in the vicinity of the user; A process for transmitting the user's voice to a chat server and receiving the voice from the chat server. Tep and determining a correlation between the distributed voice and the voice of another person; reducing the other person's voice included in the distributed voice; The distributed audio with the other person's audio reduced is output from an audio output device connected to the chat terminal. an outputting step; A method for controlling a chat system including: [Explanation of symbols]

[0116] 2A:Participant 2B:Participant 2C:Participant 3: Web conferencing terminal 3A: Web conferencing terminal 3B: Web conference terminal 3C: Web conferencing terminal 3D: Web conferencing terminal 3E: Web conference terminal 3F: Web conferencing terminal 4: Network 5: Web conference server 11: Camera 12: Microphone 12a: Microphone for other people's voices 12b: Microphone for other people's voices 13: Display 14: Audio output device 15:Communication device 16: Processor 17:RAM 19: Input device 20: Sensor group 21: Bus 24: Communications Department 30: Basic operation program 31: Web conference application program 32: Data 33: Server program 34: Client program 35: Near field communication 36: Proximity communication 50: Audio distribution section 51A: Microphone audio 52: Audio multiplexing section 53: Streaming audio 55: Packet multiplexing unit 56: Streaming audio 57: Packet removal unit 58: Audio multiplexing section 60: Packet removal unit 61: Audio multiplexing section 62: Do Not Distribute List 63: Streaming audio 100: Web conferencing system 151 :LAN communication device 152: Near field wireless communication device 161: Correlation calculation unit 161a: Variable delay section 161b: Delay amount setting unit 161c: sum of products 161d: Output processing section 162: Audio reduction section 162a: subtraction section 163: Participant List Creation Department 201: Illuminance sensor 202: Motion sensor

Claims

1. A voice transmission / reception system configured by communicatively connecting an information terminal and a server, The information terminal is A microphone that collects speech, a communication device for transmitting and receiving data to and from a server; An audio output device; a processor, The processor: When a new information terminal is found by searching the vicinity through proximity communication while already connected to at least one other information terminal through proximity communication, the device connects to the new information terminal and transmits information about the new information terminal to the other information terminal; receiving, from another information terminal in proximity connection, information about an information terminal with which the other information terminal has newly established proximity connection; creating a distribution prohibition list for the voice to be distributed that prohibits speech sounds collected by all other information terminals connected by short-range communication and new information terminals searched by those other information terminals from being included in the distributed voice, and transmitting the list to the server; receiving, from the server, a distribution voice that has been subjected to a volume reduction process based on the distribution prohibition list, and outputting the distribution voice to the audio output device; Voice transmission and reception system.

2. 2. The audio transmitting and receiving system according to claim 1, The server a packet removal unit; an audio multiplexing unit; inputting the collected voices transmitted from all of the information terminals to the packet removal unit; the packet removal unit removes packets of audio defined in the distribution prohibition list when generating audio to be distributed to a predetermined information terminal; the audio multiplexing unit adds the audio remaining after passing through the packet removal unit to generate a distribution audio, and distributes the generated distribution audio to the predetermined information terminal; Voice transmission and reception system.

Images