Chat terminal, chat system and control method of chat system

Abstract

The chat system is formed by connecting a chat terminal and a chat server through communication. The chat terminal collects a user sound emitted by a terminal user from a microphone connected to the chat terminal and another sound emitted by another person located in the vicinity of the terminal user, transmits the user sound to a chat server, and receives a distribution sound from the chat server. The correlation between the distributed sound and the sound of another person is obtained, the sound of another person included in the distributed sound is reduced, and the distributed sound in which the sound of another person is reduced is output from a sound output device connected to the chat terminal.

Description

Technical Field

[0001] The invention relates to a chat terminal, a chat system and a control method of the chat system. Background Art

[0002] For business purposes, people use chat systems integrated into web conference systems to conduct remote conferences by sending and receiving voice data between remote locations. In the past, each site used a chat terminal to chat with multiple participants in the site sharing screen images and voices, but in recent years, it has changed to a chat application running on personal computers and smartphones, and participants in the same site also use their own chat terminals to chat.

[0003] As for the sound processing of inter-site conference, there is a related record in Patent Document 1 (Japanese Patent Laid-Open No. 8-237627). Patent Document 1 discloses a multi-site video conferencing system, the purpose of which is to "prevent the speaker's own voice from being heard on the speaker's terminal (excerpt from the abstract)". According to this multi-site video conferencing system, the speech sound will not be distributed to the speaker's terminal and will not be output.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Laid-Open No. 8-237627 Summary of the invention

[0007] Technical problem to be solved by the invention

[0008] On the other hand, when participants in the same site also use their own chat terminals to participate in the meeting, the sound emitted by the participant (set as participant A) for the remote meeting (recorded as an inter-site meeting) is collected by the microphone in the chat terminal of participant A and sent to the chat server, and then distributed to the chat terminals of participants in other sites and the chat terminals of other participants in the same site (for example, participant B), and output from the speakers or headphones of the chat terminals. As a result, participant B will directly hear the voice of participant A (other people's voice) and will also hear the distributed sound output from the chat terminal of participant B. Hereinafter, in this specification, "other people's voice" refers to the sound of other people (people other than the user of the chat terminal) speaking at the scene, which is the sound heard directly. In addition, "distributed sound" refers to the sound output from the chat terminal.

[0009] The distributed sound has a delay caused by the chat server, so when two sounds overlap, the same sound is reproduced twice with a time difference, causing the problem of being very difficult to hear clearly.

[0010] Although Patent Document 1 can prevent the speaker's own voice from being heard on the speaker's terminal, it does not describe the problem of sound interference between other people's voices and the distributed voice when there are multiple chat terminals in the same site, and cannot solve the above problem.

[0011] The present invention is made in view of the above-mentioned situation, and its object is to solve the problem that when multiple participants participate in a chat from the same site using their own chat terminals, the voices of other nearby participants interfere with the distributed voice and become difficult to hear.

[0012] Technical means of solving problems

[0013] In order to solve the above-mentioned problems, the present invention has the structures described in the claims.

[0014] Effects of the Invention

[0015] According to the present invention, when multiple participants use their own chat terminals to participate in a chat from the same site, the problem of other participants' voices interfering with the distributed voice and making it difficult to hear is solved. Other purposes, features and effects will be described in the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is the structural diagram of the WEB conference system.

[0017] Figure 2A This is the hardware structure diagram of the WEB conference terminal.

[0018] Figure 2B This is the hardware structure diagram of the WEB conference terminal.

[0019] Figure 3 This is a functional block diagram of the WEB conference terminal according to the first embodiment.

[0020] Figure 4 It is a functional block diagram showing details of the correlation calculation unit.

[0021] Figure 5 This is a diagram for explaining a first example of a sound reduction process for the sound of others near the user included in the distributed sound.

[0022] Figure 6 This is a flowchart showing the flow of processing in the WEB conference system according to the first embodiment.

[0023] Figure 7 It is a diagram for explaining a second example of the sound reduction process of other people's voices.

[0024] Figure 8This is a flowchart showing the flow of processing of the WEB conference system including the second sound reduction processing of speech sound (speech sound of others distributed on the system).

[0025] Fig. 9 It is a diagram of the mesh network structure between the WEB conference terminals within a site.

[0026] Fig.10 This is a diagram for explaining the sound reduction processing of the speech sound based on the distribution prohibition list.

[0027] Fig.11 This is a flowchart showing the flow of processing of the web conference system corresponding to the third sound reduction processing of other person's voice.

[0028] Fig.12 It is a block diagram of a WEB conference system according to a third embodiment.

[0029] Fig.13 It is a block diagram of a WEB conference terminal implemented by an information processing device.

[0030] Fig.14 This is a functional block diagram of a WEB conference terminal according to a fourth embodiment.

[0031] Fig.15 This is a flowchart showing the flow of processing of a WEB conference system corresponding to a serverless WEB conference system. DETAILED DESCRIPTION

[0032] The following describes the embodiments of the present invention with reference to the accompanying drawings. The same reference numerals are used for the same structures and steps in all the drawings, and repeated descriptions are omitted.

[0033] The chat system of the present invention is a system for transmitting and receiving voice data between a plurality of chat terminals directly or via a chat server. The chat system can be applied to, for example, a work support system that transmits and receives voice data between chat terminals worn by workers working at a work site and between a terminal of a management center located at a location far from the work site.

[0034] In addition, the chat system of the present invention can be applied to a voice chat system, which sends and receives voice data between chat terminals worn by each player directly or via a chat server when multiple players form a team to play e-sports. Furthermore, it can also be applied to e-sports systems and game systems that integrate the voice chat system.

[0035] In the following description, a web conference system in which the chat system of the present invention is integrated is taken as an example for explanation. The present invention is expected to contribute to 8.2 of the Sustainable Development Goals (SDGs: Sustainable Development Goals) advocated by the United Nations (achieving higher levels of economic productivity through diversified operations, technological upgrades and innovations, including focusing on the development of high value-added and labor-intensive industries) because it is expected to achieve diversification and technological improvement of labor-intensive industries, for example.

[0036] [First embodiment of the present invention]

[0037] Reference Figures 1 to 8 , a first embodiment of the present invention is described.

[0038] Figure 1 It is the structural diagram of the WEB conference system.

[0039] Figure 1 In the WEB conference system 100, WEB conference terminals 3A to 3F (equivalent to chat terminals, hereinafter also referred to as "terminals") respectively set up at sites A, B, and C of the WEB conference and a WEB conference server 5 (equivalent to the chat server) are connected to each other via a network 4. Office AO is set up indoors in the office of site A of the WEB conference.

[0040] The following description uses site A as an example, but the description of site A is also applicable to sites B and C.

[0041] At site A, there are participants 2A, 2B, and 2C of a web conference. The web conference terminals used by the participants 2A, 2B, and 2C are terminals 3A, 3B, and 3C.

[0042] Even when the participants 2A, 2B, and 2C of the site A gather in the same room such as a conference room to conduct a WEB conference, the participants 2A, 2B, and 2C conduct the WEB conference using their own terminals 3A, 3B, and 3C, respectively.

[0043] Participants 2A, 2B, and 2C participate in a web conference from site A, and use terminals 3A, 3B, and 3C to access web conference server 5 via network 4, and receive the web conference services provided. For example, the image and speech sound of participant A (hereinafter referred to as "user sound") are collected by terminal A and sent to web conference server 5.

[0044] The web conference server 5 receives the images and sounds of all participants connected to the web conference service, generates distributed images and distributed sounds for the web conference, and distributes them to the terminals of each participant. For example, the speech sound (user voice) of participant A is distributed to the terminals of participants of sites B and C (terminal D, terminal E, terminal F) as part of the distributed sounds for the web conference.

[0045] However, the distributed sound outputted by the terminals 3B and 3C operated by other participants located near the participant A, in this example, the participants B and C, does not include the speech sound (user voice) of the participant A. Thus, the problem that the participants B and C hear the speech sound (other people's voice) of the participant A directly through the air in the office AO with a time difference and the speech sound (user voice) of the participant A included in the distributed sound outputted from the terminals 3B and 3C can be solved. This is one of the common features in each embodiment of the present invention.

[0046] Figure 2A , Figure 2B 2 is a diagram showing the hardware configuration of a WEB conference terminal. The WEB conference terminals 3A to 3F have the same configuration and are therefore referred to as terminals 3 when not distinguishing between the terminals.

[0047] The terminal 3 includes a camera 11, a microphone 12, a display 13, a sound output device 14, a communicator 15, a processor 16, a first storage device (RAM) 17, a second storage device (FROM) 18, an input device 19, and a sensor group 20, which are interconnected via a bus 21. The camera 11 and the display 13 are not essential to the terminal 3, and in this case, the WEB conference is conducted only by sound.

[0048] The processor 16 is constituted by, for example, a CPU.

[0049] RAM 17 is an example of a volatile memory.

[0050] The FROM 18 is an example of a nonvolatile memory and includes a basic operation program 30 , a WEB conference application 31 , and data 32 .

[0051] The camera 11 may be integrally formed with the terminal 3, or may be a camera connected via a USB terminal.

[0052] The microphone 12 collects the voice of the user of the terminal 3 (user voice) and the voice of other participants speaking in the web conference at the same site (other voices). In the case where there is only one microphone 12 and it has no directionality, both the user voice and other voices are collected. In the case where the sound collected by the microphone 12 is simply referred to without distinguishing between the user voice and other voices, it is called microphone collected sound.

[0053] Figure 2A A microphone 12 (user microphone) is shown in the figure, and an example is given of the case where the same microphone collects the user's voice and the voices of others, but separate microphones with directivities suitable for collecting each sound may also be provided. A microphone with directivity suitable for collecting the user's voice of the terminal 3 is called a user-specific microphone, and a microphone with directivity suitable for collecting surrounding sounds is called a shared microphone. A user-specific microphone is, for example, a microphone of a headset. In addition, a shared microphone is, for example, a microphone suitable for omnidirectional sound pickup placed on a conference room table. Figure 2B As shown, a dedicated microphone 12a for other people's voices (referred to as "dedicated microphone") can be connected to the bus 21, or a dedicated microphone 12b for other people's voices can be connected via Bluetooth (registered trademark) via a short-range wireless communication device 152. Other people's voices are sounds collected by a microphone (a user microphone or a dedicated microphone for collecting other people's voices) when the speaker is not speaking. If the user microphone is used as a dedicated microphone, there is no need to newly add a dedicated microphone 12b for other people's voices, so it is more preferred. The microphone 12 (user microphone) is set to a mute state when the speaker is not speaking (set to a state where the function of the microphone 12 itself is working, but the sound of the microphone 12 will not be played as a distributed sound), and the sound collected during this period is processed as not the speaker's own voice, that is, the sound of others.

[0054] The input device 19 is a keyboard or a touch sensor. In the case of a smartphone, a flat display (display 13 ) and a touch sensor are integrated, and the keyboard is operated by a basic operation program 30 .

[0055] The sound output device 14 is a device for outputting distributed sound, and may be a speaker, earphones, headphones, a headset, or a sound output terminal.

[0056] The communicator 15 includes a LAN communicator 151 for exchanging data such as images and sounds with the WEB conference server 5, and various communication methods and protocols such as a Bluetooth (registered trademark) short-range wireless communicator 152 for executing between terminals within the site.

[0057] The sensor group 20 includes, for example, an illumination sensor 201 , a motion sensor 202 , etc., to assist in the use of the terminal.

[0058] Figure 3 This is a functional block diagram of the WEB conference terminal according to the first embodiment.

[0059] The WEB conference terminal 3 includes a correlation calculation unit 161 and a sound reduction unit 162. The correlation calculation unit 161 and the sound reduction unit 162 are implemented by the processor 16 loading the basic operation program 30 and the WEB conference application 31 into the RAM 17 and executing them. The data 32 includes data required for executing the basic operation program 30 and the WEB conference application 31, and the data is appropriately read when the processor 16 executes the WEB conference application 31 for processing by each unit.

[0060] The image of the terminal user captured by the camera 11 is transmitted from the LAN communicator 151 to the WEB conference server 5 via the network 4 .

[0061] The LAN communicator 151 receives a distributed image and distributed audio for a web conference from the web conference server 5. The distributed image is displayed on the display 13. The distributed audio is supplied to the correlation calculation unit 161 and the audio reduction unit 162.

[0062] The user's voice and other people's voices (microphone-collected voices) collected by the microphone 12 are transmitted from the LAN communicator 151 to the WEB conference server 5 via the network 4 , and are supplied to the correlation calculation unit 161 and the voice reduction unit 162 .

[0063] The correlation calculation unit 161 receives the distribution sound and the user's voice and other people's voice from the microphone 12 as input, performs correlation calculation, obtains the delay amount and correlation amount between the two sounds, and sends the obtained information to the sound reduction unit 162 .

[0064] The sound reducing unit 162 refers to the delay amount and the correlation amount to subtract the user's voice and other people's voice from the distributed sound, thereby reducing the user's voice and other people's voice from the distributed sound and generating output sound for the terminal 3 .

[0065] The sound output device 14 outputs the output sound from the sound reducing unit 162. This prevents the microphone-collected sound (user voice and other people's voice) collected by the microphone 12 of the terminal from being output as the distributed sound from the sound output device 14, thereby reducing interference with other people's voices heard directly.

[0066] Figure 4 It is a functional block diagram showing details of the correlation calculation unit.

[0067] The correlation calculation unit 161 includes an adjustable delay unit 161 a , a delay amount setting unit 161 b , a product accumulation unit 161 c , and an output processing unit 161 d .

[0068] The sound collected by the microphone (user's voice and other people's voice) is input to the adjustable delay unit 161a. The delay time in the adjustable delay unit 161a is set by the delay amount setting unit 161b. The "speech sound" input to the adjustable delay unit 161a is the user's voice or the sound of others collected in silence.

[0069] The microphone collected sound (user's voice and other people's voice) and the distributed sound after delay processing are input to the multiplication and accumulation unit 161c, and a multiplication and accumulation operation is performed to obtain the correlation amount with the set delay time as a parameter. The multiplication and accumulation unit 161c adjusts the delay time to obtain the delay time with the maximum correlation amount, and uses them as the delay amount and correlation amount generated in the distribution.

[0070] The output processing unit 161d distributes the sound as described below. Figure 5 In the case of overlapping sounds as shown, the delay amount and the correlation amount are output. Figure 7 In the case of packet multiplexed voice as shown, correlation amounts are compared for each voice separated into packets, and a packet ID corresponding to the voice collected by the microphone (user voice and other people's voice) is output.

[0071] (First example of sound reduction processing)

[0072] Figure 5 This is a diagram for explaining a first example of a sound reduction process for the sound of others near the user included in the distributed sound.

[0073] The WEB conference server 5 includes a sound distribution unit 50 . The sound distribution unit 50 transmits a distribution sound 53 to the sound reducing unit 162 .

[0074] Each terminal—— Figure 5 The sound of terminal A is 51A, and the sound collected by other terminals is ( Figure 5 51E, 51D, 51F) are overlapped and added by the sound multiplexing unit 52 and distributed as the distribution sound 53.

[0075] The collected sound subtraction unit 162 a of the sound reduction unit 162 refers to the delay amount and the correlation amount obtained by the correlation calculation unit 161 , and subtracts the sound of others near the user (other person's sound) from the distributed sound 53 .

[0076] Figure 6 This is a flowchart showing the flow of processing in the WEB conference system according to the first embodiment.

[0077] When the terminal 3 starts the WEB conference application 31 (S10), the terminal 3 logs in to the WEB conference service provided by the WEB conference server 5 (S11) to participate in the WEB conference.

[0078] The terminal 3 captures a camera image using the camera 11 ( S12 ), and collects sound using the microphone 12 ( S13 ).

[0079] The terminal 3 transmits the camera image and the microphone collected sound collected by the microphone 12 of the terminal 3 to the WEB conference server 5 (S14). The WEB conference server 5 receives the distributed image and the distributed sound (S15).

[0080] When the microphone mute button of terminal 3 is pressed and terminal 3 is in mute ON state (S16: Yes), terminal 3 determines that the sound collected by the microphone is another person's voice because the terminal user has no intention of speaking.

[0081] By making the user microphone work even when it is muted, it can be used as a microphone for collecting other people's voices. In addition, a microphone for collecting other people's voices can be set up in addition to the user microphone. By configuring the microphone for collecting other people's voices near the conference speaker near the terminal user, the voices of others can be collected more accurately, and the accuracy of correlation calculation can be improved. In addition, when using the microphone for collecting other people's voices, the microphone sound collection of S13 is collected by the microphone for collecting other people's voices. After the terminal 3 becomes muted ON, the microphone sound collection of S13 can be performed by the microphone for collecting other people's voices.

[0082] When the terminal 3 is in the mute ON state ( S16 : Yes), the correlation calculation unit 161 performs correlation calculation between the distributed voice and the voice of another person, calculates the delay amount and the correlation amount, and outputs them to the sound reduction unit 162 .

[0083] Specifically, the sound reducing unit 162 subtracts the sound collected by the microphone (other people's voice) from the distributed sound (S17, S18), and outputs the distributed sound from which the other people's voice is subtracted from the sound output device 14 (S18, S19). The sound output from the sound output device 14 is called "amplified sound".

[0084] When the terminal 3 is in the mute OFF state (S16: No), the distributed sound is output directly as the amplified sound from the sound output device 14 (S19) because the distributed sound does not contain the user's voice (the user's voice has been removed by the conventional method).

[0085] If the user does not log out (S21: No), the process returns to step S12 to repeat the process. If the user logs out (S21: Yes), the web conference application is terminated (S22).

[0086] (Second example of sound reduction processing)

[0087] Figure 7 It is a diagram for explaining a second example of the sound reduction process of other people's voices.

[0088] Figure 7 In, with Figure 5 Similarly, the web conference server 5 is provided with a sound distribution unit 50 . The distribution sound 56 is transmitted from the sound distribution unit 50 to the sound reducing unit 162 .

[0089] WEB conference speech sound ( Figure 5 The sound 51A of terminal A and the sound collected from other terminals D, E, and F ( Figure 5 The packets 51D, 51E, and 51F are multiplexed by the packet multiplexing unit 55, and the sounds of each terminal are stored in packets with different numbers (hereinafter referred to as IDs) and distributed as distributed sound 56.

[0090] The group removing unit 57 of the sound reducing unit 162 separates the speech sound (sound of speech by others distributed on the system) from the distributed sound 56 based on the group ID obtained by the correlation calculation unit 161, and removes it by the group removing unit 57. The terminal sounds after removal are 51D, 51E, and 51F, which are multiplexed by the sound multiplexing unit 58 and sent to the sound output device 14.

[0091] Figure 8 This is a flowchart showing the flow of processing of the WEB conference system including the second sound reduction processing of speech sound (speech sound of others distributed on the system).

[0092] Sound reduction processing according to Figure 7 The sound reduction method shown. Figure 6 The steps with the same functions as those in the first flowchart described in the above description are marked with the same symbols and repeated descriptions are omitted.

[0093] Figure 8 Flowchart and Figure 6 The difference of the flowchart is step S30, in which the Figure 7 The method of group removal described above removes speech sounds (speech sounds of others distributed on the system).

[0094] As described above, the WEB conference terminal, WEB conference application and WEB conference system according to the first embodiment of the present invention have such a feature that, in a WEB conference conducted by participants using their respective WEB conference terminals, a WEB conference can be provided in which the speech sounds of the participants have little interference with the distributed sounds of the WEB conference and the speech sounds are easy to hear.

[0095] [Second embodiment of the present invention]

[0096] use Figures 9 to 11 A second embodiment of the present invention will be described.

[0097] Fig. 9 It is a diagram of the mesh network structure between the WEB conference terminals within a site. Fig. 9 The state shown is that terminal A, terminal B, and terminal C exist at site A and are connected to each other via proximity communication 36, and terminal H is added.

[0098] When terminal H enters site A, it searches the vicinity through proximity communication 36 and completes connection with connectable terminal C. Terminal C detects that terminal H has newly joined, and transmits the message to terminals A and B, and transmits information of terminals A and B to terminal H. As a result, terminals A, B, C, and H can obtain information of all terminals in site A, and generate a prohibited list of distributed sounds, which is used to prohibit speech sounds collected by terminals in the same site from being included in the distributed sounds.

[0099] Fig.10 This is a diagram for explaining the sound reduction processing of speech sound based on the distribution prohibition list, and shows a sound distribution unit of the WEB conference server.

[0100] The microphone collected sound 51A collected by the microphone 12 of the terminal A and the microphone collected sounds (51D, 51E, 51F) collected from other terminals are input to the packet removal unit 60. The data values ​​are added by the sound multiplexing unit 61 and distributed as the distribution sound 63.

[0101] The sound distribution unit 50 of the WEB conference server 5 receives the distribution prohibition list 62 from the terminal of the participant. For example, in the distribution prohibition list 62 of terminal B, the terminals A, C, and H located at the same site are recorded. In this way, the sound to be removed from the distributed sound of the terminal is specified for each terminal in the distribution prohibition list 62. The sound to be removed is defined by the terminal name (terminal A, C, H, etc.) to which the microphone that collected the sound is connected.

[0102] When generating the distributed audio to the terminal B, the group removing unit 60 removes the audio group set in the distribution prohibition list 62 for each terminal based on the distribution prohibition list 62 .

[0103] The audio multiplexing unit 61 adds the audio remaining after passing through the packet removal unit 60 (multiplexing process) to generate a distribution audio 63, and distributes it to the terminal B.

[0104] Fig.11 This is a flowchart showing the flow of processing of the web conference system corresponding to the third sound reduction processing of other person's voice.

[0105] Fig.11 In the flowchart of Figure 6 The steps with the same functions in the flowcharts described above are marked with the same symbols and repeated descriptions are omitted.

[0106] Fig.11 Flowchart and Figure 6 The difference of the first flowchart is steps S40, S41, S42. In S40, a new generation or update Fig. 9 In S41, the distribution prohibition list 62 is newly generated or updated. In S42, the distribution prohibition list 62 is sent to the WEB conference server 5.

[0107] In S15, the distributed image and distributed sound are received from the WEB conference server 5, but the received distributed sound is Fig.10 As stated, the sounds of other participants at the same site are not included.

[0108] As described above, the web conference terminal, web conference application, and web conference system according to the second embodiment of the present invention have the same features as the first embodiment, and have the feature of being able to reliably remove the voices of other participants at the same site.

[0109] [Third Embodiment of the Present Invention]

[0110] Reference Figure 12 to Figure 14 The third embodiment of the present invention will be described. This embodiment is an example in which a web conference can be performed even without the web conference server 5.

[0111] Fig.12 It is a block diagram of a WEB conference system according to a third embodiment.

[0112] Fig.12 and Figure 1 The difference of the WEB conference system is that it is a serverless system without a WEB conference server 5. For example, the camera image and microphone sound of the participant 2A captured and collected by the terminal 3A are distributed to the terminals of all participants participating in the WEB conference (terminal B to terminal F).

[0113] In addition, the terminal 3A receives images and sounds from all the terminals (terminal B to terminal F), and generates images and sounds of the WEB conference in the terminal.

[0114] Fig.13 This is a block diagram of a WEB conference terminal implemented by an information processing device, and is a WEB conference terminal corresponding to a serverless WEB conference. Fig.13 In the WEB conference terminal, for Figure 3 Modules with the same functions in the WEB conference terminal are marked with the same symbols and repeated descriptions are omitted.

[0115] Fig.13 In the terminal 3, the WEB conference application 31 contained in FROM18 includes a server program 33 and a client program 34. The server program 33 distributes the camera image and microphone sound collected by the terminal user to other terminals, and receives images and sounds from other terminals.

[0116] The client program 34 captures and collects the camera images and microphone sounds of the terminal user, and shares the camera images and microphone sounds of the terminal user with the server program 33, as well as the camera images and microphone sounds from other terminals.

[0117] The server program 33 generates images and sounds of the web conference, and outputs the images and sounds to the display 13 and the sound output device 14 via the client program 34. In addition, the server program 33 does not need to be installed on all terminals participating in the web conference, and the web conference can be implemented as long as it is installed on at least one terminal. In this case, the terminal where the server program 33 is installed exchanges images and sounds with the client programs 34 of other terminals via the communication unit 24.

[0118] Fig.14 This is a functional block diagram of a WEB conference terminal according to a fourth embodiment.

[0119] Fig.14 The terminal 3 of FIG. 2 further includes a participant list generating unit 163 that generates a participant list based on the communication result of the short-distance communication 35 performed by the short-distance wireless communication device 152 .

[0120] Fig.15 This is a flowchart showing the flow of processing of a WEB conference system corresponding to a serverless WEB conference system.

[0121] For Figure 6 In the flowchart showing the processing flow of the WEB conference system, the same steps are marked with the same symbols.

[0122] The program is started (S10). The flowchart showing the flow of processing of the WEB conference system is composed of a client process and a server process.

[0123] In the client process, a notification of joining the web conference is made (S50). The notification is made to the terminals of the candidate participants listed in the candidate participant list acquired in advance.

[0124] After the camera image is captured (S12) and the sound is collected by the microphone 12 (S13), the camera image and the microphone collected sound are shared with the server process.

[0125] Furthermore, in S51, the image and sound output by the server process are shared.

[0126] In S16, it is confirmed whether the sound collected by the shared microphone in S51 includes the sound of other participants located at the same site. When it is determined that there is sound of other people (S16: "Yes"), the correlation calculation unit 161 performs a correlation calculation on the output sound of the server process and the sound of other people emitted by other participants located at the same site (S17), outputs parameters indicating the delay amount and the correlation amount to the sound reduction unit 162, and the sound reduction unit 162 subtracts the sound of other people emitted by the participant (S18), and outputs the amplified sound (S19). In addition, the image shared in step S51 is displayed on the display 13 (S20).

[0127] In the server process, upon receiving notifications from each terminal ( S52 ), the participant list generation unit 163 newly generates or updates a list of participants who actually attend the conference based on the candidate participant list distributed in advance ( S53 ).

[0128] In S54, the camera image and the collected sound are shared with the client process, and then the camera image and sound from other terminals are received in S55. In S56, the output image of the WEB conference is obtained according to the camera images of all terminals.

[0129] In S57, it is confirmed whether there is a distribution prohibition list and whether the speech sound is included in the distribution prohibition list. If there is a distribution prohibition list and the speech sound is included in the distribution prohibition list (S57: "Yes"), the other person's voice is removed (S58). The distribution prohibition list is formed by adding a distribution prohibition item (flag) to the participant list. In this case, the list of distribution prohibition participants in the participant list is equivalent to the distribution prohibition list.

[0130] If there is no distribution prohibition list and the voice of another person is not included in the distribution prohibition list (S57: No), step S58 is skipped. Then, in step S59, an output voice is generated and shared with the client process.

[0131] The output image and output sound of the server process are equivalent to the distribution image and distribution sound of the WEB conference system existing in the server.

[0132] As described above, the web conference terminal, web conference application and web conference system according to the third embodiment of the present invention have the same features as the first and second embodiments, and can realize serverless web conferences. When a web conference is performed using a small number of terminals, it has an advantage in terms of cost.

[0133] [Fourth embodiment of the present invention]

[0134] When the participants of a web conference use noise reduction headphones (hereinafter referred to as NCH) to conduct a web conference, instead of reducing the speaker's speech sound included in the system sound output from the NCH, the actual (speaker's) speech sound emitted on the spot can be reduced by using noise reduction technology. However, in this case, external sounds other than the speaker's speech sound are also reduced, so there will be problems such as not being able to notice the call tone of the phone or other people's calls during the web conference. Therefore, by enabling the noise reduction function of the NCH only during the period when the actual speaker's speech sound exists, the actual speaker's speech sound can be reduced, and by disabling the noise reduction function during the period when the actual speaker's speech sound does not exist, without reducing the external sound, other external sounds can be recognized. Furthermore, by performing noise reduction on the external sound only for the speech sound of the system sound, only the speech sound can be reduced, and other external sounds can also be recognized during the period when the speech sound exists.

[0135] In addition, each embodiment has been described by taking a WEB conference as an example, but the method of the present invention is not limited to the WEB conference, and is also effective in a system in which a conversation is conducted between remote locations using an information terminal with participants nearby.

[0136] The embodiments of the present invention are described above, but the structure of the technology for realizing the present invention is not limited to the above-mentioned embodiments, and various modifications can be considered. For example, the above-mentioned embodiments are described in detail in order to explain the present invention in an easy-to-understand manner, and are not limited to all the structures described. In addition, a part of the structure of a certain embodiment can be replaced with the structure of other embodiments, and the structure of other embodiments can be added to the structure of a certain embodiment. All of these belong to the scope of the present invention. In addition, the numerical values ​​and messages appearing in the text and the figures are only examples, and the effects of the present invention will not be impaired even if different ones are used.

[0137] In addition, the programs described in each processing example may be independent programs, or a plurality of programs may constitute one application program. In addition, the order of each processing may be changed and executed.

[0138] Regarding the functions of the present invention described above, part or all of them can be implemented by hardware, for example, through integrated circuit design, etc. In addition, it can also be implemented by software by interpreting and executing action programs that implement various functions, such as microprocessor units, CPUs, etc. In addition, the scope of software implementation is not limited, and hardware and software can also be used at the same time. In addition, part or all of each function can also be implemented by a server. In addition, as long as the server can perform functions in collaboration with other components via communication, such as a local server, cloud server, edge server, network service, etc., the method can be arbitrary. Information such as programs, tables, files, etc. that implement various functions can be stored in recording devices such as memories, hard disks, SSDs (Solid State Drives), or recording media such as IC cards, SD cards, DVDs, etc., and can also be stored in devices on the communication network.

[0139] In addition, the control lines and information lines shown in the figure show the necessary parts for explanation, and do not necessarily show all the control lines and information lines on the product. In fact, it can be considered that almost all structures are connected to each other.

[0140] The above-mentioned embodiment includes the following aspects.

[0141] (Note 1)

[0142] A chat terminal, characterized by comprising:

[0143] microphone;

[0144] A communicator, which sends and receives data with the chat server;

[0145] a sound output device; and

[0146] processor, where

[0147] The microphone collects a user voice emitted by the terminal user and other person voices emitted by other persons located near the terminal user.

[0148] The communicator sends the user voice to the chat server and receives the distribution voice from the chat server.

[0149] The processor obtains the correlation between the distributed sound and the other person's sound,

[0150] reducing the other person's voice contained in the distributed voice,

[0151] The distributed sound with the other person's voice reduced is output to the sound output device.

[0152] (Note 2)

[0153] A chat terminal, characterized by comprising:

[0154] microphone;

[0155] A communicator, which sends and receives data with other chat terminals;

[0156] a sound output device; and

[0157] processor, where

[0158] The microphone collects a user voice emitted by the terminal user and other person voices emitted by other persons located near the terminal user.

[0159] The communicator transmits the user voice to the external device and the other chat terminal, and receives the distribution voice from the other chat terminal.

[0160] The processor obtains the correlation between the distributed sound and the other person's sound,

[0161] reducing the other person's voice contained in the distributed voice,

[0162] The distributed sound with the other person's voice reduced is output to the sound output device.

[0163] (Note 3)

[0164] A chat system, comprising a chat terminal and a chat server connected via communication, characterized in that:

[0165] The chat terminal includes:

[0166] microphone;

[0167] A communicator, which sends and receives data with the chat server;

[0168] a sound output device; and

[0169] processor, where

[0170] The microphone collects the user voice emitted by the terminal user and the other person voice emitted by other persons located near the terminal user.

[0171] The communicator sends the user voice to the chat server and receives the distribution voice from the chat server.

[0172] The processor obtains the correlation between the distributed sound and the other person's sound,

[0173] reducing the other person's voice contained in the distributed voice,

[0174] The distributed sound with the other person's voice reduced is output to the sound output device.

[0175] (Note 4)

[0176] A control method for a chat system, wherein the chat system is composed of a chat terminal and a chat server connected via communication, the method being characterized in that it comprises:

[0177] The step of collecting a user voice emitted by a terminal user and other person voices emitted by other persons located near the terminal user from a microphone connected to the chat terminal;

[0178] The step of sending the user voice to the chat server and receiving the distributed voice from the chat server;

[0179] A step of obtaining the correlation between the distributed sound and the other person's sound;

[0180] a step of reducing the other person's voice contained in the distributed voice; and

[0181] The step of outputting the distributed sound with the other person's voice reduced from a sound output device connected to the chat terminal.

[0182] Description of Reference Numerals

[0183] 2A: Participants

[0184] 2B: Participants

[0185] 2C: Participants

[0186] 3: Web conference terminal

[0187] 3A: Web conference terminal

[0188] 3B: WEB conference terminal

[0189] 3C: WEB conference terminal

[0190] 3D: WEB conference terminal

[0191] 3E: Web conference terminal

[0192] 3F: WEB conference terminal

[0193] 4: Network

[0194] 5: Web conference server

[0195] 11: Camera

[0196] 12: Microphone

[0197] 12a: Microphone for other people's voices

[0198] 12b: Microphone for other people's voices

[0199] 13: Display

[0200] 14: Sound output device

[0201] 15: Communicator

[0202] 16: Processor

[0203] 17: RAM

[0204] 19: Input Device

[0205] 20: Sensor Group

[0206] 21: Bus

[0207] 24: Ministry of Communications

[0208] 30: Basic action program

[0209] 31: Web conference application

[0210] 32: Data

[0211] 33: Server Program

[0212] 34: Client Program

[0213] 35: Near Field Communication

[0214] 36: Proximity communication

[0215] 50: Sound Distribution Department

[0216] 51A: Microphone to collect sound

[0217] 52: Sound multiplexing unit

[0218] 53: Distribute Sound

[0219] 55: Packet multiplexing unit

[0220] 56: Distribute Sound

[0221] 57: Group removal unit

[0222] 58: Sound multiplexing unit

[0223] 60: Group removal unit

[0224] 61: Audio multiplexing unit

[0225] 62: Distribution Ban List

[0226] 63: Distribute Sound

[0227] 100: Web conference system

[0228] 151: LAN communicator

[0229] 152: Short-range wireless communication device

[0230] 161: Correlation calculation unit

[0231] 161a: Adjustable delay section

[0232] 161b: Delay amount setting unit

[0233] 161c: Multiplication and accumulation

[0234] 161d: Output processing unit

[0235] 162: Sound reduction unit

[0236] 162a: Subtraction Department

[0237] 163: Participant list generation department

[0238] 201: Illuminance sensor

[0239] 202: Motion Sensor

Claims

1. A chat terminal, characterized in that: include: microphone; A communicator, which sends and receives data with the chat server; Sound output device; and processor, where The microphone collects a user voice emitted by the terminal user and other person voices emitted by other persons located near the terminal user. The communicator sends the user voice to the chat server and receives the distribution voice from the chat server. The processor obtains the correlation between the distributed sound and the other person's sound, reducing the other person's voice contained in the distributed voice, The distributed sound with the other person's voice reduced is output to the sound output device.

2. The chat terminal according to claim 1, characterized in that: Also includes: Camera; and Display, where The communicator also sends the image captured by the camera to the chat server, and also receives the distributed image from the chat server. The processor displays the distribution image on the display.

3. The chat terminal according to claim 1, characterized in that: Also includes a short-range wireless communicator, The short-range wireless communicator recognizes the presence of a nearby terminal, The processor generates a distribution prohibition list based on a communication result of the short-range wireless communication device, and outputs, from the sound output device, a sound obtained by reducing the sound of others listed in the distribution prohibition list.

4. A chat terminal, characterized in that: include: microphone; A communicator, which sends and receives data with other chat terminals; Sound output device; and processor, where The microphone collects a user voice emitted by the terminal user and other person voices emitted by other persons located near the terminal user. The communicator transmits the user voice to the external device and the other chat terminal, and receives the distribution voice from the other chat terminal. The processor obtains the correlation between the distributed sound and the other person's sound, reducing the other person's voice contained in the distributed voice, The distributed sound with the other person's voice reduced is output to the sound output device.

5. A chat system, comprising a chat terminal and a chat server connected via communication, characterized in that: The chat terminal includes: microphone; A communicator, which sends and receives data with the chat server; a sound output device; and processor, where The microphone collects a user voice emitted by the terminal user and other person voices emitted by other persons located near the terminal user. The communicator sends the user voice to the chat server and receives the distribution voice from the chat server. The processor obtains the correlation between the distributed sound and the other person's sound, reducing the other person's voice contained in the distributed voice, The distributed sound with the other person's voice reduced is output to the sound output device.

6. A control method for a chat system, wherein the chat system is composed of a chat terminal and a chat server connected via communication, the method comprising: The step of collecting a user voice emitted by a terminal user and other person voices emitted by other persons located near the terminal user from a microphone connected to the chat terminal; The step of sending the user voice to the chat server and receiving the distributed voice from the chat server; A step of obtaining the correlation between the distributed sound and the other person's sound; A step of reducing the other person's voice contained in the distributed voice; and The step of outputting the distributed sound with the other person's voice reduced from a sound output device connected to the chat terminal.

Images