Conference server and conference system

Abstract

To provide a conference server that restricts a transmission destination of a response voice signal in response to a command voice signal.SOLUTION: A conference server according to an embodiment includes: a processor that obtains a response voice signal corresponding to a command voice signal included in a first input voice signal transmitted from a first communication device participating in an online conference, and outputs a first output voice signal including a command exclusion voice signal obtained by excluding the command voice signal from the first input voice signal; and a communication interface that receives the first input voice signal transmitted from the first communication device, transmits the response voice signal to the first communication device, and transmits the first output voice signal to another communication device participating in the online conference.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] FIELD An embodiment of the present invention relates to a conference server and a conference system. [Background technology]

[0002] A method for providing a virtual assistant to assist users participating in an online conference is known, which allows users participating in the online conference to use the virtual assistant's support function by voice commands and share the response voice from the virtual assistant with other users participating in the online conference.

[0003] On the other hand, there are cases where you do not want to share your use of a virtual assistant with other users in an online meeting. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2016-533690 Summary of the Invention [Problem to be solved by the invention]

[0005] The problem to be solved by the present invention is to provide a conference server that limits the destinations of response voice signals in response to command voice signals. [Means for solving the problem]

[0006] A conference server according to one embodiment includes a processor that acquires a response audio signal corresponding to a command audio signal included in a first input audio signal transmitted from a first communication device participating in an online conference, and outputs a first output audio signal including a command-excluded audio signal obtained by excluding the command audio signal from the first input audio signal; and a communication interface that receives the first input audio signal transmitted from the first communication device, transmits the response audio signal to the first communication device, and transmits the first output audio signal to other communication devices participating in the online conference. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of a conference system according to an embodiment. [Figure 2] FIG. 2 is a block diagram showing an example of a circuit configuration of the conference server according to the embodiment. [Figure 3] FIG. 3 is a block diagram showing an example of a circuit configuration of a virtual assistant server according to an embodiment. [Figure 4] FIG. 4 is a block diagram illustrating an example of a circuit configuration of a user terminal according to the embodiment. [Figure 5] FIG. 5 is a schematic diagram showing an example of a first screen displayed by a user terminal that has selected the personal mode. [Figure 6] FIG. 6 is a schematic diagram showing an example of a second screen displayed by a user terminal that has selected the sharing mode. [Figure 7] FIG. 7 is a block diagram illustrating an example of the configuration of a processor of the conference server according to the embodiment. [Figure 8] FIG. 8 is a flowchart showing an example of the overall operation of the conference server according to the embodiment. [Figure 9] FIG. 9 is a flowchart showing an example of a speech recognition processing operation by the conference server according to the embodiment. [Figure 10] FIG. 10 is a flowchart illustrating an example of a voice synthesis processing operation by the conference server according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, a conference system according to an embodiment will be described with reference to the drawings. Note that the scale of each part in each drawing used in the following description of the embodiment has been changed as appropriate. Also, for the sake of explanation, the configuration of each drawing used in the following description of the embodiment has been omitted as appropriate.

[0009] FIG. 1 is a schematic diagram showing an example of a conference system according to an embodiment. The conference system is a system in which users in remote locations hold a conference via communication, such as an online conference. As an example, the conference system includes a conference server 100, a virtual assistant server 200, and a user terminal 300 (communication device). The conference server 100 can be communicably connected to the virtual assistant server 200 and the user terminal 300 via a network. Note that the number of conference servers 100, virtual assistant servers 200, and user terminals 300 in FIG. 1 is not limited to the number shown.

[0010] In the description of the embodiment, the conference system is assumed to be a video conference, but it may also be a conference between users via communication such as the Internet, such as a telephone conference. In addition, the description of the embodiment will proceed on the assumption that a user is participating in an online conference.

[0011] An example of the circuit configuration of the conference server 100 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the circuit configuration of the conference server 100 according to the embodiment.

[0012] The conference server 100 includes, for example, a processor 101 , a main memory device 102 , an auxiliary memory device 103 , a communication interface 104 , an input device 105 , an output device 106 , and an RTC (real-time clock) 107 .

[0013] The processor 101 corresponds to the central part of a computer that performs processes such as calculations and controls required for the operation of the conference server 100. The processor 101 controls each part to realize various functions of the conference server 100 based on programs such as system software, application software, and firmware stored in the auxiliary storage device 103, a voice recognition and command voice signal detection program P1, a virtual assistant calling program P2, or a voice synthesis program P3.

[0014] The voice recognition and command voice signal detection program P1, for example, causes the processor 101 to detect the waveform of an input voice signal received from the user terminal 300 and recognize it as a voice waveform. The voice recognition and command voice signal detection program P1 causes the processor 101 to analyze the recognized voice waveform and, if a specific voice signal (hereinafter referred to as a command voice signal) is included in the analysis result, to detect the command voice signal.

[0015] The virtual assistant call program P2 causes the processor 101 to, for example, operate so that when a command voice signal is detected, the processor 101 transmits the detection result to the virtual assistant server 200 and issues an instruction to start the virtual assistant.

[0016] The voice synthesis program P3 causes the processor 101 to perform synthesis processing of a call voice signal and an input voice signal, a call voice signal and a command-exclusion voice signal, or a call voice signal and a response voice signal. The operation of the processor 101 will be described in detail later.

[0017] The processor 101 may be, for example, a central processing unit (CPU), a micro processing unit (MPU), a system on a chip (SoC), a digital signal processor (DSP), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a programmable logic device (PLD), a field-programmable gate array (FPGA), etc. Alternatively, the processor 101 may be a combination of a plurality of these.

[0018] The main memory device 102 is, for example, a read-only memory (ROM) or a random-access memory (RAM). ROM is a non-volatile memory used exclusively for reading data, and stores data or various setting values ​​used by the processor 101 when performing various processes. RAM is also used for reading and writing data, and is used by the processor 101 as a so-called work area for temporarily storing data when performing various processes. In this embodiment, the main memory device 102 is assumed to be exclusively a RAM and is treated as a memory. The main memory device 102 temporarily stores an input voice signal when the processor 101 performs voice recognition processing on the input voice signal received from the user terminal 300. When the processor 101 detects a command voice signal, the main memory device 102 temporarily stores the command voice signal. The main memory device 102 temporarily stores a call voice signal, a command-exclusion voice signal, or a response voice signal to be synthesized by the processor 101.

[0019] The auxiliary storage device 103 is a non-transitory computer-readable storage medium of a computer centered around the processor 101. The auxiliary storage device 103 is, for example, an EEPROM (electric erasable programmable read-only memory) (registered trademark), an HDD (hard disk drive), or an SSD (solid state drive). The auxiliary storage device 103 stores data used by the processor 101 when performing various processes, data generated by the processes in the processor 101, various setting values, etc. For example, the auxiliary storage device 103 is a memory that stores various information, and stores a voice recognition and command voice signal detection program P1, a virtual assistant call program P2, a voice synthesis program P3, a call voice signal, an input voice signal, a command voice signal, a command exclusion voice signal, and a response voice signal.

[0020] The processor 101 acquires a call audio signal for each user terminal, synthesizes other audio signals with the call audio signal according to the mode and signal to generate an output audio signal, and outputs the output audio signal to the auxiliary storage device 103. In this embodiment, the selected mode is either the shared mode or the personal mode. The processor 101 controls the generation and output of an audio signal corresponding to the selection of the shared mode, and, for example, places the user terminals 300 of all users participating in the online conference in an audio sharing state. The audio sharing state refers to a state in which, after calling the virtual assistant, all users participating in the online conference can hear the audio uttered by the user to execute commands such as questions or instructions to the virtual assistant, as well as the virtual assistant's responses to the user's commands. The processor 101 also controls the generation and output of an audio signal corresponding to the selection of the personal mode, and, for example, places the user terminal 300 of a specific user participating in the online conference in an audio sharing restricted state with the user terminals 300 of other users. The voice sharing restricted state is a state in which, after calling a virtual assistant, the user who called the virtual assistant can hear the voice uttered to execute commands such as questions or instructions to the virtual assistant, as well as the virtual assistant's responses to the user's commands, but other users cannot hear them.

[0021] In addition to the above modes, for example, a mode may be provided in which a group is formed of at least two or more users and a virtual assistant from among all users participating in an online conference, and after a user in the group calls a virtual assistant, the group can hear the voice that the user utters to execute commands such as questions or instructions to the virtual assistant, and the virtual assistant's response to the user's commands.

[0022] Instead of the auxiliary storage device 103, or in addition to the auxiliary storage device 103, the conference server 100 may be provided with an interface into which a storage medium such as a removable optical disk, a memory card, or a USB (universal serial bus) memory can be inserted.

[0023] The programs stored in the auxiliary storage device 103 include programs for executing the processes described below. As an example, the conference server 100 is transferred to an administrator or the like of the conference server 100 with the programs stored in the auxiliary storage device 103. However, the conference server 100 may also be transferred to the administrator or the like without the programs stored in the auxiliary storage device 103. The programs for executing the processes described below may then be transferred separately to the administrator or the like and written into the auxiliary storage device 103 under the operation of the administrator or the like. The transfer of the programs in this case may be realized by recording them on a removable storage medium such as a magnetic disk, a magneto-optical disk, an optical disk, or a semiconductor memory, or by downloading them via a network or the like.

[0024] The communication interface 104 is an interface for communicating with the virtual assistant server 200 and the user terminal 300 via a network or the like. For example, the communication interface 104 receives an input voice signal transmitted from the user terminal 300. The communication interface 104 also receives a response voice signal transmitted from the virtual assistant server 200.

[0025] The input device 105 is, for example, a device that receives voice data such as a microphone, or a device that recognizes character data entered by an administrator who manages the conference server 100 using an input detection sheet that employs an electrostatic or pressure method.

[0026] The output device 106 is a display device that uses, for example, a liquid crystal display, an organic electroluminescence display, or the like, and displays voice, text, images, and the like in response to signals input from the input device 105 .

[0027] The RTC 107 is a clock or a circuit with a built-in clock function.

[0028] An example of the circuit configuration of the virtual assistant server 200 will be described with reference to Fig. 3. Fig. 3 is a block diagram showing an example of the circuit configuration of the virtual assistant server 200 according to the embodiment.

[0029] The virtual assistant server 200 includes, for example, a processor 201, a main memory device 202, an auxiliary memory device 203, a communication interface 204, an input device 205, an output device 206, and an RTC (real-time clock) 207.

[0030] The processor 201 corresponds to the central part of the computer that performs processing such as calculations and controls necessary for the operation of the virtual assistant server 200. The processor 201 controls each part to realize various functions of the virtual assistant server 200 based on programs such as system software, application software, firmware, etc. stored in the auxiliary storage device 203 and the response voice signal generation program P4.

[0031] The response voice signal generating program P4 causes the processor 201 to operate to generate a response voice corresponding to a command voice transmitted from the communication interface 104, for example.

[0032] The processor 201 may be, for example, a central processing unit (CPU), a micro processing unit (MPU), a system on a chip (SoC), a digital signal processor (DSP), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a programmable logic device (PLD), a field-programmable gate array (FPGA), etc. Alternatively, the processor 201 may be a combination of a plurality of these.

[0033] The main memory device 202 is, for example, a read-only memory (ROM) or a random-access memory (RAM). ROM is a non-volatile memory used exclusively for reading data, and stores data or various setting values ​​used by the processor 201 when performing various processes. RAM is also a memory used for reading and writing data, and is used by the processor 201 as a so-called work area for temporarily storing data when performing various processes. In this embodiment, the main memory device 202 is assumed to be exclusively a RAM, and is treated as a memory. The main memory device 202 temporarily stores command voice signals received from the conference server 100, and temporarily stores response voice signals generated by the processor 201 in response to command voice signals.

[0034] The auxiliary storage device 203 is a non-transitory computer-readable storage medium of a computer centered around the processor 201. The auxiliary storage device 203 is, for example, an EEPROM (electric erasable programmable read-only memory) (registered trademark), an HDD (hard disk drive), or an SSD (solid state drive). The auxiliary storage device 203 stores data used by the processor 201 when performing various processes, data generated by the processes in the processor 201, various setting values, etc. For example, the auxiliary storage device 203 stores the response voice signal generating program P4, command voice signals, and response voice signals.

[0035] The processor 201 outputs a response voice signal corresponding to a command voice signal received from the conference server 100 to the auxiliary storage device 203, and the auxiliary storage device 203 stores the command voice signal and the response voice signal. An example of a method by which the virtual assistant server 200 generates a response voice signal corresponding to a command voice signal will be described. The processor 201 converts the voice waveform data of the command voice signal transmitted from the conference server into text data, for example, from data stored in the virtual assistant server 200 as an acoustic model. Next, the processor 201 converts the text data into a statistically most appropriate character string from data stored in the virtual assistant server 200 as a language model. The processor 201 analyzes the syntax of the character string based on the part of speech, etc., analyzes the user's intention using a neural network or machine learning, and generates a response voice signal corresponding to the command voice signal. Various methods are generally known for generating a response voice signal from the command voice signal of the virtual assistant server 200, and are not limited to the method described above. A virtual assistant is an AI engine that provides support functions in response to a call command from a user. Virtual assistants are constructed using statistical dialogue techniques that use rule-based dialogue technology and deep learning technology.

[0036] The program stored in the auxiliary storage device 203 includes a program for executing the processing described below. As an example, the virtual assistant server 200 is transferred to the administrator of the virtual assistant server 200 with the program stored in the auxiliary storage device 203. However, the virtual assistant server 200 may be transferred to the administrator without the program being stored in the auxiliary storage device 203. Then, the program for executing the processing described below may be transferred separately to the administrator and written to the auxiliary storage device 203 under the operation of the administrator. The transfer of the program at this time can be realized, for example, by recording it on a removable storage medium such as a magnetic disk, magneto-optical disk, optical disk, or semiconductor memory, or by downloading it via a network.

[0037] The communication interface 204 is an interface through which the virtual assistant server 200 communicates with the conference server 100 and the user terminal 300 via a network or the like. For example, the communication interface 204 receives a command voice signal transmitted from the conference server 100. The communication interface 204 also transmits a response voice signal generated in response to the command voice signal to the conference server 100.

[0038] The configurations of the input device 205, output device 206 and RTC 207, which are the circuit configurations of the virtual assistant server 200 according to the embodiment, are similar to the circuit configurations of the conference server 100 according to the embodiment, and therefore will not be described.

[0039] An example of the circuit configuration of the user terminal 300 will be described with reference to Fig. 4. Fig. 4 is a block diagram showing an example of the circuit configuration of the user terminal 300 according to the embodiment.

[0040] The user terminal 300 may be an electronic device such as a computer, and may be, for example, a television set (including Internet television), a PC (Personal Computer), a mobile terminal (for example, a tablet, smartphone, laptop, feature phone, portable game console, digital music player, e-book reader, etc.), a VR (Virtual Reality) terminal, an AR (Augmented Reality) terminal, etc., but is not limited to these.

[0041] The user terminal 300 includes, for example, a processor 301, a main memory device 302, an auxiliary memory device 303, a communication interface 304, an input device 305, an output device 306, and an RTC (real time clock).

[0042] The processor 301 corresponds to the central part of a computer that performs processes such as calculations and controls required for the operation of the user terminal 300. The processor 301 controls each part to realize various functions of the user terminal 300 based on programs such as system software, application software, and firmware stored in the auxiliary storage device 303, and the mode change operation reception program P5.

[0043] The processor 301 may be, for example, a central processing unit (CPU), a micro processing unit (MPU), a system on a chip (SoC), a digital signal processor (DSP), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a programmable logic device (PLD), a field-programmable gate array (FPGA), etc. Alternatively, the processor 301 may be a combination of a plurality of these.

[0044] The mode change operation receiving program P5, for example, when it detects that a button for switching between personal mode and shared mode displayed on the screen of the user terminal 300 has been pressed, causes the processor 301 to change the current mode.

[0045] The main memory device 202 is, for example, a read-only memory (ROM) or a random-access memory (RAM). ROM is a non-volatile memory used exclusively for reading data, and stores data or various setting values ​​used by the processor 201 when performing various processes. RAM is also used for reading and writing data, and is used by the processor 201 as a so-called work area for temporarily storing data when performing various processes. In this embodiment, the main memory device 202 is assumed to be exclusively a RAM and is treated as a memory. The main memory device 302 temporarily stores an input voice signal, which is the voice of a user using the user terminal 300. The main memory device 302 also temporarily stores the current mode selected by the user, and temporarily stores a call voice signal that has been synthesized according to the mode and transmitted from the conference server 100.

[0046] The auxiliary storage device 303 is a non-transitory computer-readable storage medium of a computer centered around the processor 301. The auxiliary storage device 303 is, for example, an EEPROM (electric erasable programmable read-only memory) (registered trademark), an HDD (hard disk drive), or an SSD (solid state drive). The auxiliary storage device 303 stores data used by the processor 301 when performing various processes, data generated by the processes in the processor 301, various setting values, etc. For example, the auxiliary storage device 303 stores the mode change operation acceptance program P5, input voice signals, and call voice signals.

[0047] The processor 301 acquires the voice uttered by the user with the input device 305, outputs the acquired voice to the auxiliary storage device 303, and stores the voice as an input voice signal in the auxiliary storage device 303. The processor 301 also outputs a call voice signal synthesized according to the mode transmitted by the conference server 100, and stores the signal in the auxiliary storage device 303.

[0048] The programs stored in the auxiliary storage device 303 include programs for executing the processes described below. As an example, the user terminal 300 is transferred to an administrator or the like of the user terminal 300 with the programs stored in the auxiliary storage device 303. However, the user terminal 300 may also be transferred to the administrator or the like without the programs stored in the auxiliary storage device 303. The programs for executing the processes described below may then be transferred separately to the administrator or the like and written into the auxiliary storage device 303 under the operation of the administrator or the like. The transfer of the programs in this case may be realized, for example, by recording them on a removable storage medium such as a magnetic disk, a magneto-optical disk, an optical disk, or a semiconductor memory, or by downloading them via a network or the like.

[0049] The communication interface 304 is an interface through which the user terminal 300 communicates with the conference server 100, the virtual assistant server 200, etc. via a network or the like. For example, the communication interface 304 transmits an audio signal input to the input device 305 to the conference server 100 as an input audio signal. The communication interface 304 also receives a call audio signal synthesized by the conference server 100 according to the current mode of the user terminal 300.

[0050] The circuit configuration of the input device 305, output device 306, and RTC 307 of the user terminal 300 according to the embodiment is the same as the circuit configuration of the conference server 100 according to the embodiment, and therefore description thereof will be omitted.

[0051] The screen display when the user terminal 300 is in personal mode and when another user is using personal mode will be described with reference to Figs. 5 and 6. The output device 306 of the user terminal 300 that has joined an online conference via a network displays a screen for the online conference, etc. Fig. 5 is a schematic diagram showing an example of a first screen displayed by the user terminal 300 that has selected personal mode. Fig. 6 is a schematic diagram showing an example of a second screen displayed by the user terminal 300 that has selected shared mode.

[0052] The first screen shown in FIG. 5 includes areas E11 and E12. For example, area E11 includes multiple buttons. The buttons in area E11 accept setting changes for the user terminal 300 that participates in a specific online conference and displays the first screen. The buttons in area E11 also indicate the status of the accepted settings. Area E12 includes a status display and a user display. The status display in area E12 displays the status of other user terminals 300 participating in the specific online conference. The user display in area E12 displays images provided by other user terminals 300 participating in the specific online conference.

[0053] For example, button B1 included on the first screen accepts switching between personal mode and shared mode. Button B1 without a slash mark indicates that personal mode has been selected on the user terminal 300 displaying the first screen. When the first screen is displayed, the output audio signal output on the other user terminal 300 that will be the conference partner (for example, the conference partner displayed in a four-part split) displayed on the first screen is a signal from which at least a portion of the command audio signal and the response audio signal have been excluded. The command audio signal includes a call audio signal that calls the virtual assistant and a command execution audio signal that causes the virtual assistant to execute various commands and output a response audio signal, and the output audio signal output on the other user terminal 300 that will be the conference partner is a signal from which the command execution audio signal has been excluded.

[0054] The mode indicator D1 included in the first screen indicates personal mode or shared mode in the other user terminal 300 who will be the conference partner. The mode indicator D1 including a slash symbol indicates that shared mode is selected in the other user terminal 300 who will be the conference partner. When the first screen is displayed, the other user terminal 300 who will be the conference partner is in shared mode, and the user terminal 300 displaying the first screen is in personal mode. Therefore, the user of the user terminal 300 displaying the first screen outputs audio from the other user terminal 300 who will be the conference partner. This allows the user of the user terminal 300 displaying the first screen to hear the audio from the other user terminal 300 who will be the conference partner. Furthermore, the other user terminal 300 displayed on the first screen outputs audio from the user terminal 300 displaying the first screen, excluding command execution audio signals and response audio signals. This allows the user of the other user terminal 300 to hear audio from the user terminal 300 displaying the first screen, excluding command execution audio signals and response audio signals.

[0055] The second screen shown in FIG. 6 includes areas E21 and E22. For example, area E21 includes multiple buttons. The buttons in area E21 accept setting changes for the user terminal 300 that participates in a specific online conference and displays the second screen. The buttons in area E21 also indicate the status of the accepted settings. Area E22 includes a status display and a user display. The status display in area E22 displays the status of other user terminals 300 participating in the specific online conference. The user display in area E22 displays images provided by other user terminals 300 participating in the specific online conference.

[0056] For example, button B2 included on the second screen accepts switching between personal mode and shared mode. Button B2 including a slash mark indicates that shared mode is selected on the user terminal 300 displaying the second screen. When the second screen is displayed, the output audio signal output from the other user terminal 300 that is the conference partner (e.g., the conference partner displayed in a four-part split screen) displayed on the second screen will be all audio including command audio signals and response audio signals.

[0057] The mode indicator D2 included in the second screen indicates personal mode or shared mode in the other user terminal 300 that is the conference partner. The mode indicator D2 without a slash mark indicates that personal mode has been selected in the other user terminal 300 that is the conference partner. When the second screen is displayed, the user terminal 300 displaying the second screen outputs audio from the other user terminal 300 that corresponds to the mode indicator D2 without a slash mark, excluding command execution audio signals and response audio signals. In addition, the user terminal 300 displaying the second screen outputs all audio from the other user terminals 300 that correspond to the buttons that do not include a slash mark. This allows the user of the user terminal 300 displaying the second screen to hear audio from the other user terminals 300 that correspond to the mode indicator D2 without a slash mark, excluding command execution audio signals and response audio signals, and to hear all audio from the other user terminals 300 that correspond to the buttons that do not include a slash mark.

[0058] An example of the configuration of the processor 101 of the conference server 100 will be described with reference to Fig. 7. Fig. 7 is a block diagram showing an example of the configuration of the processor 101 of the conference server 100 according to the embodiment.

[0059] The processor 101 includes a voice recognition processing unit 108 and a voice synthesis processing unit 109. The processor 101 executes a voice recognition and command voice signal detection program P1, a virtual assistant call program P2, and a voice synthesis program P3 stored in the auxiliary storage device 103, thereby realizing the functions of the respective units of the voice recognition processing unit 108 and the voice synthesis processing unit 109. Note that the voice recognition processing unit 108 and the voice synthesis processing unit 109 may be realized by hardware such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field-Programmable Gate Array) that has the same functions as the processor 101 executing the programs.

[0060] The voice recognition processing unit 108 performs voice recognition processing at any timing during an online conference. For example, the voice recognition processing unit 108 acquires a call voice signal stored in the auxiliary storage device 103, initializes the call voice signal, detects an input voice signal from another user terminal 300, executes voice recognition processing, and calls a virtual assistant if it detects a command voice signal.

[0061] Furthermore, the voice synthesis processing unit 109 executes voice synthesis processing. After executing the voice recognition processing, for example, if the input terminal of the command voice signal is in personal mode, the voice synthesis processing unit 109 synthesizes the command exclusion voice signal and the call voice signal, and synthesizes the response voice signal and the call voice signal corresponding to the command voice signal. If the input terminal of the command voice signal is in shared mode, the voice synthesis processing unit 109 synthesizes the input voice signal and the call voice signal depending on whether the input voice signal is input from the terminal, and synthesizes the response voice signal corresponding to the command voice signal. The voice synthesis processing unit 109 stores the call voice signal in the auxiliary storage device 103, and instructs the user terminal 300 to transmit the call voice signal.

[0062] The operation of the conference server 100 according to the embodiment will be described below with reference to Figures 8, 9, and 10. Note that the content of the processing in the following operation description is an example, and various processing that can achieve similar effects can be used as appropriate. Figure 8 is a flowchart showing an example of the overall operation of the conference server 100 according to the embodiment. The processor 101 executes this processing based on a program stored in the auxiliary storage device 103 or the like. Note that unless otherwise specified, the processing of the processor 101 transitions to Act(N+1) after ActN (N is a natural number).

[0063] For example, the processor 301 of the first user terminal 300 issues an instruction to request the setting up of an online conference based on the request for setting up an online conference received via the input device 305. The communication interface 304 transmits the request for setting up an online conference to the conference server 100. The communication interface 104 of the conference server 100 receives the request for setting up an online conference. The processor 101 generates participation information and invitation information for the online conference based on the request for setting up the online conference. The communication interface 104 transmits the participation information and invitation information to the first user terminal 300. The communication interface 304 of the first user terminal 300 receives the participation information and invitation information, and the auxiliary storage device 303 stores the participation information and invitation information. The participation information and invitation information include access information for the online conference on the web.

[0064] The first user terminal 300 requests participation in the online conference based on the participation information, the conference server 100 permits the first user terminal 300 to participate in the online conference based on the participation information, and the first user terminal 300 participates in the online conference via the communication network.

[0065] Furthermore, the first user terminal 300 transmits invitation information to the second user terminal 300. The second user terminal 300 requests participation in the online conference based on the invitation information, and the conference server 100 inquires of the first user terminal 300 about whether the second user terminal 300 can participate in the online conference based on the invitation information. Upon receiving participation approval from the first user terminal 300, the conference server 100 permits the second user terminal 300 to participate in the online conference based on the invitation information, and the second user terminal 300 participates in the online conference via the communication network. As described above, the online conference is conducted through communication connections between the conference server 100, the first user terminal 300 (first communication device), and the second user terminal 300 (second communication device).

[0066] The conference server 100 performs the processing shown in FIG. 8 at any timing during an online conference. In Act 1, the processor 101 (voice recognition processing unit 108) acquires, for example, a call voice signal stored in the auxiliary storage device 103 and initializes the call voice signal. The voice recognition processing unit 108 detects an input voice signal received from the user terminal 300. The voice recognition processing unit 108 performs voice recognition processing on the input voice signal. If the voice recognition processing unit 108 detects a command voice signal after performing the voice recognition processing, it calls a virtual assistant. A detailed description of the voice recognition processing will be given later.

[0067] In Act 2, after performing the voice recognition process, the processor 101 (voice synthesis processing unit 109) synthesizes the command excluding voice signal and the call voice signal, for example, when the input terminal of the command voice signal is in personal mode. The voice synthesis processing unit 109 synthesizes the call voice signal for the terminal that input the command voice signal with a response voice signal and a call voice signal corresponding to the command voice signal. On the other hand, the voice synthesis processing unit 109 synthesizes the command excluding voice signal and the call voice signal with the call voice signal for a terminal other than the terminal that input the command voice signal. When the input terminal of the command voice signal is in shared mode, the voice synthesis processing unit 109 synthesizes the input voice signal and the call voice signal depending on whether the input voice signal is the terminal that input the input voice signal. Thereafter, the voice synthesis processing unit 109 synthesizes the response voice signal corresponding to the command voice signal. The voice synthesis processing unit 109 stores the call voice signal for each user terminal 300 in the auxiliary storage device 103. The voice synthesis processing unit 109 acquires a call voice signal for the user terminal 300 from the auxiliary storage device 103, and instructs the user terminal 300 to transmit the call voice signal. Details of the voice synthesis processing will be described later.

[0068] 9 is a flowchart showing an example of the speech recognition processing operation by the conference server 100 according to the embodiment. That is, Fig. 9 is a flowchart for explaining in detail the speech recognition processing of Act 1 shown in Fig. 8.

[0069] The conference server 100 starts the processing shown in FIG. 9. The processor 101 starts the processing at any timing, for example, when it detects that an online conference has started. The processor 101 stores the call voice signal stored in the auxiliary storage device 103 in the main storage device 102 and acquires it (Act 11). The processor 101 initializes the call voice signal stored in the main storage device 102 (Act 12). The processor 101 can perform initialization at any timing, and can also initialize the call voice signal after transmitting it to the user terminal 300, for example. The processor 101 waits for reception of an input voice signal from the user terminal 300, and when it detects that it has received an input voice signal (first input voice signal) transmitted from the communication interface 304 of the user terminal 300 (first communication device) (Act 13), it proceeds to the next processing. After performing the processing of Act 13, the processor 101 performs speech recognition processing on the input voice signal. The speech recognition process of the processor 101, for example, converts the input speech signal into a language by comparing the waveform of the input speech signal with the waveform of language data stored in the auxiliary storage device 103 or language data searched while connected to the Internet, and converts the language into an appropriate character string, etc. There are various other methods for speech recognition, and the method is not limited to the above, and other commonly known methods may also be used.

[0070] After performing the voice recognition process in Act 14, the processor 101 detects a specific command voice signal contained in the text data or waveform data obtained from the result of the voice recognition process (Act 15). When the processor 101 detects the command voice signal (Act 15, YES), it calls the virtual assistant from the virtual assistant server 200 based on the call voice signal contained in the command voice signal. The call voice signal calls the virtual assistant, and is, for example, the name of the virtual assistant or a combination of an interjection and the name of the virtual assistant. For example, when the command voice signal is detected, the processor 101 sends a virtual assistant request to the virtual assistant server 200 via the communication interface 104. The processor 201 receives the virtual assistant request via the communication interface 104 and starts the virtual assistant corresponding to the response voice signal generation program P4.

[0071] In addition, the call of the virtual assistant may be made in the conference server 100, not in the virtual assistant server 200. In other words, by implementing the functions of the virtual assistant server 200 in the conference server 100, the above-described calling means can be realized.

[0072] 10 is a flowchart showing an example of the voice synthesis processing operation by the conference server 100 according to the embodiment. That is, Fig. 10 is a flowchart for explaining in detail the voice synthesis processing in Act 2 shown in Fig. 8.

[0073] The communication interface 104 receives the current mode stored in the main storage device 302 of the user terminal 300 that input the command voice signal S10. The auxiliary storage device 103 (memory) stores the received current mode. The processor 101 acquires the current mode stored in the auxiliary storage device 103 (Act 21). The processor 101 may acquire the current mode of the user terminal 300 every time the mode of the user terminal 300 is changed, or the processor 101 may acquire the current mode selectively.

[0074] The processor 101 acquires the current mode specified by the user terminal 300, i.e., personal mode or shared mode. For example, based on the read result, the processor 101 detects that the current mode of the user terminal 300 (first communication device) that inputs the input voice signal S11 (first input voice signal) including the command voice signal S10 is the shared mode (second mode) (Act 21, NO). Furthermore, when the processor 101 detects that the user terminal 300 is the input terminal of the input voice signal S11 including the command voice signal S10 (Act 22, YES), it performs synthesis processing on the response voice signal S31 and the call voice signal S22 corresponding to the command voice signal S10 included in the input voice signal S11, and generates an output voice signal S42 (second output voice signal) including the response voice signal S31 and the call voice signal S22 (Act 26). The call voice signal S22 is a voice signal sent from another user terminal 300 excluding the input voice signal S11. The processor 101 outputs the output audio signal S42 to the auxiliary storage device 103, and the auxiliary storage device 103 stores the output audio signal S42 (Act 28).

[0075] When the processor 101 detects that the current mode of the terminal that inputs the command voice signal S10 is the shared mode (Act 21, NO), and detects that the terminal is a terminal other than the input terminal of the input voice signal S11 including the command voice signal S10 (Act 22, NO), the processor 101 synthesizes the input voice signal S11 and the call voice signal S21 to generate an output voice signal S41 including the input voice signal S11 and the call voice signal S21 (Act 23). Furthermore, the processor 101 synthesizes the response voice signal S31 and the call voice signal S22 to generate an output voice signal S22 including the response voice signal S31 and the call voice signal S22 (Act 26). The call voice signals S21 and S22 are voice signals sent from the other user terminal 300 other than the input voice signal S11, and the call voice signal S22 is a voice signal generated at a timing later than the call voice signal S21. The processor 101 outputs the output audio signals S41 and S42 to the auxiliary storage device 103, and the auxiliary storage device 103 stores the output audio signals S41 and S42 (Act 28).

[0076] The processor 101 acquires the current mode specified by the user terminal 300, i.e., personal mode or shared mode. For example, the processor 101 detects that the current mode of the terminal that input the command voice signal S10 is the personal mode (first mode) (Act 21, YES). The processor 101 synthesizes a command-excluded voice signal S12, which is obtained by excluding the command voice signal S10 from the input voice signal S11, and a call voice signal S21, to generate an output voice signal S43 (first output voice signal) including the command-excluded voice signal S12 and the call voice signal S21 (Act 24). When the processor 101 detects that the terminal is the one that input the command voice signal S10 (Act 25, YES), the processor 101 synthesizes a response voice signal S31 and a call voice signal S22 according to the command voice signal S10, to generate an output voice signal S42 including the response voice signal S31 and the call voice signal S22 (Act 26). The processor 101 outputs the output audio signal S42 to the auxiliary storage device 103, and the auxiliary storage device 103 stores the output audio signal S42 (Act 28).

[0077] When the processor 101 detects that the terminal is other than the terminal that input the command voice signal S10 (Act 25, NO), it synthesizes a blank voice signal S32 that replaces the response voice signal S31 corresponding to the virtual assistant's command voice signal S10 and the call voice signal S22, and generates an output voice signal S44 including the blank voice signal S32 and the call voice signal S22 (Act 27). The processor 101 outputs the output voice signals S43 and S44 to the auxiliary storage device 103, and the auxiliary storage device 103 stores the output voice signals S43 and S44 (Act 28).

[0078] In addition, the response voice signal S31 in the processing of Act 26 is obtained by the communication interface 104 transmitting a command voice signal S10 including a question or instruction to the virtual assistant to the virtual assistant server 200, the processor 201 generating a response voice signal 31 according to the command voice signal S10, and the communication interface 104 receiving the response voice signal 31.

[0079] When the processor 101 detects that the terminal that input the command voice signal S10 is in personal mode and is a terminal other than the terminal that input the command voice signal S10, it acquires the output voice signals S43 and S44 stored in the auxiliary storage device 103 (Act 29). The processor 101 instructs the communication interface 104 to transmit the output voice signals S43 and S44 to the communication interface 304 of the other terminal participating in the online conference (Act 30), and the communication interface 104 transmits the output voice signals S43 and S44 to the specified user terminal 300.

[0080] When the processor 101 detects that the user terminal 300 that input the command voice signal S10 is in personal mode and is the terminal that input the command voice signal S10, it acquires the output voice signal S42 stored in the auxiliary storage device 103 (Act 29). The processor 101 instructs the communication interface 104 to transmit the output voice signal S42 to the communication interface 304 of the terminal participating in the online conference (Act 30), and the communication interface 104 transmits the output voice signal S42 to the specified user terminal 300.

[0081] When the processor 101 detects that the terminal that input the command voice signal S10 is in the sharing mode and is a terminal other than the terminal that input the command voice signal S10, the processor 101 acquires the output voice signals S41 and S42 stored in the auxiliary storage device 103 (Act 29). The processor 101 instructs the communication interface 104 to transmit the output voice signals S41 and S42 to the communication interface 304 of the other terminal participating in the online conference (Act 30), and the communication interface 104 transmits the output voice signals S41 and S42 to the specified user terminal 300.

[0082] When the processor 101 detects that the user terminal 300 that input the command voice signal S10 is in the sharing mode and is the terminal that input the command voice signal S10, it acquires the output voice signal S42 stored in the auxiliary storage device 103 (Act 29). The processor 101 instructs the communication interface 104 to transmit the output voice signal S42 to the communication interface 304 of the terminal participating in the online conference (Act 30), and the communication interface 104 transmits the output voice signal S42 to the specified user terminal 300.

[0083] When the terminal that inputs the command voice signal S10 is in personal mode, the other user terminals 300 other than the terminal that input the command voice signal S10 output the output voice signals S43 and S44 received via the communication interface 304 as voice on the output device 306. When the terminal that inputs the command voice signal S10 is in shared mode, the other user terminals 300 other than the terminal that input the command voice signal S10 output the output voice signals S41 and S42 received via the communication interface 304 as voice on the output device 306. When there are multiple other user terminals 300 participating in the online conference, the above operation is performed for each user terminal 300.

[0084] For example, the communication interface 104 receives an input voice signal S51 (second input voice signal) transmitted from the communication interface 304 of a user terminal 300 (second communication device) participating in an online conference. The processor 101 synthesizes the command-excluded voice signal S12 and the call voice signal S21 (including the input voice signal S51) based on the designation of the personal mode, to generate an output voice signal S43. The processor 101 also synthesizes the response voice signal S31 and the call voice signal S22 (including the input voice signal S11 and the input voice signal S51) based on the designation of the shared mode, to generate an output voice signal S42.

[0085] According to an embodiment, the conference server 100 limits the destinations of a response voice signal corresponding to a command voice signal. The conference server 100 also outputs an output voice signal corresponding to the personal mode or shared mode specified by the user terminal 300 participating in the online conference. Based on the personal mode, the conference server 100 outputs an output voice signal that does not include a call voice signal for calling the virtual assistant and a response voice signal from the virtual assistant to user terminals 300 other than the user terminal 300 that specified the personal mode. Based on the shared mode, the conference server 100 also outputs an output voice signal that includes a call voice signal for calling the virtual assistant and a response voice signal from the virtual assistant to user terminals 300 other than the user terminal 300 that specified the shared mode. The conference server 100's control of the output of voice signals makes it possible to select whether other users can hear interactions with the virtual assistant. This prevents other users from hearing content that users do not want to hear or that disrupts the progress of the online conference. The first and second screens displayed by the user terminal 300 also allow users to confirm that other user terminals 300 participating in a given online conference are in personal mode or shared mode.

[0086] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims. The inventions described in the claims of the present application as originally filed are as follows: [Appendix 1] a processor that acquires a response voice signal corresponding to a command voice signal included in a first input voice signal transmitted from a first communication device participating in the online conference, and outputs a first output voice signal including a command-excluded voice signal obtained by excluding the command voice signal from the first input voice signal; a communication interface that receives the first input audio signal transmitted from the first communication device, transmits the response audio signal to the first communication device, and transmits the first output audio signal to other communication devices participating in the online conference; A conference server comprising: [Appendix 2] a memory for storing a first or second mode designated by the first communication device, wherein the processor outputs the first output audio signal based on the designation of the first mode, and outputs a second output audio signal including the first input audio signal and the response audio signal based on the designation of the second mode; the communication interface transmits the first output audio signal to the other communication device based on the designation of the first mode, and transmits the second output audio signal to the other communication device based on the designation of the second mode; 2. The conference server of claim 1. [Appendix 3] the communication interface receives a second input audio signal transmitted from a second communication device participating in the online conference; the processor outputs the first output audio signal including the command excluding audio signal and the second input audio signal based on the designation of the first mode, and outputs the second output audio signal including the first input audio signal, the second input audio signal, and the response audio signal based on the designation of the second mode. 2. The conference server of claim 1, [Appendix 4] The processor acquires the response voice signal output from an assistant program in response to the command voice signal. 4. The conference server according to claim 1, [Appendix 5] A conference system comprising first and second servers, The first server a first processor that acquires a response voice signal corresponding to a command voice signal included in a first input voice signal transmitted from a first communication device participating in an online conference, and outputs a first output voice signal including a command-excluded voice signal obtained by excluding the command voice signal from the first input voice signal; a first communication interface that receives the first input audio signal transmitted from the first communication device, transmits the command audio signal to the second server, receives the response audio signal from the second server, transmits the response audio signal to the first communication device, and transmits the first output audio signal to other communication devices participating in the online conference; Equipped with The second server a second processor that outputs the response voice signal based on the command voice signal; a second communication interface that receives the command voice signal transmitted from the first server and transmits the response voice signal to the first server; A conference system comprising: [Appendix 6] On the computer, means for acquiring a response voice signal corresponding to a command voice signal included in a first input voice signal transmitted from a first communication device participating in an online conference; means for outputting a first output audio signal including a command-excluded audio signal obtained by excluding the command audio signal from the first input audio signal; A program to execute. [Explanation of symbols]

[0087] 100...Conference server 101...Processor 102...Main memory 103…Auxiliary storage device 104...Communication interface 105...Input device 106...Output device 107…RTC 108...Speech recognition processing unit 109...Speech synthesis processing unit 200...Virtual assistant server 201...Processor 202…Main storage device 203…Auxiliary storage device 204...Communication interface 205...Input device 206...Output device 207…RTC 300...User terminal 301...Processor 302…Main storage device 303…Auxiliary storage device 304...Communication interface 305...Input device 306...Output device 307…RTC

Claims

1. a processor that acquires a response voice signal corresponding to a command voice signal included in a first input voice signal transmitted from a first communication device participating in an online conference, and outputs a first output voice signal including a command-excluded voice signal obtained by excluding the command voice signal from the first input voice signal; a communication interface that receives the first input audio signal transmitted from the first communication device, transmits the response audio signal to the first communication device, and transmits the first output audio signal to other communication devices participating in the online conference; a memory for storing a first or second mode designated by the first communication device; Equipped with the processor outputs the first output audio signal based on the designation of the first mode, and outputs a second output audio signal including the first input audio signal and the response audio signal based on the designation of the second mode; The communication interface transmits the first output audio signal to the other communication device based on the designation of the first mode, and transmits the second output audio signal to the other communication device based on the designation of the second mode.

2. the communication interface receives a second input audio signal transmitted from a second communication device participating in the online conference; the processor outputs the first output audio signal including the command-exclusion audio signal and the second input audio signal based on the designation of the first mode, and outputs the second output audio signal including the first input audio signal, the second input audio signal, and the response audio signal based on the designation of the second mode. The conference server of claim 1 .

3. The processor acquires the response voice signal output from an assistant program in response to the command voice signal. The conference server according to claim 1 or 2.

4. A conference system comprising first and second servers, The first server a first processor that acquires a response voice signal corresponding to a command voice signal included in a first input voice signal transmitted from a first communication device participating in an online conference, and outputs a first output voice signal including a command-excluded voice signal obtained by excluding the command voice signal from the first input voice signal; a first communication interface that receives the first input voice signal transmitted from the first communication device, transmits the command voice signal to the second server, receives the response voice signal from the second server, transmits the response voice signal to the first communication device, and transmits the first output voice signal to other communication devices participating in the online conference; Equipped with The second server a second processor that outputs the response voice signal based on the command voice signal; a second communication interface that receives the command voice signal transmitted from the first server and transmits the response voice signal to the first server; A conference system comprising:

5. On the conference server computer, means for acquiring a response voice signal corresponding to a command voice signal included in a first input voice signal transmitted from a first communication device participating in an online conference; means for outputting a first output audio signal including a command-excluded audio signal obtained by excluding the command audio signal from the first input audio signal based on a first mode specified by the first communication device, and for outputting a second output audio signal including the first input audio signal and the response audio signal based on a second mode specified by the first communication device; A program for executing When the program is executed, the communication interface transmits the response audio signal to the first communication device, transmits the first output audio signal to other communication devices participating in the online conference based on the designation of the first mode, and transmits the second output audio signal to the other communication devices based on the designation of the second mode.

Images