Information processing system, information processing device, information processing method, and voice input control program

Abstract

To provide a system which suppresses erroneous mute cancellation during a speech unrelated to a conference or a speech of the other participant and enables each participant to visually recognize an input voice content even if automatic mute cancellation is delayed in the case of an intended speech subjected to the conference, a device, a method and a program.SOLUTION: In an information processing system 1, each terminal device 200 comprises a voice input section 25, a first communication section 290, a voice data control section 211e, a muted state detection section 211f and a muted state control section 211g. A second communication section 140 of an information processing device 100 transmits a determination result of an intention analysis processing section 113 and a determination section of another person speech determination section 115 to a first terminal device and transmits character data to at least the first terminal device and a second terminal device. In a case where a voice is intended to be subjected to a conference and voice data are not being received, the muted state control section of the first terminal cancels a muted state and a display control section of each terminal device displays the character data on a display section 230.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an information processing system, an information processing device, an information processing method, and a voice input control program. [Background technology]

[0002] Patent Document 1 discloses a technology that constantly stores audio data during an electronic conference, sets thresholds for the volume and duration of the microphone audio, and switches between muting and unmuting the microphone. When unmuting, the stored audio data from a predetermined time before until the real-time time is played back to the other party in fast forward. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-016997 Summary of the Invention [Problem to be solved by the invention]

[0004] In conventional conference systems for electronic conferences (sometimes simply called "conferences"), each participant's microphone may be muted in advance. In some cases, the mute is automatically released when there is audio input to the microphone, or when there is audio input but no other participants are speaking. As a result, the mute may be accidentally released by a comment unrelated to the conference or a loud external sound, causing the speaker's voice to be sent to other participants unintentionally. Furthermore, when stored audio data is played back at fast forward, a large time lag can occur, resulting in discrepancies in the conversation due to differences between what is being said and what is being played back on the other end.

[0005] The present disclosure aims to provide a technology such as an information processing system that prevents erroneous unmuting of a speech unrelated to the meeting or during a speech by another participant, and that allows each participant to view the content of the input voice even if automatic unmuting is delayed when a speech is intended to target the meeting. [Means for solving the problem]

[0006] An information processing system according to a first aspect of the present disclosure includes a plurality of terminal devices, including a first terminal device and a second terminal device, and an information processing device. The terminal devices include a voice input unit, a first communication unit, a voice data control unit, a mute state detection unit, a mute state control unit, a display unit, and a display control unit. Voices of conference participants are input to the voice input unit. The first communication unit communicates with the information processing device, and the voice data control unit transmits voice data corresponding to the voices input to the voice input unit to the information processing device. The mute state detection unit detects whether a running conference application is in a mute state. The mute state control unit controls switching of the mute state of the conference application. The display control unit controls the display unit. The information processing device includes a second communication unit, an intention analysis processing unit, and an other person's speech determination unit. The second communication unit communicates with the first communication unit of the terminal devices. The intention analysis processing unit converts the voice data received by the second communication unit from the first communication unit of the first terminal device in the mute state into character data, analyzes the character data, and determines whether the voice is intended to be directed to the conference. The other person's remark determination unit determines whether the second communication unit is receiving the voice data from the second terminal device from which the mute state has been canceled. The second communication unit then transmits a first determination result of the intention analysis processing unit and a second determination result of the other person's remark determination unit to the first terminal device, and transmits the character data to at least the first terminal device and the second terminal device. When the first determination result indicates that the voice is intended to be directed to the conference and the second determination result indicates that the voice data is not being received, the mute state control unit of the first terminal device cancels the mute state, and the display control units of the first terminal device and the second terminal device display the character data on the display unit.

[0007] An information processing system according to a second aspect of the present disclosure includes a plurality of terminal devices, including a first terminal device, and an information processing device. The terminal device includes a voice input unit, a first communication unit, a voice data control unit, a mute state detection unit, a mute state control unit, a display unit, and a display control unit. Voices of conference participants are input to the voice input unit. The first communication unit communicates with the information processing device. The voice data control unit transmits voice data corresponding to the voices input to the voice input unit to the information processing device. The mute state detection unit detects whether a running conference application is in a mute state. The mute state control unit controls switching of the mute state of the conference application. The display control unit controls the display unit. The information processing device includes a second communication unit and a voice recognition unit. The second communication unit communicates with the first communication unit of the terminal device and detects that the voice data has been received from the first terminal device in the mute state. The voice recognition unit converts the voice data into character data. In a first case where the second communication unit detects that it has received the voice data from the first terminal device, the second communication unit transmits the character data converted from the voice data to each of the other terminal devices. The display control unit of each of the other terminal devices displays on the display unit information indicating that the voice has been input to the first terminal device and the character data.

[0008] The information processing device disclosed herein includes a second communication unit, an intention analysis processing unit, and an other-party speech determination unit. The second communication unit receives audio data output corresponding to the speech from a first terminal device, where the running conference application is in a muted state, among a plurality of terminal devices including a first terminal device and a second terminal device, each having an audio input unit for inputting the speech of a conference participant. The intention analysis processing unit converts the audio data into character data, analyzes the character data, and determines whether the speech has an intention directed to the conference. The other-party speech determination unit determines whether the second communication unit is receiving the audio data from the second terminal device, which has been unmuted. The second communication unit then transmits a first determination result of the intention analysis processing unit and a second determination result of the other-party speech determination unit to the first terminal device, and transmits the character data to at least the first terminal device and the second terminal device.

[0009] The information processing method disclosed herein includes the steps of: receiving audio data output in response to audio from a first terminal device, the first terminal device and a second terminal device each having an audio input unit into which the audio of a conference participant is input, the first terminal device having a running conference application in a muted state; converting the audio data into character data, analyzing the character data, and making a first determination to determine whether the audio is intended to be directed to the conference; making a second determination to determine whether the audio data is being received from the second terminal device from which the muted state has been canceled; transmitting the result of the first determination, the result of the second determination, and the character data to the terminal device; and, when the result of the first determination indicates that the audio is intended to be directed to the conference and the result of the second determination indicates that the audio data is not being received, canceling the mute state of the first terminal device and displaying the character data on the first terminal device and the second terminal device.

[0010] The voice input control program of the present disclosure is a voice input control program that causes a computer to execute voice input control of a terminal device, and the voice input control includes a step of detecting whether a running conference application is in a muted state, a step of transmitting voice data corresponding to voice input by a conference participant, and a step of canceling the mute state and displaying character data converted from the voice data when it is determined that the voice is intended to be directed to the conference and the information processing device is not currently receiving the voice data from another terminal device. [Effects of the Invention]

[0011] According to the present disclosure, it is possible to prevent the mute from being accidentally released during a speech unrelated to the meeting or when another participant is speaking, and even if the automatic release of the mute is delayed when a speech is intended to be directed at the meeting, each participant can see the content of the input voice. [Brief explanation of the drawings]

[0012] [Figure 1] 2 is a block diagram showing in detail the configuration of the information processing device 100 and the terminal device 200 of the information processing system 1 according to the first embodiment of the present disclosure. FIG. [Figure 2] 1 is a block diagram showing the connection relationship between the information processing device 100 and each terminal device 200 of the information processing system 1, the schedule management service device 10, and the electronic conference service device 20. FIG. [Figure 3] 1 is a block diagram showing the configuration of a schedule management service device 10. FIG. [Figure 4] FIG. 2 is a block diagram showing the configuration of an electronic conference service device 20. [Figure 5] 1 is a flowchart showing the basics of an information processing method by the information processing system 1. [Figure 6A] 10 is a first half of a flowchart showing an application example of an information processing method by the information processing system 1. [Figure 6B] 10 is a second half of a flowchart showing an application example of the information processing method by the information processing system 1. [Figure 7] FIG. 10 is an explanatory diagram illustrating an example of a login screen of the conference support application 211. [Figure 8] FIG. 10 is an explanatory diagram illustrating an example of a screen for selecting a schedule in the conference support application 211. [Figure 9] 10 is an explanatory diagram illustrating an example of a detailed display screen of a schedule selected by the conference support application 211. FIG. [Figure 10] FIG. 10 is an explanatory diagram showing a display example when the application microphone 212b of the electronic conference application 212 is enabled. [Figure 11] FIG. 10 is an explanatory diagram showing a display example when the application microphone 212b is in a muted state. [Figure 12] FIG. 10 is an explanatory diagram illustrating a situation in which a speaker Ps starts speaking while the application microphone 212b remains muted. [Figure 13] FIG. 10 is an explanatory diagram illustrating a situation immediately before the mute state of the application microphone 212b is automatically released. [Figure 14] FIG. 10 is an explanatory diagram illustrating the situation immediately after the application microphone 212b is automatically unmuted. [Figure 15] FIG. 15 is an explanatory diagram showing a display example of the conference support application 211 in the case of FIG. 14. [Figure 16] FIG. 10 is an explanatory diagram illustrating a situation in which a speaker Ps starts speaking while the application microphone 212b is still in a muted state, while another participant P is speaking. [Figure 17] FIG. 10 is an explanatory diagram illustrating an example in which the mute state is not automatically released while another participant P is speaking. [Figure 18] 18 is an explanatory diagram showing an example of the display of the conference support application 211 of the first terminal device 200-1 of the speaker Ps in the case of FIG. 17. FIG. [Figure 19] 19 is an explanatory diagram showing an example of the display of the conference support application 211 on the terminal device 200 of another participant P in the case of FIG. 18. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0013] Embodiments of the present disclosure will be described with reference to the drawings. In the drawings, the same or corresponding parts are designated by the same reference numerals, and description thereof will not be repeated.

[0014] First Embodiment 1.1 Overall configuration of information processing system 1 First, with reference to FIGS. 1 to 4, the overall configuration of an information processing system 1 capable of holding a conference with multiple participants will be described. The "conference" referred to here may also be referred to as an electronic conference, a video conference, or a web conference. FIG. 1 is a block diagram showing in detail the configuration of an information processing device 100 and a terminal device 200 of the information processing system 1 according to the first embodiment of the present disclosure. FIG. 2 is a block diagram showing the connection relationship between the information processing device 100 and each terminal device 200 of the information processing system 1, and the schedule management service device 10 and the electronic conference service device 20. FIG. 3 is a block diagram showing the configuration of the schedule management service device 10. FIG. 4 is a block diagram showing the configuration of the electronic conference service device 20.

[0015] As shown in FIGS. 1 and 2, the information processing system 1 includes an information processing device 100 and a plurality of terminal devices 200, including a first terminal device 200-1 and a second terminal device 200-2. The information processing device 100 and the terminal devices 200 are connected to each other by wire or wirelessly. Note that FIG. 1 shows only one terminal device 200. In FIG. 2, one of the plurality of terminal devices 200 is designated as the first terminal device 200-1, and the other is designated as the second terminal device 200-2. Note that the names of the first terminal device 200-1 and the second terminal device 200-2 are not fixed, but are used for convenience of explanation. The first terminal device 200-1 is the terminal device 200 at which a speaker Ps starts speaking while the running electronic conference application 212 remains muted. If another participant P is already speaking at that time, the terminal device 200 used by that participant P is the second terminal device 200-2.

[0016] The information processing device 100 and each terminal device 200 are also connected to the schedule management service device 10 and the electronic conference service device 20 by wire or wirelessly.

[0017] The schedule management service device 10 is a device that provides a schedule management service. Various schedule management services are managed and operated by external companies, but they can also be provided in-house. As shown in FIG. 3, the schedule management service device 10 includes a schedule management unit 11, an authentication unit 12, a communication unit 13, a schedule management database D11, and a user management database D12. The schedule management unit 11 manages the schedule management database D11. The authentication unit 12 manages the user management database D12. Since schedule management services are well known, a detailed description of the schedule management service device 10 will be omitted.

[0018] The electronic conference service device 20 is a device that provides an electronic conference service. Various electronic conference services are managed and operated by external companies, but they can also be provided in-house. As shown in FIG. 4, the electronic conference service device 20 includes an electronic conference management unit 21, an authentication unit 22, a communication unit 23, an electronic conference management database D21, and a user management database D22. The electronic conference management unit 21 manages the electronic conference management database D21. The authentication unit 22 manages the user management database D22. Since electronic conference services are well known, a detailed description of the electronic conference service device 20 will be omitted.

[0019] As shown in FIG. 1, the information processing device 100 is a server device that supports each participant P in using an electronic conference service. The information processing device 100 includes a CPU (Central Processing Unit) 110, a storage unit 120, a display unit 130, and a second communication unit 140. The CPU 110 includes a conference management unit 111, an authentication unit 112, an intention analysis processing unit 113, a speaker determination unit 114, and an other person's remark determination unit 115. The intention analysis processing unit 113 includes a voice recognition unit 113a, an analysis unit 113b, and an intention determination unit 113c. The storage unit 120 stores a conference management database D121, an authentication result management database D122, a voice recognition result database D123, and keywords D124.

[0020] The terminal device 200 is a client device used by each participant P of a conference to use the teleconference service provided by the teleconference service device 20. The terminal device 200 includes a CPU 210, a storage unit 220, a display unit 230, an operation unit 240, an audio input unit 250, a speaker 260, a camera 270, and a first communication unit 290. The CPU 210 executes a conference support application 211 and a teleconference application 212. The conference support application 211 includes a conference control unit 211a, a display control unit 211b, a device control unit 211c, a first communication control unit 211d, an audio data control unit 211e, a mute status detection unit 211f, and a mute status control unit 211g. The teleconference application 212 includes a second communication control unit 212a and an application microphone 212b, which is a microphone for this application. Examples of the electronic conference application 212 include, but are not limited to, Zoom (registered trademark), Microsoft Teams (registered trademark), Google Meet (registered trademark), etc. The conference support application 211 communicates with each terminal device 200, the schedule management service device 10, and the electronic conference service device 20 via the first communication unit 290 using the first communication control unit 211d. Similarly, the electronic conference application 212 communicates with each terminal device 200, the schedule management service device 10, and the electronic conference service device 20 via the first communication unit 290 using the second communication control unit 212a. The conference support application 211 and the electronic conference application 212 executed in the CPU 210 can communicate directly with each other.

[0021] 1.2 Main parts of the information processing device 100 and each terminal device 200 The main components of the information processing device 100 and each terminal device 200 will be described first.

[0022] In each terminal device 200, the voice S of a conference participant P is input to the voice input unit 250. The first communication unit 290 communicates with the information processing device 100. The voice data control unit 211e transmits voice data Ds corresponding to the voice S input to the voice input unit 250 to the information processing device 100. The mute state detection unit 211f detects whether the currently running electronic conference application 212 is in a mute state. The mute state control unit 211g controls switching of the mute state of the electronic conference application 212. The display control unit 211b controls the display unit 230.

[0023] In the information processing device 100, the second communication unit 140 communicates with the first communication unit 290 of the terminal device 200. The intention analysis processing unit 113 converts the voice data Ds received by the second communication unit 140 from the first communication unit 290 of the first terminal device 200-1, which is in a muted state, into character data Dc, analyzes the character data Dc, and determines whether the voice S has an intention to target the conference. The other person's remark determination unit 115 determines whether the second communication unit 140 is receiving voice data Ds from the second terminal device 200-2, which has been unmuted.

[0024] Then, the second communication unit 140 transmits the first determination result of the intention analysis processing unit 113 and the second determination result of the other person's remark determination unit 115 to the first terminal device 200-1, and transmits the character data Dc to at least the first terminal device 200-1 and the second terminal device 200-2. If the first determination result indicates that the voice S is intended to target the conference and the second determination result indicates that the voice data Ds is not being received, the mute state control unit 211g of the first terminal device 200-1 cancels the mute state, and the display control units 211b of the first terminal device 200-1 and the second terminal device 200-2 display the character data Dc on the display unit 230. Note that the second communication unit 140 may also transmit the character data Dc to some or all of the terminal devices 200 other than the first terminal device 200-1 and the second terminal device 200-2.

[0025] Therefore, when a voice S of a participant in an electronic conference is input, if the input voice is intended to be directed to the conference and is not currently being spoken by another participant, the mute state is released and character data Dc converted from the voice data Ds is displayed on each terminal device 200. As a result, even if the release of mute is delayed due to the information processing system's processing to analyze the intention of the input voice, each participant can visually recognize the content of the input voice as text from the beginning, allowing the conference to proceed smoothly.

[0026] As a method for the information processing device 100 to recognize whether each terminal device 200 is in a mute state, for example, the conference support application 211 may transmit the detection result of the mute state detection unit 211f to the information processing device 100 when necessary (for example, when the mute state changes). If the received detection result is stored in the storage unit 120 for each terminal device 200, the information processing device 100 can always recognize whether each terminal device 200 is in a mute state. However, the method is not limited to this.

[0027] If the first determination result indicates that the audio S is intended to be directed to a conference and the second determination result indicates that audio data Ds is being received, the display control unit 211b of the first terminal device 200-1 may cause the display unit 230 to display a notification urging the user to cancel the mute state, and the display control unit 211b of the second terminal device 200-2 may cause the display unit 230 to display character data Dc.

[0028] Therefore, when a voice S of a participant in an electronic conference is input, even if the input voice is intended to be directed at the conference, if another participant is speaking, a notification urging the speaker to unmute is displayed on the terminal device 200 of the speaker, and character data Dc converted from the voice data Ds is displayed on the terminal device 200 of the other participant who was speaking. As a result, the speaker can realize that they need to unmute, and the other participant who was speaking can visually recognize the content of the input voice as text from the beginning, allowing the conference to proceed more smoothly.

[0029] The information processing device 100 may further include a storage unit 120 that stores keywords D124 extracted from at least one of the title of the meeting, the character data Dc, and materials used in the meeting. In this case, the intention analysis processing unit 113 determines whether the speech S has an intention to target the meeting, based on the character data Dc and the keywords D124 stored in the storage unit 120.

[0030] Therefore, it is determined whether the input voice is intended for the conference based on the conference title, the character data Dc converted from the voice data Ds, and the keywords D124 extracted from the materials used in the conference. As a result, it is possible to prevent the mute state from being erroneously released for input voice that is not intended for the conference.

[0031] 1.3 Configuration of the information processing device 100 As described above, the information processing device 100 is a server device that supports the use of electronic conference services by each participant P. The following describes configurations and supplementary matters that have not yet been mentioned.

[0032] The CPU 110 controls each unit of the information processing device 100. The CPU 110 may be, for example, a processor or an MPU (Micro Processing Unit), but is not limited to these.

[0033] For example, if an OS (Operating System, sometimes called "basic software") that runs on the CPU 110 and programs for the functions corresponding to the conference manager 111, the authentication unit 112, the intention analysis processor 113, the speaker determiner 114, and the other person's remark determiner 115 are stored in the storage unit 120, the CPU 110 executes the OS and programs to realize the above-mentioned units. Examples of OS include, but are not limited to, Microsoft Windows (registered trademark) and Android (registered trademark). The above-mentioned units may be configured with, but are not limited to, electronic circuits, DSPs (Digital Signal Processors), FPGAs (Field Programmable Gate Arrays), ASICs (Application Specific Integrated Circuits), etc.

[0034] The conference management unit 111 manages conference information such as conference schedules of the schedule management service.

[0035] The authentication unit 112 authenticates a user (hereinafter referred to as "terminal user U") participating in a conference using the terminal device 200. When login is performed using single sign-on (SSO) provided by another company's service, the authentication unit 112 associates the used other company's service with user information. Note that single sign-on is a mechanism that allows a user to log in to multiple services with a single set of ID and password.

[0036] As described above, the intention analysis processing unit 113 analyzes the voice data Ds received by the second communication unit 140 from the first communication unit 290 of the terminal device 200 in a muted state, and determines whether the voice S is intended to be about a conference. The intention analysis processing unit 113 may include, for example, a voice recognition unit 113a that converts the voice data Ds into character data Dc, an analysis unit 113b that analyzes the character data Dc to extract features, and an intention determination unit 113c that determines whether the voice S is intended to be about a conference based on the extracted features. The determination of whether the voice S is intended to be about a conference may be made based on, for example, the extracted features and keywords D124 extracted from the character data Dc or materials used in the conference. Alternatively, the determination may be made by AI (Artificial Intelligence). However, the present invention is not limited to these examples.

[0037] The speaker determination unit 114 determines whether the voice data Ds has one speaker Ps or two or more speakers Ps. If there is one speaker Ps, the terminal user U has already been authenticated, and so it can be assumed that the voice data Ds corresponds to the voice of the terminal user U. The terminal user U is also one of the conference participants P. If there are two or more speakers Ps, there is a possibility that all speakers Ps other than the terminal user U are also conference participants P, or that participants other than the conference participants P are included. An example of a case in which participants other than the conference participants P are included is when the terminal user U is wearing the headset 280 and talking with someone other than those involved in the conference. However, this is not a limited example.

[0038] As described above, the other person's speech determination unit 115 determines whether the second communication unit 140 is receiving the voice data Ds from the second terminal device 200-2, which has been released from the mute state.

[0039] As described above, the storage unit 120 stores keywords D124 extracted from the character data Dc and materials used in the conference. The storage unit 120 may also store in advance the voice data Ds of the participant P as identification information for identifying the speaker Ps. In this case, the speaker determination unit 114 identifies the speaker Ps based on the identification information and determines whether the speaker Ps includes anyone other than the participant P. If the speaker Ps includes anyone other than the participant P, the mute state control unit 211g does not cancel the mute state.

[0040] The storage unit 120 stores information and data necessary for controlling each unit of the information processing device 100. The storage unit 120 may store or include an OS, programs, etc. executed by the CPU 110. The storage unit 120 includes memory, specifically, volatile memory and nonvolatile memory. Volatile memory includes, for example, dynamic random access memory (DRAM) and static random access memory (SRAM), but is not limited to these. Nonvolatile memory includes, for example, read-only memory (ROM), flash memory, solid state drive (SSD), and hard disk, but is not limited to these.

[0041] The display unit 130 includes both those with and without memory properties. Examples of display units without memory properties include, but are not limited to, liquid crystal and organic electroluminescence (EL). Examples of display units with memory properties include, but are not limited to, electronic paper.

[0042] As described above, the second communication unit 140 communicates with the first communication unit 290 of the terminal device 200. The second communication unit 140 also communicates with the schedule management service device 10.

[0043] 1.4 Configuration of each terminal device 200 As described above, the terminal device 200 is a client device used by each participant P of the conference to use the electronic conference service provided by the electronic conference service device 20. The following describes configurations and supplementary matters that have not yet been mentioned. However, the CPU 210, storage unit 220, and display unit 230 have the same configurations as the CPU 110, storage unit 120, and display unit 130 described above, respectively, and therefore their description will be omitted.

[0044] The conference control unit 211 a of the conference support application 211 manages information received from the electronic conference application 212 .

[0045] The display control unit 211b controls the display and display content on the display unit 230 by the electronic conference application 212.

[0046] The device control unit 211c manages the device information and input / output states of the devices included in the terminal device 200, that is, the storage unit 220, the display unit 230, the operation unit 240, the audio input unit 250, the speaker 260, and the camera 270.

[0047] As described above, the first communication unit 290 communicates with the second communication unit 140 of the information processing device 100. Furthermore, the first communication control unit 211d also communicates with the second communication control unit 212a of the electronic conference application 212.

[0048] The application microphone 212 b is interposed between the electronic conference application 212 and the audio input unit 250 .

[0049] The operation unit 240 accepts operations on the terminal device 200. Examples of the operation unit 240 include, but are not limited to, a mouse and a keyboard. The display unit 230 and the operation unit 240 may be integrated using a touch panel that allows touch operations and the like.

[0050] As described above, the voice S of a conference participant P is input to the voice input unit 250. An example of the voice input unit 250 is a microphone that inputs voice, but is not limited to this. In FIG. 1 , the voice input unit 250 is built into the terminal device 200. However, the voice input unit 250 may be arranged outside the terminal device 200 and connected to the terminal device 200.

[0051] The speaker 260 outputs the voice S of another participant P of the conference. In FIG. 1, the speaker 260 is built into the terminal device 200. However, the speaker 260 may be arranged outside the terminal device 200 and connected to the terminal device 200. Furthermore, the voice input unit 250 may be built into the speaker 260. Alternatively, the voice input unit 250 and the speaker 260 may be substituted by a headset 280 that integrates these functions. The headset 280 is extremely suitable in that it can stably and reliably pick up the voice of the terminal user U by maintaining a substantially constant distance between the mouth of the terminal user U and the voice input unit 250.

[0052] The camera 270 captures an image of the face of the terminal user U. In FIG. 1, the camera 270 is built into the terminal device 200. However, the camera 270 may be disposed outside the terminal device 200 and connected to the terminal device 200. Such a camera 270 may have a built-in audio input unit 250.

[0053] 1.5 Basics of information processing method by information processing system 1 Next, the most basic information processing method by the information processing system 1 will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the basics of the information processing method by the information processing system 1.

[0054] 5, in step S1, the second communication unit 140 of the information processing device 100 receives audio data Ds output corresponding to audio S from the first terminal device 200-1, whose running electronic conference application 212 is in a muted state. Here, each of the multiple terminal devices 200, including the first terminal device 200-1 and the second terminal device 200-2, has an audio input unit 250 to which the audio S of a conference participant P is input. Note that step S1 is an example of a "step of receiving audio data" in the present disclosure.

[0055] In step S2, the intention analysis processing unit 113 converts the voice data Ds into character data Dc, analyzes the character data Dc, and determines whether the voice S has an intention to refer to a conference (first determination). Note that step S2 is an example of the "step of making a first determination" of the present disclosure.

[0056] In step S3, the second communication unit 140 determines whether or not the audio data Ds is being received from the second terminal device 200-2, which has been unmuted (second determination). Note that step S3 is an example of the "step of making a second determination" of the present disclosure.

[0057] In step S4, the second communication unit 140 transmits the result of the first determination, the result of the second determination, and the character data Dc to the terminal device 200. Note that step S4 is an example of the "step of transmitting the result of the first determination, the result of the second determination, and the character data" of the present disclosure.

[0058] In step S5, the mute state control unit 211g of the terminal device 200 first determines the result of the first determination. If the result of the first determination indicates that the voice S is intended to be directed to the conference, the process proceeds to the next step S6; otherwise, the process ends.

[0059] In step S6, the mute state control unit 211g also determines the result of the second determination. If the result of the second determination indicates that the audio data Ds is not being received, the process proceeds to the next step S7; otherwise, the process proceeds to step S9.

[0060] In step S7, the mute state control unit 211g also cancels the mute state of the electronic conference application 212 currently being executed on the terminal device 200.

[0061] In step S8, the display control unit 211b of the first terminal device 200-1 displays the character data Dc received from the information processing device 100 in step S4 on the display unit 230. Similarly, the display control unit 211b of the second terminal device 200-2 displays the character data Dc on the display unit 230. Note that these steps S5 to S8 are an example of "a process of canceling the mute state and displaying character data" in the present disclosure.

[0062] On the other hand, in step S9, the display control unit 211b of the first terminal device 200-1 causes the display unit 230 to display a notice prompting the user to cancel muting.

[0063] In step S10, the display control unit 211b of the second terminal device 200-2 causes the display unit 230 to display the character data Dc received from the information processing device 100 in step S4.

[0064] Therefore, when a voice S of a participant in an electronic conference is input, if the input voice is intended to be directed to the conference and is not currently being spoken by another participant, the mute state is released and character data Dc converted from the voice data Ds is displayed on each terminal device 200. As a result, even if the release of mute is delayed due to analysis of the intention of the input voice, each participant can visually recognize the content of the input voice as text from the beginning, allowing the conference to proceed smoothly.

[0065] 1.6 Application examples of the information processing method by the information processing system 1 Next, with reference to FIGS. 6A to 19, an application example of the information processing method by the information processing system 1 will be described. FIGS. 6A and 6B are flowcharts showing an application example of the information processing method by the information processing system 1. FIG. 7 is an explanatory diagram illustrating an example of a login screen of the conference support application 211. FIG. 8 is an explanatory diagram illustrating an example of a screen for selecting a schedule from the conference support application 211. FIG. 9 is an explanatory diagram illustrating an example of a detailed display screen of a schedule selected by the conference support application 211. FIG. 10 is an explanatory diagram illustrating an example of a display when the application microphone 212b of the electronic conference application 212 is enabled. FIG. 11 is an explanatory diagram illustrating an example of a display when the application microphone 212b is in a muted state. FIG. 12 is an explanatory diagram illustrating a situation in which the speaker Ps starts speaking while the application microphone 212b remains in a muted state. FIG. 13 is an explanatory diagram illustrating a situation immediately before the mute state of the application microphone 212b is automatically released. FIG. 14 is an explanatory diagram illustrating a situation immediately after the mute state of the application microphone 212b is automatically released. FIG. 15 is an explanatory diagram showing an example of the display of the conference support application 211 in the case of FIG. 14. FIG. 16 is an explanatory diagram illustrating a situation in which another participant P is speaking when a speaker Ps starts speaking with the application microphone 212b still in a muted state. FIG. 17 is an explanatory diagram illustrating that the mute state is not automatically canceled while another participant P is speaking. FIG. 18 is an explanatory diagram showing an example of the display of the conference support application 211 in the first terminal device 200-1 of the speaker Ps in the case of FIG. 17. FIG. 19 is an explanatory diagram showing an example of the display of the conference support application 211 in the terminal device 200 of another participant P in the case of FIG. 18.

[0066] As shown in FIGS. 6A and 6B, in step S11, the authentication unit 112 of the information processing device 100 authenticates a user (hereinafter referred to as "terminal user U") who is attempting to participate in a conference using the terminal device 200. Prior to this, the terminal user U launches the conference support application 211 on the terminal device 200 that he or she is using. For example, a login screen such as that shown in FIG. 7 appears, and the terminal user U logs in using registered account information (user ID and password). This account information at the time of authentication allows the information processing device 100 to directly identify the terminal user U without relying on voice recognition or the like. After logging in, for example, a schedule list such as that shown in FIG. 8 is displayed on the display unit 230, and the terminal user U selects a desired schedule by operating the operation unit 240. When no schedule is selected, the "Start" button at the bottom is grayed out, indicating that it cannot be clicked. Note that the terminal user U is also a participant P of the conference.

[0067] For example, single sign-on provided by another company's service can be used to log in to the conference support application 211. In this case, since it is possible to log in to the conference support application 211 using account information for the other company's service, there is no need to create a new account dedicated to the conference support application 211. During the login process, the user information is linked between the other company's service used and the conference support service provided by the information processing device 100.

[0068] In step S12, if the details of the selected schedule are for an electronic conference, a screen such as that shown in FIG. 9 is displayed. The selected schedule is highlighted (in FIG. 9, the background is displayed as dots), and the "Start" button at the bottom is also highlighted and becomes clickable. The terminal user U can start the electronic conference application 212 with the simple operation of clicking "Start." This allows the terminal user U to participate in the electronic conference of the selected schedule.

[0069] On the display unit 230, for example, as shown in FIG. 10, the electronic conference application 212 and the conference support application 211 are displayed side by side. This conference support application 211 is a separate application from the electronic conference application 212, and displays a timer for the remaining time in a separate window in conjunction with the start of the conference. In the case of FIG. 10, the application microphone 212b is enabled, so the icon Mon indicating that the mute state is canceled is displayed. When each participant disables the application microphone 212b using the operation unit 240 to put it into a mute state, the icon Moff indicating the mute state is displayed, as shown in FIG. 11. Note that depending on the specifications of the electronic conference application 212, the conference organizer may be able to change the mute state of the application microphone 212b of each participant.

[0070] In step S13, the audio data control unit 211e, which monitors the input of audio S to the audio input unit 250, determines whether audio input has been detected, and if audio input is detected, the process proceeds to the next step S14; if not, the process returns to step S13.

[0071] In step S14, the voice data control unit 211e transmits voice data Ds corresponding to the voice S input to the voice input unit 250 to the information processing device 100 via the first communication unit 290.

[0072] In step S15, the second communication unit 140 of the information processing device 100 receives the audio data Ds output in response to the audio S from the first terminal device 200-1, whose running electronic conference application 212 is in a muted state.

[0073] In step S16, the intention analysis processing unit 113 converts the voice data Ds into character data Dc, and stores the result in the voice recognition result database D123 in the storage unit 120.

[0074] In step S17, the intention analysis processing unit 113 further analyzes the character data Dc to determine whether the voice S has an intention to be directed to a conference.

[0075] In step S18, if the determination result in step S17 is "intended," the process proceeds to step S19. Otherwise, since it is highly likely that the voice S was a statement unrelated to the ongoing conference, the process proceeds to step S28.

[0076] In step S19, the mute state detection unit 211f detects whether the running electronic conference application 212 is in a muted state. A muted state here means that the speaker Ps has started speaking without realizing that the application microphone 212b is in a muted state and without unmuting it, as shown in FIG. 12, for example. In such a case, the processing from this step onwards is executed, and if it is determined to be necessary, the mute state of the application microphone 212b is automatically unmuted. On the other hand, a non-muted state means that the speaker Ps has started speaking after consciously unmuting the mute state, and therefore no processing for automatically unmuting the mute state is required.

[0077] In step S20, the second communication unit 140 of the information processing device 100 transmits to each terminal device 200 the character data Dc converted from the voice data Ds in step S16.

[0078] In step S21, the first communication unit 290 of each terminal device 200 receives the character data Dc transmitted from the second communication unit 140.

[0079] In step S22, the display control unit 211b of each terminal device 200 displays the character data Dc received by the first communication unit 290 on the display unit 230.

[0080] In step S23, the other-person's utterance determination unit 115 determines whether or not voice data Ds is being received from a terminal device 200 other than the first terminal device 200-1. If not, the process proceeds to step S24, and if receiving, the process proceeds to step S26.

[0081] In step S24, the second communication unit 140 transmits an instruction to cancel the mute state to the first terminal device 200-1.

[0082] In step S25, the mute state control unit 211g cancels the mute state of the electronic conference application 212 running on the first terminal device 200-1 in accordance with the received instruction. However, until then, the other participants P have seen the speaker Ps as silent, as shown in FIG. 13, for example. By canceling the mute state, the speaker Ps's speech becomes audible, as shown in FIG. 14, for example, but the beginning of the speech cannot be heard again. However, by performing step S22 described above, the speaker Ps's speech is displayed as text in the conference support application 211, as shown in FIG. 15, for example. This eliminates the need for the speaker Ps to start their speech over again, allowing the conference to proceed smoothly.

[0083] Meanwhile, in step S26, the second communication unit 140 transmits to the first terminal device 200-1 an instruction to display a notification prompting the first terminal device 200-1 to cancel the muted state. At this time, as shown in Fig. 16, for example, the speaker Ps is unaware that the application microphone 212b is in the muted state and has started speaking without canceling the muted state. However, because another participant P is speaking, the muted state of the application microphone 212b is maintained, as shown in Fig. 17.

[0084] In step S27, the display control unit 211b displays the notification received by the first communication unit 290, urging the user to cancel the mute state, on the display unit 230, as shown in Fig. 18. Note that, in step S22 described above, the speech of the speaker Ps is displayed as text in the conference support application 211 on the terminal device 200 of the other participant P, as shown in Fig. 19, for example.

[0085] In step S28, it is determined whether or not the conference is to be continued. If the conference is to be continued, the process returns to step S13; if not, the process ends.

[0086] 1.7 Information Processing System According to Modification of First Embodiment Next, a description will be given of an information processing system according to a modification of the first embodiment. This information processing system is obtained by partially simplifying the configuration of the information processing device 100.

[0087] Specifically, the second communication unit 140 communicates with the first communication unit 290 of the terminal device 200, and detects that voice data Ds has been received from the first terminal device 200-1, which is in a muted state. The intention analysis processing unit 113 only has a voice recognition unit 113a that converts the voice data Ds into character data Dc.

[0088] In the "first case" when the second communication unit 140 detects that it has received voice data Ds from the first terminal device 200-1, the second communication unit 140 transmits character data Dc converted from the voice data Ds to each of the other terminal devices 200. The display control unit 211b of each of the other terminal devices 200 causes the display unit 230 to display information indicating that voice S has been input to the first terminal device 200-1 and the character data Dc.

[0089] Therefore, when a voice S of a participant in an electronic conference is input, the fact that a voice input has been made and the character data Dc converted from the voice data Ds are displayed on each terminal device 200. As a result, even if one of the participants forgets to unmute and starts speaking, the other participants can not only know that a voice input has been made but also visually confirm the content of the input voice, allowing the conference to proceed smoothly.

[0090] Even in the above-mentioned "first case," when the mute state detection unit 211f detects that the mute state of the first terminal device 200-1 has been canceled by the user, each display control unit 211b may not display the character data Dc.

[0091] Therefore, when the speaker cancels the mute state, the input voice is not displayed as text on each terminal device 200. As a result, when the speaker is aware of canceling the mute state, it is possible to avoid a situation in which the contents of the input voice are made known to other participants against the speaker's intention.

[0092] Second Embodiment The information processing device 100 included in the above-described information processing system 1 is within the scope of the present disclosure even when used alone. Therefore, the following description will be given with reference to FIG.

[0093] As shown in FIG. 1, the information processing device 100 includes a second communication unit 140, an intention analysis processing unit 113, and an other person's statement determination unit 115.

[0094] The second communication unit 140 receives audio data Ds output corresponding to audio S from the first terminal device 200-1, whose running electronic conference application 212 is in a muted state. Here, the multiple terminal devices 200 include the first terminal device 200-1 and the second terminal device 200-2. Each terminal device 200 has an audio input unit 250 to which the audio S of a conference participant P is input. The intention analysis processing unit 113 converts the audio data Ds into character data Dc, analyzes the character data Dc, and determines whether the audio S has an intention to target the conference. The other person's remark determination unit 115 determines whether the second communication unit 140 is receiving audio data Ds from the second terminal device 200-2, whose muted state has been canceled.

[0095] The second communication unit 140 transmits the first determination result of the intention analysis processing unit 113 and the second determination result of the other-person's utterance determination unit 115 to the first terminal device 200-1, and also transmits the character data Dc to at least the first terminal device 200-1 and the second terminal device 200-2. Note that the second communication unit 140 may also transmit the character data Dc to some or all of the terminal devices 200 other than the first terminal device 200-1 and the second terminal device 200-2.

[0096] Therefore, when a voice S of a participant in an electronic conference is input, if the input voice is intended to be directed to the conference and is not currently being spoken by another participant, the mute state is released and character data Dc converted from the voice data Ds is displayed on each terminal device 200. As a result, even if the release of mute is delayed due to the information processing system's processing to analyze the intention of the input voice, each participant can visually recognize the content of the input voice as text from the beginning, allowing the conference to proceed smoothly.

[0097] <Third embodiment> In the conference support application 211 of the terminal device 200 provided in the information processing system 1 described above, the scope of the present disclosure also includes a voice input control program that causes a computer to execute voice input control such as detecting and controlling the mute state of the application microphone 212b of the electronic conference application 212.

[0098] This voice input control includes a step of detecting whether the running electronic conference application 212 is in a muted state, a step of transmitting voice data Ds corresponding to the voice S input by the conference participant, and a step of canceling the muted state and displaying character data Dc converted from the voice data Ds when it is determined that the voice S is intended to be directed to the conference and that the information processing device 100 is not currently receiving voice data Ds from another terminal device 200.

[0099] Therefore, the terminal device 200 of the information processing system can be realized simply by having a general-purpose terminal device execute this voice input control program. As a result, it becomes easy to realize the terminal device 200 of the information processing system that enables smooth conference progress.

[0100] The present invention can be embodied in various other forms without departing from its spirit or main features. Therefore, the above-described embodiments are merely illustrative in all respects and should not be interpreted as limiting. The scope of the present invention is defined by the claims and is not limited to the text of the specification. Furthermore, all modifications and variations within the equivalent range of the claims are within the scope of the present invention. [Industrial Applicability]

[0101] The present disclosure is applicable to an information processing system, an information processing device, an information processing method, and a voice input control program. [Explanation of symbols]

[0102] 1. Information Processing Systems 10 Schedule management service device 20. Electronic conference service equipment 100 Information processing device 113 Intention analysis processing unit 113a Voice Recognition Unit 113b Analysis Department 113c Intention determination part 114 Speaker Determination Unit 115 Others' speech determination unit 120 Storage section 130 Display section 140 Second Communications Department 200 Terminal Device 200-1 First terminal device 211 Meeting Support Application 211d First communication control unit 211e Audio data control unit 211f Mute state detection unit 211g Mute status control section 212 Electronic Conferencing Applications 212a Second communication control unit 212b App Microphone 220 Storage section 230 Display section 240 Operation section 250 Audio input section 260 speakers 270 Camera 280 Headset 290 First Communications Department

Claims

1. An information processing system including a plurality of terminal devices including a first terminal device and a second terminal device and an information processing device, The terminal device a voice input unit to which voices of participants in a conference are input; a first communication unit that communicates with the information processing device; a voice data control unit that transmits voice data corresponding to the voice input to the voice input unit to the information processing device; a mute state detection unit that detects whether a running conference application is in a mute state; a mute state control unit that controls switching of the mute state of the conference application; A display unit; a display control unit that controls the display unit; and The information processing device includes: a second communication unit that communicates with the first communication unit of the terminal device; an intention analysis processing unit that converts the voice data received by the second communication unit from the first communication unit of the first terminal device in the mute state into character data, analyzes the character data, and determines whether the voice has an intention to be directed to the conference; an other person's speech determination unit that determines whether the second communication unit is receiving the voice data from the second terminal device from which the mute state has been released; and the second communication unit transmits the first determination result of the intention analysis processing unit and the second determination result of the other person's utterance determination unit to the first terminal device, and transmits the character data to at least the first terminal device and the second terminal device; When the first determination result indicates that the audio is intended to be directed to the conference and the second determination result indicates that the audio data is not being received, the mute state control unit of the first terminal device cancels the mute state, and the display control units of the first terminal device and the second terminal device display the character data on the display unit.

2. 2. The information processing system of claim 1, wherein when the first determination result indicates that the audio is intended to be directed to the conference and the second determination result indicates that the audio data is being received, the display control unit of the first terminal device causes the display unit to display a notification prompting the user to cancel the mute state, and the display control unit of the second terminal device causes the display unit to display the character data.

3. the information processing device further includes a storage unit that stores keywords extracted from at least one of the title of the conference, the character data, and materials used in the conference; 3. The information processing system according to claim 1, wherein the intention analysis processing unit determines whether the speech has an intention to be directed to the conference based on the character data and the keywords stored in the storage unit.

4. An information processing system including a plurality of terminal devices including a first terminal device and an information processing device, The terminal device a voice input unit to which voices of participants in a conference are input; a first communication unit that communicates with the information processing device; a voice data control unit that transmits voice data corresponding to the voice input to the voice input unit to the information processing device; a mute state detection unit that detects whether a running conference application is in a mute state; a mute state control unit that controls switching of the mute state of the conference application; A display unit; a display control unit that controls the display unit; and The information processing device includes: a second communication unit that communicates with the first communication unit of the terminal device and detects that the audio data has been received from the first terminal device that is in the mute state; a voice recognition unit that converts the voice data into character data; and An information processing system in which, in a first case in which the second communication unit detects that it has received the voice data from the first terminal device, the second communication unit transmits the character data converted from the voice data to each of the other terminal devices, and the display control unit of each of the other terminal devices displays information indicating that the voice has been input on the first terminal device and the character data on the display unit.

5. 5. The information processing system of claim 4, wherein even in the first case, when the mute state detection unit detects that the mute state of the first terminal device has been canceled by the user, each of the display control units does not display the character data.

6. a second communication unit that receives audio data output in response to the audio from a first terminal device, the first terminal device and the second terminal device each having an audio input unit to which the audio of a conference participant is input, the first terminal device having a conference application currently being executed in a muted state; an intention analysis processing unit that converts the voice data into character data, analyzes the character data, and determines whether the voice has an intention to be directed to the conference; an other person's speech determination unit that determines whether the second communication unit is receiving the voice data from the second terminal device from which the mute state has been released; Equipped with the second communication unit transmits a first determination result of the intention analysis processing unit and a second determination result of the other person's utterance determination unit to the first terminal device, and transmits the character data to at least the first terminal device and the second terminal device.

7. receiving audio data output in response to the audio from a first terminal device, of a plurality of terminal devices including a first terminal device and a second terminal device, each having an audio input unit to which the audio of a conference participant is input, the first terminal device having a conference application currently being executed in a muted state; converting the voice data into text data and analyzing the text data to make a first determination as to whether the voice has an intention to be directed to the conference; performing a second determination step of determining whether the audio data is being received from the second terminal device whose mute state has been released; transmitting the result of the first determination, the result of the second determination, and the character data to the terminal device; canceling the mute state of the first terminal device and displaying the character data on the first terminal device and the second terminal device when the result of the first determination indicates that the voice is intended to be directed to the conference and the result of the second determination indicates that the voice data is not being received; An information processing method, including:

8. A voice input control program that causes a computer to execute voice input control of a terminal device, The voice input control includes: detecting whether a running conferencing application is in a muted state; transmitting voice data corresponding to voices input by the participants of the conference; canceling the mute state and displaying character data converted from the voice data when it is determined that the voice has an intention to be directed to the conference and when it is determined that the information processing device is not currently receiving the voice data from another terminal device; A voice input control program, including:

Images