Remote conference system, method for generating text information, and program for generating text information

Abstract

To provide a remote conference system capable of selecting an appropriate voice from among input voices.SOLUTION: In a remote conference system in which a server and at least two terminals are communicatively connected to each other, a CPU of the server is equipped with: a voice recognition unit 15 that recognizes voice of each of the at least two terminals; and a selection unit 17 that selects either first character information recognized from the voice of a first terminal or second character information recognized from the voice of a second terminal, when the voice of the first terminal overlaps with at least a part of the voice of the second terminal.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a remote conference system, a character information generating method, and a character information generating program, and more particularly to a remote conference system that generates character information from voice, a character information generating method, and a character information generating program that causes a computer to execute the character information generating method. [Background technology]

[0002] In recent years, remote conference systems have become widespread, in which multiple users located in different locations converse via a computer. Each of the multiple conference participants uses a computer to participate in the remote conference. The participants' voices are picked up by a microphone, and audio data corresponding to the voices is sent to a computer operated by another participant. In these remote conference systems, a technique is known for automatically generating minutes by converting voice into text. When creating the minutes, the speaker is identified, and the speaker and the content of their remarks are recorded in association with each other. In this case, the speaker is identified by identifying the computer that sent the voice.

[0003] On the other hand, there are cases where two or more participants in a remote conference are located in the same area. In this case, the voice of one participant may be picked up by both the computer operated by that participant and the computer operated by the other participant. In this case, the minutes are created as if the comments were made by multiple different people, even though they were made by the same person. Also, a participant may make an operational mistake and record a conversation with a person other than the person in the conference.

[0004] JP 2022-71960 A describes a speech separation system that detects multiple acquired and mixed voice data for each speech section from the start to the end of each voice data, accumulates the own voice data input from the own voice input unit, synchronously references the accumulated voice data for each speaker acquired from each voice input unit, calculates the similarity of the acquired voice data for each speaker, and compares and determines the high and low similarities. Voice data with low similarity is regarded as voice data of different speakers, and voice data with high similarity is regarded as voice data of the same speaker. The voice energy of these voice data that are considered to be the same speaker with high similarity is compared and determined, and the voice data determined to have a relatively large voice energy is identified as the own speech input from the own voice input unit, and the self- and other-speech are separated.

[0005] However, the speech separation system described in JP 2022-71960 A must compare the voice energy of each of the multiple voice data to determine the similarity of the multiple voice data. This requires complex processing. In addition, there is a problem that the voice of a conversation unrelated to the conference may be selected. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent Publication No. 2022-71960 Summary of the Invention [Problem to be solved by the invention]

[0007] An object of the present invention is to provide a remote conference system capable of selecting an appropriate voice from among voices input to a remote conference.

[0008] Another object of the present invention is to provide a text information generating method capable of selecting an appropriate voice from among voices input to a remote conference.

[0009] Another object of the present invention is to provide a character information generating program capable of selecting an appropriate voice from among voices input to a remote conference. [Means for solving the problem]

[0010] According to one aspect of the present invention, a remote conference system in which at least two terminals are communicatively connected and distribute voice input at those terminals to each other, includes a voice recognition means for voice recognizing each of the at least two terminals, and a selection means for selecting, when the voice of the first terminal overlaps with at least a portion of the voice of the second terminal, either first character information obtained by voice recognition of the voice of the first terminal or second character information obtained by voice recognition of the voice of the second terminal.

[0011] According to another aspect of the present invention, a character information generating method is executed on a computer to which at least two terminals are communicatively connected, and includes a voice recognition step of voice recognizing each of the at least two terminals, and a selection step of voice recognizing the voice of the first terminal and selecting either or both of first character information and second character information obtained by voice recognizing the voice of the first terminal when the voice of the first terminal and the voice of the second terminal overlap.

[0012] According to yet another aspect of the present invention, a character information generation program is executed on a computer to which at least two terminals are communicatively connected, and includes a voice recognition step of voice recognizing each of the at least two terminals, and a selection step of voice recognizing the voice of the first terminal and selecting either or both of first character information and second character information obtained by voice recognizing the voice of the first terminal when the voice of the first terminal and the voice of the second terminal overlap. [Brief description of the drawings]

[0013] [Figure 1] 1 is a diagram illustrating an example of a system configuration of a remote conference system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram showing an example of a hardware configuration of a server. [Diagram 3] FIG. 2 is a block diagram showing an example of a hardware configuration of a PC. [Figure 4] FIG. 2 is a diagram illustrating an example of functions of a CPU included in the server according to the present embodiment. [Diagram 5] FIG. 2 is a diagram illustrating an example of functions of a CPU included in a PC according to the present embodiment. [Figure 6] 13 is a flowchart illustrating an example of the flow of a server-side remote conference process. [Figure 7] 10 is a flowchart showing an example of the flow of a voice data recognition process. [Figure 8] 13 is a flowchart showing an example of the flow of a conversation information selection process. [Figure 9] 13 is a flowchart illustrating an example of a terminal-side remote conference process. [Figure 10] FIG. 11 is a first diagram showing an example of a conversation information display screen. [Figure 11] FIG. 2 is a second diagram showing an example of a conversation information display screen. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the following description, the same components are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated.

[0015] Fig. 1 is a diagram showing an example of a system configuration of a remote conference system according to an embodiment of the present invention. With reference to Fig. 1, remote conference system 1 includes a server 100 and personal computers (hereinafter referred to as "PCs") 200-1, 200-2, 200-3, 200-4 to 200-N, where N is a positive integer, 5 or more. Server 100 and PCs 200-1 to 200-N are connected to Internet 5 and are capable of communicating with each other.

[0016] Each of the PCs 200-1 to 200-N includes a camera, a microphone for collecting sound, and a speaker for outputting sound. Each of the PCs 200-1 to 200-N is a general-purpose computer, and has the same main hardware configuration and functions.

[0017] Instead of the PCs 200-1 to 200-N, an information communication device such as a PDA (Personal Digital Assistance) or a smartphone may be used as long as it has a camera, a microphone, a speaker, and a communication function. The network is not limited to the Internet 5, and may be another network as long as the server 100 and the PCs 200-1 to 200-N can communicate with each other. The network may be, for example, a local area network (LAN) or a wide area network (WAN).

[0018] In the remote conference system 1, a conference participant operates one of PCs 200-1 to 200-N to participate in the conference. The participants operating PCs 200-1 to 200-N, respectively, are referred to as users P-1 to N. In other words, user Pn (n is an integer between 1 and N) operates PC 200-n. In this embodiment, a case will be described in which PCs 200-1 and 200-2 are located at a site A, and PCs 200-3 to 200-N are located at a location other than site A. Site A may be, for example, a room in a building. Hereinafter, PCs 200-1 to 200-N will be collectively referred to as PC 200.

[0019] A program for participating in the conference is installed on each of PCs 200-1 to N, and the conference is held by each of PCs 200-1 to N communicating with server 100. The program installed on each of PCs 200-1 to N may be a dedicated program for communicating with server 100, or may be a general browser program if server 100 provides a Web service.

[0020] A remote conference system is realized by executing a remote conference program by server 100. Server 100 communicates with PCs 200-1 to 200-N, and transmits data received from each of PCs 200-1 to 200-N to each of the other PCs 200-1 to 200-N.

[0021] Data transmitted and received between each of PCs 200-1 to 200-N and server 100 includes audio data indicating audio, image data indicating images, and application data. Images include still images and moving images. Data transmitted and received between each of PCs 200-1 to 200-N and server 100 may be compressed data or may be uncompressed data.

[0022] Server 100 controls data to be transmitted to each of PCs 200-1 to N. For example, server 100 can transmit audio data received from each of PCs 200-1 to N to all of PCs 200-1 to N. Server 100 can also aggregate image data received from each of PCs 200-1 to N and transmit the image data to each of PCs 200-1 to N.

[0023] Server 100 determines and transmits the audio data, image data, and application data in response to requests from each of PCs 200-1 to N. Thus, the images displayed on each of PCs 200-1 to N may be the same or different, and the audio output from each of PCs 200-1 to N may be the same or different. Each of PCs 200-1 to N may process and output the multiple pieces of audio data received from server 100. In this case, server 100 does not need to process the audio data, and the load is reduced. Each of PCs 200-1 to N may process and display the multiple pieces of image data received from server 100. In this case, server 100 does not need to process the multiple pieces of image data, and the load is reduced.

[0024] In this embodiment, a user P-1 and a user P-2 are located at the same location A. Here, a case will be described as an example in which the user P-1 and the user P-2 are in a situation in which they can hear each other's voice.

[0025] Fig. 2 is a block diagram showing an example of a hardware configuration of a server. Referring to Fig. 2, the server 100 is a computer that performs arithmetic processing, and includes a central processing unit (CPU) 101 for controlling the entire server 100, a ROM (Read Only Memory) 102 for storing programs executed by the CPU 101, a RAM (Random Access Memory) 103 used as a working area for the CPU 101, a HDD 104 for storing data in a non-volatile manner, a communication unit 105 for connecting the CPU 101 to the Internet 5, a display unit 106 for displaying images, an operation unit 107 for receiving operation inputs, and an external storage device 110, each of which is connected to a bus 113.

[0026] The communication unit 105 is an interface for connecting the server 100 to the Internet 5. Therefore, the CPU 101 can communicate with the PCs 200-1 to 100-N connected to the Internet 5 via the communication unit 105.

[0027] The external storage device 110 is equipped with a CD-ROM (Compact Disk Read Only Memory) 111. The CPU 101 controls the external storage device 110 to read out data stored in the CD-ROM 111.

[0028] In this embodiment, CPU 101 executes a program stored in ROM 102 or HDD 104. CPU 101 may also control external storage device 110 to read a program to be executed by CPU 101 from CD-ROM 111, store the read program in RAM 103, and execute the program.

[0029] Furthermore, CPU 101 downloads a program from a computer connected to Internet 5 and stores it in HDD 104. When a computer connected to Internet 5 writes a program to HDD 104, the program is stored in HDD 104. CPU 101 may load the program stored in HDD 104 into RAM 103 and execute it.

[0030] The recording medium for storing the program executed by CPU 101 is not limited to CD-ROM 111, but may be a flexible disk, a cassette tape, an optical disk (MO (Magnetic Optical Disc) / MD (Mini Disc) / DVD (Digital Versatile Disc)), an IC card, an optical card, a mask ROM, an EPROM (Erasable Programmable ROM), or other semiconductor memory. The program here includes not only a program that can be directly executed by CPU 101, but also a source program, a compressed program, an encrypted program, and the like.

[0031] Fig. 3 is a block diagram showing an example of a hardware configuration of a PC. Referring to Fig. 3, the PC 200 is a computer that performs arithmetic processing, and includes a CPU 201 for controlling the entire PC 200, a ROM 202 for storing a program executed by the CPU 201, a RAM 203 used as a working area for the CPU 201, a HDD 204 for storing data in a non-volatile manner, a communication unit 205 for connecting the CPU 201 to the Internet 5, a display unit 206 for displaying images, an operation unit 207 for accepting input of operations by participants who are users, a camera 208 for capturing images of the participants, a speaker 209 for outputting sound, a microphone 210 for collecting sound, an external storage device 211, and a GPS receiver 212, each of which is connected to a bus 213.

[0032] A CD-ROM 211A is inserted into the external storage device 211. The CPU 201 controls the external storage device 211 to read out data stored on the CD-ROM 211A.

[0033] A module in which at least two of the camera 208, the speaker 209, and the microphone 210 are integrated may be connected to the PC 200. The module includes, for example, a headset in which the speaker 209 and the microphone 210 are integrated.

[0034] The GPS receiver 212 receives signals transmitted by a plurality of GPS (Global Positioning System) satellites and calculates a position by analyzing the signals. The GPS receiver 212 outputs position information indicating the detected position to the CPU 201. Note that the PC 200 does not need to include the GPS receiver 212. In this case, the user may input the position information to the PC 200, and the position information may be stored in the HDD 204.

[0035] Fig. 4 is a diagram showing an example of functions of a CPU included in a server in this embodiment. The functions shown in Fig. 4 are realized by CPU 101 included in server 100 as CPU 101 executes a server-side remote conference program stored in ROM 102, HDD 104 or CD-ROM 111. The server-side remote conference program is a part of the remote conference program. With reference to Fig. 4, CPU 101 includes a voice receiving unit 11, a voice transferring unit 13, a voice recognizing unit 15, a selecting unit 17, a conversation information transmitting unit 19, and a conversation history storing unit 21.

[0036] The voice receiving unit 11 controls the communication unit 105 to receive voice data from any of the PCs 200-1 to N. In response to receiving the voice data, the voice receiving unit 11 outputs the voice data to the voice transferring unit 13, the voice recognition unit 15, and the selection unit 17. The voice receiving unit 11 also outputs device identification information for identifying the device that has transmitted the voice data among the PCs 200-1 to N to the voice transferring unit 13, the voice recognition unit 15, and the selection unit 17. The voice receiving unit 11 may receive voice data in parallel from two or more of the PCs 200-1 to N.

[0037] The audio transfer unit 13 transfers the audio data input from the audio receiving unit 11. The audio transfer unit 13 receives the audio data and device identification information from the audio receiving unit 11. The audio transfer unit 13 controls the communication unit 105 to transmit the audio data to all devices among the PCs 200-1 to N except for the device identified by the device identification information input from the audio receiving unit 11. When multiple pieces of audio data are input in parallel from the audio receiving unit 11, the audio transfer unit 13 may transmit each of the multiple pieces of audio data to the destination device, or may transmit a single piece of audio data obtained by synthesizing the multiple pieces of audio data.

[0038] The voice recognition unit 15 receives voice data and device identification information from the voice receiving unit 11. The voice recognition unit recognizes the voice data and converts it into character data. The voice recognition unit 15 refers to a correspondence table in which users P-1 to N and PCs 200-1 to N are associated with each other and stored in advance in the HDD 104, and identifies a user corresponding to the device identification information input from the voice receiving unit 11. The voice recognition unit 15 associates the character information converted from the voice data with the user identification information of the user identified from the device identification information. The voice recognition unit 15 outputs a set of the character information and the user identification information to the selection unit 17. When multiple pieces of voice data are input in parallel from the voice receiving unit 11, the voice recognition unit 15 processes the multiple pieces of voice data in parallel. For this reason, the voice recognition unit 15 may output multiple pieces of character information to the selection unit 17 at the same time.

[0039] Selection unit 17 receives one or more pieces of character information from voice recognition unit 15. When selection unit 17 receives one piece of character information from voice recognition unit 15, it outputs a pair of the character information and user identification information to conversation information transmission unit 19 and conversation history storage unit 21. When selection unit 17 receives multiple pieces of character information from voice recognition unit 15, it selects one or more pieces of character information from the multiple pieces of character information if a predetermined relationship is established between the multiple pieces of character information or information related to the multiple pieces of character information.

[0040] When a plurality of pieces of character information are input from the voice recognition unit 15, the selection unit 17 determines that a plurality of pieces of voice data are superimposed. The plurality of pieces of character information input to the selection unit 17 correspond to a plurality of pieces of voice data input in parallel. Here, a case where a plurality of pieces of voice data are superimposed will be described. Voice data includes a time element and is data that is continuous in time. A case where a plurality of pieces of voice data are superimposed is a case where the plurality of pieces of voice data overlap in time. A plurality of pieces of voice data are superimposed when the periods determined by the time elements of each of the plurality of pieces of voice data completely match, or when the periods of the plurality of pieces of voice data overlap in part. When a plurality of pieces of voice data are superimposed, the selection unit 17 processes a plurality of pieces of character information corresponding to each of the plurality of pieces of voice data.

[0041] When selecting unit 17 selects one or more pieces of character information, it outputs a pair of the character information and user identification information for each of the selected one or more pieces of character information to conversation information transmitting unit 19 and conversation history storage unit 21. When selecting unit 17 does not select one or more pieces of character information, it outputs a pair of the character information and user identification information for each of the multiple pieces of character information to conversation information transmitting unit 19 and conversation history storage unit 21.

[0042] The selection unit 17 includes a similarity determination unit 23, a volume comparison unit 25, and a history reference unit 27. The similarity determination unit 23 compares a plurality of pieces of character information. The similarity determination unit 23 detects a predetermined relationship between the plurality of pieces of character information. The predetermined relationship here is a relationship in which two pieces of character information are similar. When at least two pieces of character information among the plurality of pieces of character information are similar, the similarity determination unit 23 identifies two pieces of similar character information. When the similarity determination unit 23 identifies two pieces of similar character information, it determines the pair of the two pieces of character information to be selected.

[0043] The similarity determination unit 23 analyzes the entire text information and determines whether the text information is similar or not using a distributed representation that quantifies the text information as a vector. Specifically, the similarity determination unit 23 quantifies each of the multiple pieces of text information as a continuous vector and calculates the similarity. The similarity determination unit 23 may also compare the text information itself. In this case, the similarity of the text information includes a complete match and a match of a predetermined percentage or more. The predetermined percentage is determined according to the accuracy of the voice recognition.

[0044] For example, the user P-1 and the user P-2 are in a positional relationship where they can hear each other's voice. Therefore, the voice uttered by either one of them may be input to a device operated by the other. For example, the voice uttered by the user P-1 may be input to the PC 200-1 operated by the user P-1 and the PC 200-2 operated by the user P-2. In this case, the voice data of the voice uttered by the user P-1 is received from each of the PC 200-1 and the PC 200-2. Therefore, the character information paired with the user identification information of the user P-1 and the character information paired with the user identification information of the user P-2 are similar, so that each is identified by the similarity determination unit 23 and determined as a selection target. When three or more pieces of character information are input, the similarity determination unit 23 determines the similarity of each pair of two pieces of character information. For example, when three pieces of character information C1, C2, and C3 are input, the similarity is judged for each of the three pairs: the pair of character information C1 and character information C2, the pair of character information C1 and character information C2, and the pair of character information C2 and character information C3.

[0045] The volume comparison unit 25 receives audio data from the audio receiving unit 11. The volume comparison unit 25 detects a predetermined relationship between a plurality of pieces of audio data corresponding to a plurality of pieces of character information. The predetermined relationship here is a relationship between the difference in volume between the two pieces of audio data and a first threshold value. The first threshold value is a predetermined value and is stored in advance in the HDD 104. The volume comparison unit 25 compares the volumes of the plurality of pieces of audio data simultaneously input from the audio receiving unit 11 with the first threshold value. When there is audio data whose volume is equal to or less than the first threshold value among the plurality of pieces of audio data, the volume comparison unit 25 identifies character information corresponding to the volume data whose volume is equal to or less than the first threshold value. When identifying character information corresponding to the volume data whose volume is equal to or less than the first threshold value, the volume comparison unit 25 determines a pair of the character information and another piece of character information to be selected.

[0046] Furthermore, the volume comparing unit 25 compares the volumes of two pieces of audio data corresponding to the two pieces of character information determined as selection targets by the similarity determining unit 23. The volume comparing unit 25 selects the one with the larger volume from among the two pieces of character information determined as selection targets by the similarity determining unit 23, and does not select the other. The volume comparing unit 25 outputs a pair of the selected character information and the user identification information to the conversation information transmitting unit 19 and the conversation history storing unit 21.

[0047] Furthermore, the volume comparison unit 25 selects one or more pieces of character information whose volume is greater than a first threshold value from among the plurality of pieces of character information determined to be selected, and does not select any piece of character information whose volume is equal to or less than the first threshold value. The volume comparison unit 25 outputs a pair of the selected character information and the user identification information to the conversation information transmission unit 19 and the conversation history storage unit 21.

[0048] The history reference unit 27 compares each of the plurality of pieces of character information determined as selection targets by the similarity determination unit 23 or the volume comparison unit 25 with the history information. The history information is character information stored in the HDD 104 by the conversation history storage unit 21 described later. The history reference unit 27 evaluates the character information by referring to the history information. For example, the history reference unit 27 analyzes the entire history information and calculates the occurrence frequency for each word. Then, the history reference unit 27 evaluates the character information using the occurrence frequency for each of the plurality of words included in the character information. The history reference unit 27 selects character information whose evaluation value is greater than a second threshold value and does not select character information whose evaluation value is equal to or less than the second threshold value. The second threshold value is a predetermined value and is stored in advance in the HDD 104. The history reference unit 27 outputs a pair of the selected character information and the user identification information to the conversation information transmission unit 19 and the conversation history storage unit 21. The history reference unit 27 may select one of the two pieces of text information determined by the similarity determination unit 23 as the selection target, the one with the higher evaluation value.

[0049] In response to input of a set of character information and user identification information from selection unit 17, conversation information transmission unit 19 controls communication unit 105 to transmit conversation information including the character information and the user identification information to PCs 200-1 to 200-N.

[0050] In response to input of a pair of character information and user identification information from selection unit 17, conversation history storage unit 21 stores conversation information including the character information and the user identification information in HDD 104. Specifically, conversation history storage unit 21 generates a conference table corresponding to the remote conference in HDD 104 and adds the conversation information to the conference table. Conversation history storage unit 21 may store the conversation information in the conference table in association with the time when audio data corresponding to the character information was received. The character information included in the conference table is history information previously stored in association with the remote conference.

[0051] Fig. 5 is a diagram showing an example of functions of a CPU included in a PC in this embodiment. The functions shown in Fig. 5 are realized by CPU 201 included in PC 200 as CPU 201 executes a terminal side remote conference program stored in ROM 202, HDD 204 or CD-ROM 211A. The terminal side remote conference program is a part of the remote conference program. Referring to Fig. 5, CPU 201 included in PC 200 includes terminal side transmitting unit 251, audio input control unit 253, terminal side receiving unit 255, audio output control unit 257, character information receiving unit 259, and character information display unit 261.

[0052] Analog voice data output from the microphone 210 is input to the voice input control unit 253. The voice input control unit 253 converts the analog voice data into digital voice data, and outputs the converted voice data to the terminal side transmission unit 251. The voice input control unit 253 may compress the voice data, and output the compressed voice data to the terminal side transmission unit 251. The terminal side transmission unit 251 controls the communication unit 205 to transmit the voice data input from the voice input control unit 253 to the server 100.

[0053] The terminal side receiving unit 255 controls the communication unit 205 to receive audio data from the server 100. The terminal side receiving unit 255 outputs the audio data received from the server 100 to the audio output control unit 257. The audio output control unit 257 reproduces the audio data. Specifically, the audio output control unit 257 converts the digital signal of the audio data into an analog signal, and outputs the analog audio data to the speaker 209. As a result, the audio of the audio data is output from the speaker 209.

[0054] Character information receiving unit 259 controls communication unit 205 to receive conversation information consisting of a pair of character information and user identification information from server 100. Character information receiving unit 259 outputs the conversation information received from server 100 to character information display unit 261. Character information display unit 261 displays the conversation information on display unit 206. Specifically, character information display unit 261 displays the conversation information on display unit 206. When multiple pieces of conversation information are input, character information display unit 261 displays the multiple pieces of conversation information in chronological order. As a result, remarks by participants in the remote conference are displayed on display unit 206 in chronological order.

[0055] Fig. 6 is a flowchart showing an example of the flow of server-side remote conference processing. The server-side remote conference processing is processing executed by CPU 101 included in server 100 as CPU 101 executes a server-side remote conference program stored in ROM 102, HDD 104, or CD-ROM 111. With reference to Fig. 6, CPU 101 included in server 100 acquires participant information (step S01), and advances the processing to step S02. When users P-1 to N operate any of PCs 200-1 to N to log in to server 100, CPU 101 determines that the user who has been permitted to log in is a participant of the remote conference. CPU 101 acquires user identification information of the user who is a participant as participant information. Here, a case where users P-1 to N are permitted to log in will be described as an example.

[0056] In step S02, it is determined whether or not reception of audio data has started. If reception of audio data has started, the process proceeds to step S05, but if not, the process proceeds to step S08. If audio data is received from any of PCs 200-1 to 200-N, it is determined that reception of audio data has started.

[0057] In step S03, a voice data recognition process is executed, and the process proceeds to step S04. The voice data recognition process will be described in detail later, but is a process executed on the voice data received in step S02, and conversation information is generated. If multiple voice data are received simultaneously in step S02, the voice data recognition process is executed for each of the multiple voice data. In other words, in the voice data recognition process, if multiple voice data are received, the CPU 101 executes in parallel tasks for executing multiple voice data recognition processes respectively corresponding to the multiple voice data.

[0058] Fig. 7 is a flowchart showing an example of the flow of the voice data recognition process. Before the voice data recognition process is executed, voice data is received from one of PCs 200-1 to 200-N. With reference to Fig. 7, the speaker of the voice data is identified (step S11), and the process proceeds to step S12. CPU 101 refers to the correspondence table stored in HDD 104, and identifies the user identified by user identification information associated with the device that has transmitted the voice data, among PCs 200-1 to 200-N, as the speaker.

[0059] In step S12, transfer of the voice data is started, and the process proceeds to step S13. CPU 101 controls communication unit 105 to start a process of transmitting the voice data to all devices among PCs 200-1 to 200-N except the device that transmitted the voice data. CPU 101 transmits the user identification information of the speaker together with the voice data. As a result, playback of the voice data is started in the other devices, so that the voice uttered by the speaker is played back in all the other devices. In addition, since the user identification information is received in the other devices together with the voice data, the voice data can be played back with the speaker identified.

[0060] In step S13, speech recognition is started, and the process proceeds to step S14. As a result, text information is generated from the speech data. In the next step S14, the text information is associated with the speaker, and the process proceeds to step S15.

[0061] In step S15, it is determined whether the reception of the voice data has ended. If the voice data continues to be received (YES in step S15), the process returns to step S14, but if the voice data is no longer received (NO in step S15), the process proceeds to step S16.

[0062] In step S16, the conversation information is generated and temporarily stored, and the process returns to the server-side remote conference process. Conversation information including the user identification information of the speaker identified in step S11 and the character information generated in step S13 is generated and temporarily stored in ROM 102.

[0063] Returning to Fig. 6, in step S04, it is determined whether two or more pieces of voice data have been received. The process branches depending on the number of tasks that execute the voice data recognition process. If there are multiple tasks that execute the voice data recognition process, the process proceeds to step S05, but if not, the process proceeds to step S06.

[0064] In step S05, a conversation information selection process is executed, and the process proceeds to step S06. The conversation information selection process will be described in detail later, but is a process for selecting one or more of two or more pieces of voice data. In step S06, it is determined whether the voice data recognition process has ended. It is determined whether the task that executes the voice data recognition process has ended. If multiple tasks are executed, it is determined whether all of the multiple tasks have ended. If all of the one or more tasks that execute the voice data recognition process have ended, the process proceeds to step S07, but if not, the process returns to step S02.

[0065] In step S07, the conversation information is transmitted, and the process proceeds to step S08. CPU 101 controls communication unit 105 to start a process of transmitting the conversation information generated in step S03 to each of PCs 200-1 to N. As a result, in each of PCs 200-1 to N, text information indicating the voice uttered by the speaker is displayed together with the user identification information of the speaker.

[0066] In step S08, the conversation information is stored, and the process proceeds to step S09. The conversation information is added to the conference table stored in HDD 104. When there are multiple tasks for executing the voice data recognition process, multiple pieces of conversation information corresponding to the multiple tasks are stored. Note that, among the multiple pieces of conversation information, any conversation information that was not selected in step S05 is not stored. The conversation information is stored in chronological order. Conversation information in which reception of voice data begins earlier is stored before conversation information in which reception of voice data begins later.

[0067] In step S09, it is determined whether the conference has ended. If an end command is received from any of PCs 200-1 to 200-N, it is determined that the conference has ended. If the conference has ended, the process ends, but if not, the process returns to step S02.

[0068] FIG. 8 is a flowchart showing an example of the flow of the conversation information selection process. The conversation information selection process is a process executed in step S05 of the server-side remote conference process. Referring to FIG. 8, CPU 101 judges whether two pieces of character information are similar or not (step S21). Two pieces of character information are compared, and if they match completely or partially, it is judged that they are similar. The rate of partial match is determined by the accuracy of voice recognition. When the accuracy of voice recognition is 90%, it is judged that they are similar if the number of characters that match in the character information is 90%. If the two pieces of character information are similar, the process proceeds to step S24, but if not, the process proceeds to step S22. If the two pieces of character information are similar, it corresponds to a case where a voice uttered by the same person is input to two of PCs 200-1 to N. In this embodiment, it is a case where a voice uttered by one of users P-1 and P-2 is input to each of PCs 200-1 and 200-2.

[0069] In step S22, it is determined whether the volume of one of the two voices is equal to or less than the first threshold. The first threshold is a predetermined value. When a conference participant talks to a person not participating in the conference, the participant may speak in a low voice. For example, when the user P-3 is spoken to by a person next to him / her, if the microphone 210 of the PC 200-3 is not turned off in a situation where the user P-3 responds to the person, the user P-3's voice is input to the microphone 210 of the PC 200-3. In this case, the user P-3 often responds in a low voice. For this reason, the PC 200-3 transmits voice data with a volume equal to or less than the first threshold. In step S22, if the volume of one of the two voices is equal to or less than the first threshold, the process proceeds to step S24, but if not, the process proceeds to step S23. In step S23, both of the two pieces of text information are set to a selected state, and the process returns to the server-side remote conference process.

[0070] In step S24, the process is divided according to the selection mode. The selection mode is set in advance by a user who operates the server 100. The selection modes include a volume-based mode, an input timing-based mode, and a relevance-based mode. If the selection mode is set to the volume-based mode, the process proceeds to step S25; if the selection mode is set to the input timing-based mode, the process proceeds to step S27; and if the selection mode is set to the relevance-based mode, the process proceeds to step S29.

[0071] In step S25, the voice data with the louder volume is identified, and the process proceeds to step S26. The voice data with the louder volume is identified from among the two voice data corresponding to the two pieces of text information, respectively. In step S26, the text information corresponding to the voice data identified in step S25 is selected, and the process returns to the server-side remote conference process. The text information corresponding to the voice data with the louder volume is selected.

[0072] In step S27, the voice data received first is identified, and the process proceeds to step S28. Of the two voice data corresponding to the two pieces of text information, the voice data whose reception timing was initiated first is identified. In step S28, the text information corresponding to the voice data identified in step S27 is selected, and the process returns to the server-side remote conference process. The text information corresponding to the voice data whose reception started first is selected.

[0073] In step S29, the relevance between each of the two pieces of character information and the history information is calculated. The relevance is the sum of the frequency of occurrence of one or more words included in the character information in the history information. The conference table stored in HDD 204 is referenced, and the appearance frequency of each word included in the character information is calculated to calculate the relevance. In step S30, the character information with the higher relevance is selected from the two pieces of character information, and the process returns to the server-side remote conference process. The character information with the higher relevance to the conference is selected from the plurality of pieces of character information.

[0074] The selection mode may be a combination of two or more modes. For example, all of the volume-based mode, input timing-based mode, and relevance-based mode may be set. In this case, a priority order is set for the three modes, and the reference mode is selected in descending order of priority. For example, if the priority order is set to the volume-based mode, input timing-based mode, and relevance-based mode, the volume-based mode is set if there is a difference in the volume of two pieces of audio data corresponding to two pieces of text information, and the input timing-based mode is set if there is no difference in the volume. In the case where the reception of the two voice data starts simultaneously, the association-based mode is selected. In the case where the reception of the two voice data starts simultaneously, the association-based mode is selected. The reception of the two voice data starts simultaneously includes the case where the reception starts within a predetermined period.

[0075] The presence or absence of a difference in volume is determined based on the presence or absence of a volume difference of a predetermined value or more. If the difference in volume between the two voices is a predetermined value or more, it is determined that there is a difference in volume, and if the difference in volume between the two voices is less than the predetermined value, it is determined that there is no difference in volume. The order of priority may be the volume reference mode, the relevance reference mode, and the input timing reference mode, or the input timing reference mode, the volume reference mode, and the relevance reference mode, or the input timing reference mode, the relevance reference mode, the input timing reference mode, and the volume reference mode, or the relevance reference mode, the volume reference mode, and the input timing.

[0076] Also, the combination of the selection modes may be a combination of any two of the volume reference mode, the input timing reference mode, and the relevance reference mode. For example, the combination of the selection modes may be a combination of the volume reference mode and the input timing reference mode, a combination of the volume reference mode and the relevance reference mode, or a combination of the input timing reference mode and the volume reference mode. Furthermore, the selection mode may be any one of the volume reference mode, the input timing reference mode, and the relevance reference mode.

[0077] 9 is a flowchart showing an example of terminal-side remote conference processing. The terminal-side remote conference processing is processing executed by CPU 201 included in PC 200 as CPU 201 executes a terminal-side remote conference program stored in ROM 202, HDD 204, or CD-ROM 211A. The terminal-side remote conference program is a part of the remote conference program.

[0078] 9, CPU 201 included in PC 200 determines whether or not voice has been input (step S51). CPU 201 analyzes the output of speaker 209, and determines that voice has been input when voice is detected from the voice data output by speaker 209. If voice has been input, the process proceeds to step S52, and if not, the process proceeds to step S53. In step S52, the voice data is transmitted to server 100, and the process proceeds to step S57. CPU 201 controls communication unit 205 to transmit the voice data to server 100.

[0079] In step S53, it is determined whether or not audio data has been received from server 100. It is determined whether or not communication unit 205 has received audio data from server 100. If audio data has been received, the process proceeds to step S54, otherwise the process proceeds to step S55. In step S54, the received audio data is played back at a volume level determined in the audio data, and the process proceeds to step S57.

[0080] In step S55, it is determined whether conversation information has been received from server 100. It is determined whether communication unit 205 has received conversation information from server 100. If conversation information has been received, the process proceeds to step S56, and if not, the process proceeds to step S57. In step S56, the received conversation information is displayed on display unit 206, and the process proceeds to step S57.

[0081] In step S57, it is determined whether the conference has ended. CPU 201 determines that the conference has ended when communication unit 205 receives a signal indicating the end of the conference from server 100. If the conference has not ended, the process returns to step S51, but if not, the process ends.

[0082] FIG. 10 is a first diagram showing an example of a conversation information display screen. The conversation information display screen shown in FIG. 10 shows a conversation information display screen that is displayed when a conversation information selection process is not executed. In FIG. 10, user identification information, character information, and time information are displayed as conversation information. The user identification information includes an icon image and a user ID. The icon image is an image that is predetermined for a user, and an image of a face photograph is used here. The user ID is displayed as "A" for user P-1, "B" for user P-2, and "C" for user P-3. For example, the first conversation information shows that a user with a user ID of "C" uttered the voice "Maybe. How about I ask from here?" at 10:45.

[0083] Referring to FIG. 10, focusing on the first and second conversation information, the time 10:45 is the same in all conversation information. Therefore, one of the first and second conversation information is selected. Here, it is assumed that the volume of the voice data corresponding to the first conversation information is larger than the volume of the voice data corresponding to the second conversation information. Therefore, the first conversation information is selected. The text information of the second conversation information includes "It's nice weather, isn't it?" which is unrelated to the meeting. Therefore, the text information of the second conversation information corresponds to a case where the relevance to the meeting is low, so it may not be selected based on the relevance.

[0084] Focusing on the third through eighth pieces of conversation information, the time 10:46 is the same in all of the conversation information. Therefore, one or more of the third through eighth pieces of conversation information are selected. First, focusing on the third and fourth pieces of conversation information, the text information is the same. This is the case when voice spoken by either the user with user ID "A" or the user with user ID "B" is input to PC 200-1 and PC 200-2, respectively. In this case, either the third or fourth piece of conversation information is selected.

[0085] Also, looking at the sixth and seventh conversation information, the sixth conversation information is the text information "I am listening to your flight" with user ID "A", and the seventh conversation information is the text information "My speaker is off" with user ID "B". When comparing the two in terms of sound, the sixth "I am listening to your flight" and the seventh "My speaker is off" have the same ten sounds. Therefore, the two text information are similar. This is the case where voices uttered by either user ID "A" or user ID "B" are input to PC200-1 and PC200-2, respectively. The reason why the text information is different even though the voices are uttered by the same person is because the voice recognition was incorrect. In this case, either the sixth or the seventh conversation information is selected. It may be selected based on the volume or the relevance.

[0086] Next, looking at the ninth and tenth pieces of conversation information, the time 10:47 is the same in all pieces of conversation information. Here, it is assumed that the volume of the audio data corresponding to the tenth piece of conversation information is lower than the first threshold. In this case, one or more pieces of conversation information are selected from the ninth and tenth pieces of conversation information. If the volume of the audio data corresponding to the ninth piece of conversation information is equal to or higher than the first threshold, the ninth piece of conversation information is selected.

[0087] Fig. 11 is a second diagram showing an example of a conversation information display screen. The conversation information display screen shown in Fig. 11 is displayed when a conversation information selection process is executed. The conversation information display screen shown in Fig. 11 differs from the conversation information display screen shown in Fig. 10 in that it does not include the second, fourth, sixth, and tenth conversation information displayed on the conversation information display screen shown in Fig. 10.

[0088] The third conversation information displayed on the conversation information display screen shown in FIG. 10 was shown as conversation information corresponding to user "A", but in FIG. 11, the user identification information corresponding to the conversation information is not displayed. The reason why the user identification information is not displayed is because it is not possible to distinguish whether the speech is made by user ID "A" or user ID "B". In this case, an icon image "?" indicating that the speaker is unknown and a character string "Caller Unknown" are displayed as the user identification information. It is also possible to specify whether the speech corresponding to the character information is made by user ID "A" or user ID "B" based on the volume of the voice data. For example, if the voice data D-2 received from PC 200-2 is louder than the voice data D-1 received from PC 200-1, the user ID "C" is selected.

[0089] As described above, in the remote conference system 1 in this embodiment, the server 100 and the PCs 200-1 to N are communicatively connected, the server 100 transfers the voice inputted in any of the PCs 200-1 to N to all the others, performs voice recognition on the voice data received from the first terminal and the second terminal of the PCs 200-1 to N, and when the voice of the first terminal overlaps with at least a part of the voice of the second terminal, selects either the first character information obtained by voice recognition of the voice of the first terminal or the second character information obtained by voice recognition of the voice of the second terminal. Therefore, since one of the multiple pieces of character information obtained by voice recognition of the multiple voices to be superimposed is selected, it is possible to select an appropriate voice from the voices inputted to the remote conference.

[0090] Furthermore, when the first character information and the second character information are similar to each other, the server 100 selects one of them, thereby making it possible to prevent the same character information from being duplicated.

[0091] In addition, the server 100 selects either the first character information or the second character information when the volume of the voice of one of the first terminal and the second terminal is equal to or less than the first threshold and the volume of the voice of the other is greater than the first threshold. When overlapping voice data is input, the one with the lower volume is often a voice unrelated to the remote conference. Therefore, when the volume of the voice of one of the first terminal and the second terminal is equal to or less than the first threshold and the volume of the voice of the other is greater than the first threshold, either the first character information or the second character information is selected, so that the character information of the voice unrelated to the conference is not selected.

[0092] Furthermore, the server 100 selects the first character information when the volume of the voice of the first terminal is greater than the volume of the voice of the second terminal. Since the first character information is selected when the volume of the voice data received from the first terminal is greater than the volume of the voice data received from the second terminal, the selection of the first character information is easy.

[0093] In addition, the server 100 selects the first character information when the timing at which the voice data starts to be received from the first terminal is earlier than the timing at which the voice data starts to be received from the second terminal, out of the first character information and the second character information. This makes it easy to select the first character information.

[0094] In addition, the server 100 selects one of the first character information and the second character information that has a higher evaluation based on history information of past voice recognition, thereby making it possible to prevent voices unrelated to the remote conference from being selected.

[0095] In addition, the server 100 identifies the speaker of the voice, associates the speaker with text information obtained by voice recognition of the voice, and does not associate the selected text information with the speaker when the voice of the first terminal overlaps at least a part of the voice of the second terminal, thereby preventing association with an incorrect speaker.

[0096] <Summary of the embodiment> (Item 1) In a remote conference system in which at least two terminals are connected to each other so that audio inputted at each terminal is distributed to the other terminals, A voice recognition means for recognizing voices of each of the at least two terminals; A remote conference system comprising: a selection means for selecting, when the voice of a first terminal overlaps at least a part of the voice of a second terminal, either first character information obtained by voice recognition of the voice of the first terminal or second character information obtained by voice recognition of the voice of the second terminal.

[0097] According to this aspect, when the voice of the first terminal overlaps with at least a part of the voice of the second terminal, either the first character information obtained by voice recognition of the voice of the first terminal or the second character information obtained by voice recognition of the voice of the first terminal is selected. Therefore, since either one of the multiple pieces of character information obtained by voice recognition of the multiple voices to be overlapped is selected, it is possible to provide a remote conference system that can select an appropriate voice from the voices input to the remote conference.

[0098] (Item 2) The remote conference system according to item 1, wherein the selection unit selects one of the first character information and the second character information when the first character information and the second character information are similar to each other.

[0099] According to this aspect, when the first character information and the second character information are similar, one of them is selected, so that it is possible to prevent the same character information from being duplicated.

[0100] (Item 3) The remote conference system described in Item 1 or 2, wherein the selection means selects either the first character information or the second character information when the volume of the voice of one of the first terminal and the second terminal is below a predetermined threshold and the volume of the voice of the other terminal is greater than a predetermined threshold.

[0101] According to this aspect, there are cases where one of the overlapping voices input has a volume unrelated to the remote conference, and in this case, it is statistically known that the volume of the voice unrelated to the remote conference is low. Therefore, when the volume of the voice of one of the first terminal and the second terminal is equal to or lower than a predetermined threshold and the volume of the other voice is higher than a predetermined threshold, either the first character information or the second character information is selected, so that the character information of the voice unrelated to the conference is not selected.

[0102] (Item 4) The remote conference system described in Item 2 or 3, wherein the selection means selects the first character information when the volume of the voice of the first terminal is greater than the volume of the voice of the second terminal, out of the first character information and the second character information.

[0103] According to this aspect, the first character information is selected when the volume of the voice of the first terminal is larger than the volume of the voice of the second terminal, so that selection of the first character information is easy.

[0104] (Item 5) The remote conference system described in any one of Items 2 to 4, wherein the selection means selects the first character information when the timing at which the voice of the first terminal is input is earlier than the timing at which the voice of the second terminal is input, out of the first character information and the second character information.

[0105] According to this aspect, the first character information is selected when the timing at which the voice of the first terminal is input is earlier than the timing at which the voice of the second terminal is input, so that selection of the first character information is easy.

[0106] (Item 6) The remote conference system according to any one of items 2 to 5, wherein the selection unit selects one of the first character information and the second character information, which has a higher evaluation based on history information of past speech recognition.

[0107] According to this aspect, the voice with the higher evaluation based on the history information of past voice recognition is selected, so that voice unrelated to the remote conference is prevented from being selected.

[0108] (Item 7) A speaker identification means for identifying a speaker of a voice; and an association means for associating character information obtained by speech recognition with the speaker, The remote conference system according to any one of items 1 to 6, wherein when the selection means selects either the first character information or the second character information, the associating means does not associate the character information selected by the selection means with a speaker.

[0109] According to this aspect, since the speaker is not associated with the text information, it is possible to prevent the wrong speaker from being associated.

[0110] (Item 8) The remote conference system according to any one of items 1 to 7, further comprising a distribution unit that distributes the audio of each of the at least two terminals to the other terminals.

[0111] (Item 9) A character information generating method executed by a computer having at least two terminals communicably connected thereto, comprising: a speech recognition step of recognizing speech of each of the at least two terminals; and a selection step of, when a voice of a first terminal and a voice of a second terminal overlap, performing voice recognition on the voice of the first terminal and selecting either first character information or second character information obtained by voice recognition of the voice of the first terminal, or both.

[0112] According to this aspect, it is possible to provide a text information generating method capable of selecting an appropriate voice from voices input to a remote conference.

[0113] (Item 10) A character information generating program executed on a computer having at least two terminals communicably connected thereto, a speech recognition step of recognizing speech of each of the at least two terminals; and a selection step of selecting, when a voice of a first terminal and a voice of a second terminal overlap, first character information obtained by voice recognition of the voice of the first terminal and second character information obtained by voice recognition of the voice of the first terminal, or both of the first character information and the second character information.

[0114] According to this aspect, it is possible to provide a character information generating program capable of selecting an appropriate voice from voices input to a remote conference.

[0115] The embodiments disclosed herein should be considered to be illustrative and not restrictive in all respects. The scope of the present invention is defined by the claims, not the above description, and is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0116] 1 Remote conference system, 100 Server, PC200-1~N, 5 Internet, 11 Voice receiving unit, 13 Voice transfer unit, 15 Voice recognition unit, 17 Selection unit, 19 Conversation information transmission unit, 21 Conversation history storage unit, 23 Similarity determination unit, 25 Volume comparison unit, 27 History reference unit, 101 CPU, 102 ROM, 103 RAM, 104 HDD, 105 Communication unit, 106 Display unit, 107 Operation unit, 110 External storage device, 111 CD-ROM, 201 CPU, 202 ROM, 203 RAM, 204 HDD, 205 Communication unit, 206 Display unit, 207 Operation unit, 208 Camera, 209 Speaker, 210 Microphone, 211 External storage device, 211A CD-ROM, 212 GPS receiver, 213 Bus, 251 Terminal side transmission unit, 253 Voice input control unit, 255 Terminal side receiving unit, 257 voice output control unit, 259 text information receiving unit, 261 text information display unit.

Claims

1. In a remote conference system in which at least two terminals are communicatively connected and audio inputted at the terminals is distributed to the other terminals, A voice recognition means for recognizing voices of each of the at least two terminals; A remote conference system comprising: a selection means for selecting, when the voice of a first terminal overlaps with at least a portion of the voice of a second terminal, either first character information obtained by voice recognition of the voice of the first terminal or second character information obtained by voice recognition of the voice of the second terminal.

2. The remote conference system according to claim 1 , wherein the selection unit selects one of the first character information and the second character information when the first character information and the second character information are similar to each other.

3. 2. The remote conference system according to claim 1, wherein the selection means selects either the first character information or the second character information when the volume of the voice of one of the first terminal and the second terminal is below a predetermined threshold and the volume of the voice of the other terminal is greater than a predetermined threshold.

4. 4. The remote conference system according to claim 2, wherein the selection means selects the first character information when a volume of the voice of the first terminal is greater than a volume of the voice of the second terminal, out of the first character information and the second character information.

5. 4. The remote conference system according to claim 2, wherein the selection means selects the first character information when the timing at which the voice of the first terminal is input is earlier than the timing at which the voice of the second terminal is input, out of the first character information and the second character information.

6. 4. The remote conference system according to claim 2, wherein the selection unit selects one of the first character information and the second character information that has a higher evaluation based on history information of past speech recognition.

7. A speaker identification means for identifying a speaker of the voice; and an association means for associating character information obtained by speech recognition with the speaker, A remote conference system as described in any one of claims 1 to 3, wherein the associating means does not associate a speaker with the character information selected by the selection means when either the first character information or the second character information is selected by the selection means.

8. 4. The remote conference system according to claim 1, further comprising a distribution unit for distributing the audio of each of said at least two terminals to another terminal.

9. A character information generating method executed by a computer having at least two terminals communicably connected thereto, comprising: a speech recognition step of recognizing speech of each of the at least two terminals; and a selection step of, when a voice of a first terminal and a voice of a second terminal overlap, performing voice recognition on the voice of the first terminal and selecting either or both of first character information and second character information obtained by voice recognition of the voice of the first terminal.

10. A character information generating program executed on a computer to which at least two terminals are communicatively connected, a speech recognition step of recognizing speech of each of the at least two terminals; and a selection step of selecting, when the voice of a first terminal and the voice of a second terminal overlap, either first character information obtained by voice recognition of the voice of the first terminal or second character information obtained by voice recognition of the voice of the first terminal, or both of the first character information and the second character information.

Images