Interpretation relay server and interpretation relay program

Abstract

This will allow the interpreter's audio to be transmitted only to the devices of attendees present at the venue. [Solution] The interpretation relay server of this embodiment has a communication unit and a control unit. The communication unit receives a first sound transmitted from a first device, a second sound transmitted from a second device different from the first device, and a third sound transmitted from a third device. The control unit compares the first sound and the second sound, and if it determines that the first sound and the second sound are a match, it transmits the third sound to the first device.

Description

Technical Field

[0001] Embodiments of the present invention relate to an interpretation relay server and an interpretation relay program.

Background Art

[0002] Conventionally, at conferences and seminars, attendees were lent transceivers, and the attendees listened to the interpreted speech of the lecture through the lent transceivers. However, if one had the lent transceiver, even a person not present at the venue might be able to listen to the content of the conference or seminar.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The problem to be solved by the embodiments of the present invention is to provide a technique that enables transmitting the voice of an interpreter only to the terminals of the attendees present at the venue.

Means for Solving the Problems

[0005] The interpretation relay server of the embodiment includes a communication unit and a control unit. The communication unit receives a first sound transmitted from a first device, a second sound transmitted from a second device different from the first device, and a third sound transmitted from a third device. The control unit compares the first sound and the second sound, and when it is determined that the first sound and the second sound match, transmits the third sound to the first device.

Brief Description of the Drawings

[0006] [Figure 1]Figure 1 is a block diagram illustrating a schematic configuration example of an interpretation relay system according to an embodiment. [Figure 2] Figure 2 is a block diagram illustrating a schematic configuration example of an interpretation relay server according to the embodiment. [Figure 3] Figure 3 is a sequence diagram illustrating an example of the operation of the interpretation relay system according to the embodiment. [Figure 4] Figure 4 is a flowchart illustrating an example of the operation of the interpretation relay server according to the embodiment. [Figure 5] Figure 5 is a block diagram illustrating a schematic configuration example of an interpretation relay system according to the second embodiment. [Figure 6] Figure 6 is a schematic block diagram showing an example configuration of a generated speaker according to the second embodiment. [Figure 7] Figure 7 is a sequence diagram illustrating an example of operation of the interpretation relay system according to the second embodiment. [Figure 8] Figure 8 is a flowchart illustrating an example of the operation of the interpretation relay server according to the second embodiment. [Modes for carrying out the invention]

[0007] (First embodiment) The first embodiment will be described below with reference to the drawings. Figure 1 is a block diagram showing a schematic configuration example of the interpretation relay system 1. The interpretation relay system 1 includes a server 2, a speaker terminal 3, an interpreter terminal 4, and a visitor terminal 5. The speaker terminal 3, the interpreter terminal 4, and the visitor terminal 5 communicate with the server 2 via a network 6. The network 6 may be a wireless communication network or a wired communication network. The network 6 may be composed of a combination of multiple networks.

[0008] Server 2 is an information processing device such as a server device. Based on the speaker's voice (second sound) received from speaker terminal 3 (second device) and the venue sound (first sound) received from visitor terminal 5 (first device), Server 2 transmits the interpreter's voice (third sound) received from interpreter terminal 4 (third device) to visitor terminal 5.

[0009] Speaker terminal 3 acquires the speaker's audio. For example, the speaker's audio is the speaker's voice during the presentation. Speaker terminal 3 is assumed to be a PC that communicates with a microphone used by the speaker in a seminar venue, but is not limited to this.

[0010] Interpreter terminal 4 acquires the interpreter's voice. For example, the interpreter's voice is the voice of the interpreter during a lecture. Interpreter terminal 4 is envisioned as a PC communicating with a microphone used by an interpreter in a seminar, but is not limited to this.

[0011] Visitor terminal 5 acquires venue audio. For example, venue audio is the sound of the seminar venue. Visitor terminal 5 is assumed to be a smart device of a visitor attending the seminar venue, but is not limited to this. Visitor terminal 5 can be any device that can communicate with server 2 via network 6 and has a microphone that can acquire external sound, such as a tablet or a laptop computer. It is also preferable that visitor terminal 5 is a device that can connect earphones or headphones.

[0012] Next, an example of the configuration of Server 2 will be described. Figure 2 is a block diagram illustrating a schematic example of the configuration of Server 2. Server 2 includes a processor 21, system memory 22, storage 23, communication unit 24, and the like.

[0013] The processor 21 performs control and information processing for each component. For example, the processor 21 is a CPU. The processor 21 executes programs stored in the system memory 22 or storage 23. By executing programs, the processor 21 outputs operation instructions to each component and processes various information from each component.

[0014] The processor 21 compares the speaker's voice obtained from the speaker terminal 3 with the venue's voice obtained from the attendee terminal 5 to determine whether the speaker's voice and the attendee's voice match. If an attendee is present in the venue, the speaker's voice is input to the attendee terminal 5, so the venue's voice is correlated with the speaker's voice and has a high degree of similarity. If an attendee is not present in the venue, the speaker's voice is not input to the attendee terminal 5, so the venue's voice is not correlated with the speaker's voice and has a low degree of similarity. The processor 21 determines that the speaker's voice and the attendee's voice match if similarity is found between them. The processor 21 determines that the speaker's voice and the attendee's voice do not match if no similarity is found between them. For example, the cross-correlation method or the dynamic time stretching method may be used for comparing the voices, but is not limited to these.

[0015] The cross-correlation method is a technique for evaluating the similarity between two types of time series data. It quantifies the degree of similarity between two time series data signals, taking time lags into account, and evaluates the similarity. Specifically, it calculates the similarity at each shift while shifting one of the two time series data sets over time. This method allows for the identification of the lag that yields the highest correlation, i.e., the comparison starting point, even when there is a time lag. In one embodiment, for example, when server 2 constantly receives and compares speaker and audience audio, the comparison can be performed without problems even if there is a time lag in the timing of receiving both audio signals.

[0016] The dynamic time warping method is a technique for evaluating the similarity between two types of time series data. The dynamic time warping method is effective when comparing time series data recorded at different lengths and speeds, and evaluates the overall similarity by finding the optimal matching between each point of the two time series data. Specifically, the distance between each point of the two time series data is calculated, a distance matrix is created, and a path with the minimum total distance within the distance matrix is searched. This path indicates the optimal alignment of the two time series data. Then, the sum of the distances along the path is calculated, and this is taken as the similarity between the two time series data. In an embodiment, for example, when the server 2 receives and compares the speaker voice and the venue voice at a fixed timing and the data for a certain time period, even if the base data becomes different in length or speed due to some obstacle, it can be compared without problem.

[0017] The similarity between the speaker voice and the venue voice is measured using the cross-correlation method or the dynamic time warping method. Then, the server 2 determines whether the speaker voice and the venue voice match based on the similarity between the speaker voice and the venue voice and the preset threshold and determination criteria according to the method used for comparison. For example, the threshold may be set by the administrator of the server or by the operator of the seminar.

[0018] The system memory 22 includes memories such as RAM (Random Access Memory), ROM (Read Only Memory), and NVM. RAM functions as a working memory or a buffer memory. ROM is a non-rewritable non-volatile memory. ROM functions as a program memory. NVM is a rewritable non-volatile memory. NVM stores data such as programs, control data, and setting information.

[0019] The storage 23 is composed of a rewritable non-volatile memory. For example, the storage 23 is composed of a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

[0020] The communication unit 24 is an interface for communicating (receiving and transmitting) with an external device. For example, the communication unit 24 includes an interface for communicating with the speaker terminal 3. For example, the communication unit 24 includes an interface for communicating with the interpreter terminal 4. For example, the communication unit 24 includes an interface for communicating with the visitor terminal 5. The communication unit 24 may communicate either wired or wirelessly.

[0021] Next, the operation of the entire interpretation relay system 1 including the server 2, the speaker terminal 3, the interpreter terminal 4, and the visitor terminal 5 according to the embodiment will be described. FIG. 3 is a sequence diagram for explaining an operation example of the interpretation relay system 1 including the server 2, the speaker terminal 3, the interpreter terminal 4, and the visitor terminal 5 according to the embodiment.

[0022] First, the server 2 is connected to the speaker terminal 3 by a user (for example, a seminar operator) (act1), and the speaker voice is transmitted from the speaker terminal 3 to the server 2 (act2). The speaker voice may be constantly transmitted from the speaker terminal 3 to the server 2, may be transmitted only while the speaker is speaking, or may be transmitted for a certain period of time at a certain timing.

[0023] Next, the server 2 is connected to the interpreter terminal 4 by a user (for example, a seminar operator) (act3), and the interpreter voice is transmitted from the interpreter terminal 4 to the server 2 (act4). The interpreter voice may be constantly transmitted from the interpreter terminal 4 to the server 2, or may be transmitted only while the interpreter is speaking. Note that the order of connecting to the server 2 may be either the speaker terminal 3 or the interpreter terminal 4 first.

[0024] Next, Server 2 is connected to the visitor terminal 5 by the user (seminar attendee) (act 5), and the visitor terminal 5 begins to transmit venue audio (act 6). In this embodiment, as a method for connecting the visitor terminal 5 to Server 2, the seminar organizer displays a QR code (registered trademark) on a screen in the venue, and the attendee connects to Server 2 by scanning the QR code with the camera on the visitor terminal 5. Multiple visitor terminals 5 connected to Server 2 are identified by their IP addresses, but the method of identifying the visitor terminals 5 is not limited to this.

[0025] When the seminar begins, Server 2 compares the speaker's audio transmitted from Speaker Terminal 3 with the venue audio (second sound) transmitted from Attendee Terminal 5 (Act 7). If the speaker's audio and the venue audio (first sound) match, Server 2 transmits the interpreter's audio (third sound) transmitted from Interpreter Terminal 4 to Attendee Terminal 5 (Act 8). If the speaker's audio and the venue audio do not match, Server 2 does not transmit the interpreter's audio to Attendee Terminal 5 and instead sends an error notification to Attendee Terminal 5 (Act 9). Alternatively, instead of sending an error notification, the system may continue the comparison until the speaker's audio and the venue audio match. Server 2 may be configured to constantly compare the speaker's audio and the venue audio, or to do so at regular intervals.

[0026] Refer to Figure 4 to explain the operation of Server 2. Figure 4 is a flowchart illustrating an example of the operation of the interpretation relay server 2 according to this embodiment. Here, it is assumed that the speaker terminal 3, interpreter terminal 4, and visitor terminal 5 connected to Server 2 are always transmitting audio to Server 2.

[0027] Server 2 receives the speaker's audio transmitted from the speaker's terminal 3 (act101). Server 2 also receives the interpreter's audio transmitted from the interpreter's terminal 4 (act102). The order in which Server 2 receives the speaker's audio and the interpreter's audio does not matter.

[0028] Server 2 determines whether visitor terminal 5 is connected or not (act 103). For example, Server 2 determines that visitor terminal 5 is connected to Server 2 by detecting the signal sent when visitor terminal 5 connects to Server 2. For example, Server 2 sends a signal requesting a response to the connected visitor terminal 5 and determines that visitor terminal 5 is connected to Server 2 by detecting the response signal sent from visitor terminal 5. For example, if Server 2 does not detect a signal sent from visitor terminal 5, it determines that visitor terminal 5 is not connected to Server 2.

[0029] If visitor terminal 5 is not connected to server 2 (act103, No), server 2 waits until a signal is transmitted from visitor terminal 5. If visitor terminal 5 is connected to server 2 (act103, Yes), server 2 receives the venue audio transmitted from visitor terminal 5 (act104).

[0030] Next, Server 2 compares the speaker's voice received from Speaker Terminal 3 with the venue's voice received from Visitor Terminal 5 to determine if the speaker's voice and the venue's voice match (act105). For example, when using the cross-correlation method, Server 2 converts the speaker's voice and the venue's voice into digital signals and performs noise reduction and filtering as needed. Then, it calculates the correlation between one signal and the other signal while shifting one signal in time. From the calculation results, it detects the peak, which is the point where the correlation between the signals is highest, and evaluates the similarity between the speaker's voice and the venue's voice based on the detected peak value. The higher the peak value, the higher the similarity between the signals is judged to be.

[0031] For example, when using dynamic time stretching, Server 2 converts the speaker's voice and the audience's voice into digital signals and performs noise reduction and filtering as needed. Then, it calculates the distance between each point of the two signals by brute force and finds the path with the smallest distance from all possible patterns. From the calculation results, it detects the most similar path between the signals. Based on the distance of the detected path, it evaluates the similarity between the speaker's voice and the audience's voice. The smaller the distance, the higher the similarity between the signals is judged to be.

[0032] If the similarity between the speaker's voice and the venue's voice is low and it is determined that the speaker's voice and the venue's voice do not match (act105, Error), the process returns to act103 and the comparison between the speaker's voice and the venue's voice is repeated. When it is determined that the speaker's voice and the venue's voice do not match, server 2 may send an error notification to the visitor's terminal 5. If the interpreter's voice has been sent to the visitor's terminal 5 at this stage, server 2 stops sending the interpreter's voice to the visitor's terminal 5.

[0033] If the similarity between the speaker's voice and the venue's voice is high and it is determined that the speaker's voice and the venue's voice are a match (act105, Pass), server 2 transmits the interpreter's voice received from interpreter's terminal 4 to visitor terminal 5 (act106).

[0034] Server 2 determines whether a predetermined amount of time has elapsed since transmitting the interpreter's audio to the visitor terminal 5 (act107). If the predetermined amount of time has not elapsed (act107, No), Server 2 waits until the predetermined amount of time has elapsed. If the predetermined amount of time has elapsed (act107, Yes), Server 2 compares the speaker's audio and the venue audio again. Note that Server 2 may perform this process uniformly at a predetermined timing for multiple visitor terminals 5 connected to Server 2, rather than determining whether a predetermined amount of time has elapsed since transmitting the interpreter's audio to the visitor terminal 5.

[0035] The interpretation relay system 1 configured as described above allows for the transmission of the interpreter's voice to the visitor terminal 5 when the visitor is present at the venue, and not to transmit the interpreter's voice to the visitor terminal 5 when the visitor is not present at the venue. Furthermore, because the interpreter's voice is transmitted to the visitor terminal 5, the costs associated with lending out and collecting transceivers, as in conventional systems, can be reduced.

[0036] (Second embodiment) The second embodiment will be described below with reference to the drawings. Figure 5 is a schematic block diagram showing an example configuration of the interpretation relay system 7 of the second embodiment. In the first embodiment, the interpreter's voice was transmitted to the visitor terminal 5 based on the result of comparing the speaker's voice with the venue's voice, but in the second embodiment, the interpreter's voice is transmitted to the visitor terminal 5 based on the result of comparing a special sound with the venue's voice.

[0037] The interpretation relay system 7 includes a server 2, an interpreter terminal 4, a visitor terminal 5, and a generating speaker 8. The interpreter terminal 4, visitor terminal 5, and generating speaker 8 communicate with the server 2 via a network 6. Network 6 may be a wireless communication network or a wired communication network. Network 6 may be composed of a combination of multiple networks. The interpreter terminal 4 and visitor terminal 5 have the same configuration as in the first embodiment, so their description is omitted.

[0038] Server 2 is an information processing device such as the Server 2 device. Based on the special sound generated by the generating speaker 8 (special sound generating device) and the venue audio received from the visitor terminal 5, Server 2 transmits the interpreter's audio received from the interpreter terminal 4 to the visitor terminal 5.

[0039] The generating speaker 8 is a speaker that generates special sounds and emits the generated special sounds. The generating speaker 8 also transmits the generated special sounds to the server 2. For example, the special sounds are inaudible sounds, which are sounds with frequencies that are generally not perceptible to humans, but are not limited to this. For example, the special sounds could be background music (BGM) played in the venue (lecture hall).

[0040] An example of the configuration of Server 2 in the second embodiment will be described. Server 2 includes a processor 21, system memory 22, storage 23, communication unit 24, etc. The system memory 22 and storage 23 have the same configuration as in the first embodiment, so their description will be omitted.

[0041] The processor 21 compares the special sound obtained from the generating speaker 8 with the venue audio obtained from the visitor terminal 5 to determine whether the special sound and the visitor audio match. Here, if a visitor is present at the venue, the special sound from the generating speaker 8 is input to the visitor terminal 5, so the venue audio is correlated with the special sound and will be audio with a high degree of similarity. If a visitor is not present at the venue, the special sound from the generating speaker 8 is not input to the visitor terminal 5, so the venue audio is not correlated with the special sound and will be audio with a low degree of similarity. If similarity is found between the special sound and the venue audio, the processor 21 determines that the special sound and the visitor audio match. If similarity is not found between the special sound and the venue audio, the processor 21 determines that the special sound and the visitor audio do not match. For comparing the audio, the cross-correlation method or the dynamic time stretching method is used, as in the first embodiment, but is not limited to these.

[0042] The communication unit 24 is an interface for communicating with external devices. For example, the communication unit 24 includes an interface for communicating with the interpreter terminal 4. For example, the communication unit 24 includes an interface for communicating with the visitor terminal 5. For example, the communication unit 24 includes an interface for communicating with the generating speaker 8. The communication unit 24 may communicate via a wired connection or wireless connection.

[0043] Next, an example of the configuration of the generating speaker 8 will be described. Figure 6 is a block diagram showing a schematic of an example configuration of the generating speaker 8. The generating speaker 8 includes a processor 81, system memory 82, storage 83, communication unit 84, generation unit 85, output unit 86, and the like.

[0044] The processor 81 performs control and information processing for each component. For example, the processor 81 is a CPU. The processor 81 executes programs stored in the system memory 82 or storage 83. By executing programs, the processor 81 outputs operation instructions to each component and processes various information from each component.

[0045] System memory 82 includes RAM (Random Access Memory), ROM (Read-Only Memory), and NVM. RAM functions as working memory or buffer memory. ROM is non-rewritable, non-volatile memory. ROM functions as program memory. NVM is rewritable, non-volatile memory. NVM stores data such as programs, control data, and configuration information.

[0046] Storage 83 consists of rewritable, non-volatile memory. For example, storage 83 consists of a storage device such as an HDD (hard disk drive) or an SSD (solid state drive).

[0047] The communication unit 84 is an interface for communicating (receiving and transmitting) with external devices. For example, the communication unit 84 includes an interface for communicating with server 2. The communication unit 84 may communicate via a wired connection or wirelessly. The communication unit 84 transmits the sound generated by the generation unit 85.

[0048] The generation unit 85 generates special sounds. In this embodiment, the generation unit 85 generates ultra-low frequencies below 20 Hz, which are inaudible to the average human, or ultrasonic frequencies above 20 kHz.

[0049] The output unit 86 is a speaker that generates special sounds as sound.

[0050] Next, the operation of the entire interpretation relay system 7, including the server 2, generating speaker 8, interpreter terminal 4, and visitor terminal 5 according to the second embodiment, will be described. Figure 7 is a sequence diagram illustrating an example of the operation of the interpretation relay system 7, including the server 2, generating speaker 8, interpreter terminal 4, and visitor terminal 5 according to the second embodiment.

[0051] First, Server 2 is connected to the generating speaker 8 by the user (e.g., the seminar organizer) (act 11), and the generating speaker 8 begins to transmit special sounds (act 12). Furthermore, the special sounds are generated within the venue (act 13). The special sounds may be transmitted from the generating speaker 8 to Server 2 continuously, or they may be transmitted only at certain times for a certain period of time.

[0052] Next, Server 2 is connected to the interpreter terminal 4 by the user (e.g., the seminar organizer) (act 14), and the interpreter's voice is transmitted from the interpreter terminal 4 (act 15). The interpreter's voice may be transmitted from the interpreter terminal 4 to Server 2 at all times, or it may be transmitted only while the interpreter is speaking. The order in which the generating speaker 8 and the interpreter terminal 4 connect to Server 2 does not matter.

[0053] Next, Server 2 is connected to visitor terminal 5 by the user (seminar attendee) (act 16), and venue audio is transmitted from visitor terminal 5 (act 17).

[0054] When the seminar begins, Server 2 compares the special sound (second sound) generated by the generation unit 85 transmitted from the communication unit 84 with the venue sound (first sound) transmitted from the visitor terminal 5 (act 18). If the special sound and the venue sound match, Server 2 transmits the interpreter's voice (third sound) transmitted from the interpreter's terminal 4 to the visitor terminal 5 (act 19). In this case, a match means that the venue sound contains something equivalent to the special sound. Other sounds may also be included. If the special sound and the venue sound do not match, Server 2 does not transmit the interpreter's voice to the visitor terminal 5 and instead sends an error notification to the visitor terminal 5 (act 20). Alternatively, instead of sending an error notification, the system may continue the comparison until the special sound and the venue sound match. Server 2 may be configured to constantly compare the special sound and the venue sound, or to perform the comparison at regular intervals.

[0055] Referring to Figure 8, the operation of Server 2 in the second embodiment will be explained. Figure 8 is a flowchart illustrating an example of the operation of the interpretation relay server 2 according to the second embodiment. Note that the processes numbered in Figure 8 in the same way as in Figure 4 are the same as the processes in the first embodiment and will not be explained.

[0056] First, Server 2 receives the special sound transmitted from the generating speaker 8 (act201). The order in which Server 2 receives the special sound and the interpreter's voice does not matter.

[0057] When Server 2 receives venue audio transmitted from Visitor Terminal 5 (act 104), it compares the special sound received from Generator Speaker 8 with the venue audio received from Visitor Terminal 5 and determines whether the special sound and the venue audio match (act 205). Similar to the first embodiment, the comparison between the special sound and the venue audio uses either the cross-correlation method or the dynamic time stretching method.

[0058] In the interpretation relay system 7 of the second embodiment configured as described above, similar to the first embodiment, it is possible to transmit the interpreter's voice to the visitor terminal 5 when the visitor is present in the venue, and not transmit the interpreter's voice to the visitor terminal 5 when the visitor is not present in the venue.

[0059] (modified version) In the second embodiment, a special sound generated by the generating speaker 8 was used for comparison with the venue audio, but a special sound pre-stored in the server 2 may also be used. Alternatively, the server 2 may be provided with a generation unit 85, and a special sound generated by the server 2 may be used for comparison with the venue audio. In a configuration where the server 2 stores special sounds in advance, or where the server 2 generates special sounds, the speaker does not need to generate the special sound; the speaker only needs to generate the special sound transmitted from the server 2.

[0060] The program executed on server 2 in the above-described embodiment is provided pre-installed in ROM, storage, etc. The program executed on server 2 in the above-described embodiment may also be provided as an installable or executable file recorded on a computer-readable recording medium such as a CD-ROM, flexible disk (FD), CD-R, or DVD (Digital Versatile Disk).

[0061] Furthermore, the program executed on the server 2 of the above-described embodiment may be stored on a computer connected to a network such as the Internet and provided by downloading it via the network. Alternatively, the program executed on the server of the above-described embodiment may be provided or distributed via a network such as the Internet.

[0062] Although several embodiments of the present invention have been described above, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. [Explanation of Symbols]

[0063] 1. Interpretation relay system 2. Server 3. Speaker's terminal 4. Interpreter terminal 5. Visitor terminals 6. Network 21... Processor 22... System Memory 23.. Storage 24... Communications Department 8. Generating speaker 85...Generation section

Claims

1. A communication unit that receives a first sound transmitted from a first device, a second sound transmitted from a second device different from the first device, and a third sound transmitted from a third device. A control unit that compares the first sound and the second sound and, if it is determined that the first sound and the second sound are a match, transmits the third sound to the first device, An interpreter relay server that has the following features.

2. The second sound is audio data of the speaker's voice from the second device, which is the speaker device. The interpretation relay server according to claim 1.

3. The second sound is audio data of a special sound generated by the special sound generation device, which is the first device, placed in the lecture hall. The interpretation relay server according to claim 1.

4. To the server A receiving function for a first sound transmitted from a first device, a second sound transmitted from a second device different from the first device, and a third sound transmitted from a third device. An interpreter relay program that compares the first sound and the second sound, and if it is determined that the first sound and the second sound are a match, enables the first device to perform the function of transmitting the third sound.

5. The second sound is audio data of the speaker's voice from the second device, which is the speaker device. The interpretation relay program according to claim 4.

6. The second sound is audio data of a special sound generated by the special sound generation device, which is the second device, placed in the lecture hall. The interpretation relay program according to claim 4.

Images