Multilingual audio distribution method

Abstract

To provide a multilingual voice distribution method capable of translating a voice uttered by a speaker into other languages, minimizing the occurrence of a time lag accompanying the translation, and transmitting the voice to a plurality of recipients without delay. [Solution] Speech uttered by a speaker is input to a speech terminal 1 and transmitted with a PTT (Press to Talk) button held down. The speaker's speech is input to a translation engine 3 and converted to text by a text conversion unit 3a. The text converted from the speech is translated into multiple languages ​​by a multilingual translation unit 3b and converted into speech in each language by a speech synthesis engine unit 3c for each language. The converted multilingual speech is input to a speech distribution server 4, and the multilingual speech is distributed in a preselected language by a speech distribution unit 4a, and the translated and distributed speech is received by a listening terminal 5 and transmitted to the listener.

Description

[Technical field]

[0001] The present invention relates to a multilingual voice distribution method for transmitting a voice uttered by a speaker to a listener in the form of a voice translated into multiple languages. [Background technology]

[0002] With the recent increase in inbound demand, it is becoming common to use translation software to translate necessary conversations for travel and daily life into other languages. In addition, with the increase in foreign workers, it has become difficult to give work instructions in Japanese alone at the workplace, and there is a demand for tools that can smoothly convert work instructions from work managers to workers into a language that the workers can understand.

[0003] However, if it takes time for the work manager to convey instructions to the workers, it leads to a decrease in work efficiency.In addition, when many workers who speak different native languages ​​work together, it leads to confusion at the work site if information is not conveyed in each language.

[0004] In addition, with the recent increase in natural disasters such as floods and earthquakes, local governments are using disaster prevention radio broadcasting systems. However, the language used here is basically only Japanese, and with the recent increase in foreign tourists, they are unable to understand the content of the disaster prevention broadcasts, which has become a problem as the disaster prevention radio broadcasting systems are not functioning properly.

[0005] The same situation occurs when trains are delayed or cancelled, and emergency announcements on board trains and at stations are made only in Japanese, meaning that information is not adequately conveyed to foreigners.

[0006] When work instructions are transmitted from a work manager to multiple workers via communication, the communication takes the form of one-to-many group simultaneous voice communication. Patent document 1 discloses an example of technology related to a communication system that realizes one-to-many group simultaneous voice communication. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Patent No. 6318292 Summary of the Invention [Problem to be solved by the invention]

[0008] When applying this one-to-many group voice communication to cases such as a manager giving instructions to many workers who speak different native languages, it is necessary for the translated instructions to be transmitted from the speaker to the receiver without delay, minimizing the time lag that occurs with translation. The same is true for providing disaster prevention information to foreign tourists when a disaster occurs, or for providing train operation information when an emergency occurs.

[0009] In addition, various environments are assumed for the use of one-to-many group simultaneous voice communication. For example, there may be a crowded place such as a station, where there is a lot of noise, a construction site where special machine sounds are generated, or a heavy rainfall situation such as a disaster radio application. In such various usage situations, if the surrounding noise cannot be appropriately removed, accurate information cannot be transmitted.

[0010] Furthermore, each speaker's voice is unique, and it may not always be easy to hear, or it may be difficult to hear, so even in such cases, it is necessary to be able to transmit information accurately. Also, there is a possibility of echoes occurring due to the proximity of the microphone and speaker. Furthermore, packet loss may occur due to degradation of the mobile network environment.

[0011] The present invention has been made in consideration of the above circumstances, and aims to provide a multilingual voice distribution method that can translate a speaker's voice into another language, minimize the time lag associated with translation, and transmit the translated voice to a plurality of recipients without delay, and that can improve the accuracy of voice translation to provide accurate translation even when the speaker's voice is difficult to hear or there is noise around the speaker. [Means for solving the problem]

[0012] In order to solve the above problems, the multilingual voice distribution method of the present invention is characterized in that voice uttered by a speaker is transmitted from the speaking terminal while the PTT button of the speaking terminal is kept pressed, the speaker's voice transmitted from the speaking terminal is input to a translation engine and converted into text, the converted text is translated into multiple languages, the multilingual translated text is converted into voice in each language by a voice synthesis engine of each language, the converted multilingual voice is input to a voice distribution server, the multilingual voice is distributed by a voice distribution unit in a language previously selected and set by the listener, and this translated and distributed voice is received by a listening terminal and transmitted to the listener.

[0013] The speaker's voice transmitted from the speaking terminal is input to a translation engine and converted into text, the converted text is translated into multiple languages, the multilingual translated text is converted into voice in each language by a speech synthesis engine for each language, the multilingual translated text is converted into voice in each language by a speech synthesis engine for each language, and the converted multilingual voice is input to an audio distribution server, and the audio distribution unit distributes the multilingual voice in the language previously selected and set by the listener, so that the listener can receive the voice in the translated language and does not need to perform any operations as when using a translation app.

[0014] In particular, the voice of the speaker is transmitted from the speaking terminal while the PTT button of the speaking terminal is held down, so the voice of the speaker is input into the translation engine sequentially. Therefore, there is almost no time lag between the speaker and the voice being sent to the receiver, which shortens the time it takes for the receiver to receive the translated voice, and the translated voice can be transmitted to multiple receivers without delay.

[0015] In the multilingual voice distribution method of the present invention, when the voice spoken by a speaker is transmitted to a receiving terminal in the same language as the speaker without translation, the speaker's voice can be input to a voice distribution server and distributed to the receiving terminal without going through a translation engine.

[0016] When the voice of a speaker is sent to a receiving terminal in the same language as the speaker without translation, the voice of the speaker is input to the voice distribution server without going through the translation engine, which reduces the load on the translation engine. Also, since conversations between people in the same language are not generated by voice synthesis, it is possible to identify who is speaking by the characteristics of the voice.

[0017] In the multilingual audio distribution method of the present invention, a sampling frequency changeover switch provided in the audio distribution unit can be used to change between a frequency used for communication to a receiving terminal that receives translated audio and a frequency used for communication to a receiving terminal that receives the speaker's audio as its actual voice without being translated.

[0018] Since the frequency used for communication with the receiving terminal that receives the translated voice is different from the frequency used for communication with the receiving terminal that receives the speaker's voice as the actual voice without translation, by switching the frequency with a sampling frequency changeover switch, it is possible to transmit the translated voice at a frequency suitable for translation, and also to transmit the actual voice at a frequency that has been used traditionally. Therefore, it is possible to achieve both the convenience of transmitting translated voice and the use of a widespread popular type.

[0019] An example of a frequency used for communication to a handset that receives the translated speech is 16 kHz, and an example of a frequency used for communication to a handset that receives the speaker's speech in its original voice without translation is 8 kHz.

[0020] In the multilingual voice distribution method of the present invention, recognition of the speaker's voice transmitted from the speech terminal and conversion of the voice to text can be performed by utilizing a web-connected voice recognition and text conversion service, and translation of the converted text into multiple languages ​​and conversion of the multilingual translated text into voice in each language can be performed by translation engines distributed and incorporated in cloud-type PCs.

[0021] If one translation engine is configured in one PC, the load on the GPU RAM is high, so it is necessary to prepare a large capacity GPU RAM. Since the process of recognizing voice and converting it to text consumes a lot of memory, the amount of RAM used can be reduced by using an internet-based voice recognition and text conversion service for that part.

[0022] As a result, the translation engine's reduced RAM capacity means it can be incorporated into cloud-based PCs, enabling distributed installation. This allows the number of users to increase, speeding up translation processing. Also, maintenance such as updates and modifications can be controlled via internet access, making it easier than with physical servers. Also, if the translation engine goes down, the system itself will not function, so by distributing it, system downtime can be prevented.

[0023] In the multilingual speech distribution method of the present invention, the speaker's speech, including environmental noise, transmitted from the speaking terminal can be adjusted by a speech adjustment means for the purpose of improving speech translation accuracy, and then input to the translation engine.

[0024] According to the basic configuration of the present invention, it is possible to realize a system in which there is almost no time lag between when a speaker sends a speech to a listener via translation, but the speaker's speech sent from a speech terminal usually contains various kinds of environmental noise, and it is necessary for the speaker's speech, including such environmental noise, to be accurately converted into text and translated into multiple languages. For this reason, the speech is adjusted by a speech adjustment means in order to improve the accuracy of speech translation, and then input to the translation engine. By performing this processing, it is possible to reduce the amount of processing on the terminal side.

[0025] In the multilingual audio distribution method of the present invention, the audio adjustment means may monitor speaker audio and microphone audio via the Internet and subtract the speaker audio from the microphone audio.

[0026] This allows the AEC (Acoustic Echo Cancellation) function to use a program via the Internet to remove echo sounds caused by the microphone and speaker being close to each other, improving the accuracy when the speaker's voice, including environmental noise, is converted into text and translated into multiple languages.

[0027] In the multilingual voice distribution method of the present invention, the voice adjustment means can be configured to provide an aperture at the installation section of an omnidirectional condenser microphone mounted on the speaking terminal, so as to function as a physical guide that makes the direction of the voice emitted by the speaker have a single directionality.

[0028] Condenser microphones have the advantage of being small and lightweight, making them easy to install in small radios, but because they are omnidirectional, they tend to pick up echoes and ambient noise. Therefore, by providing an aperture at the installation site of the condenser microphone and making the direction of the speaker's voice unidirectional, it is possible to prevent the microphone from picking up the voice from the speaker and prevent the generation of echoes. It is also possible to prevent the microphone from picking up ambient noise. This improves the accuracy of converting the speaker's voice, including environmental noise, into text and translating it into multiple languages.

[0029] In the multilingual voice distribution method of the present invention, the voice adjustment means may generate an inverse phase sound of the external noise to cancel the external noise, thereby eliminating the noise. This can be done by detecting the frequency and spectrum of the external noise.

[0030] When a speaker speaks at a workplace or the like, mechanical noise may be generated from devices installed at the workplace, and such mechanical noise often has a specific frequency and a unique spectrum. Therefore, by generating an inverse phase sound of external noise such as mechanical noise to cancel the external noise, it is possible to eliminate noise even at a workplace or the like, and improve the accuracy when the speaker's voice, including environmental noise, is converted into text and translated into multiple languages.

[0031] In the multilingual voice distribution method of the present invention, the voice adjustment means may automatically increase the volume when the voice level is low, thereby adjusting the voice to a level that is easy to translate.

[0032] When the speaker's voice level is low and buried in the surrounding noise, it is effective to use the AGC (Automatic Gain Control) function to raise the speaker's voice level so that the speaker's voice can be input at a specified sound pressure level. At the same time, the signal-to-noise ratio can be improved. This improves the accuracy when the speaker's voice, including the environmental noise, is converted into text and translated into multiple languages.

[0033] In the multilingual voice distribution method of the present invention, the voice adjustment means may modulate the pitch of the voice by an audio equalizer.

[0034] This method uses an audio equalizer capable of emphasizing high or low tones to shift to higher or lower tones in response to changes in the surrounding environment, thereby improving the probability that the translation server will accurately recognize the voice. By modulating the pitch of the voice, the speaker's voice can be easily understood, improving the accuracy of multilingual translation. In normal voice transmission, if the speaker's voice is modulated and the pitch changes, the receiver may feel uncomfortable. However, in the voice distribution method of the present invention, the receiver receives an automated voice that has been translated, so there is no discomfort caused by modulation, and the receiver can enjoy the benefits of modulation without such problems.

[0035] In the multilingual voice distribution method of the present invention, the voice adjustment means can be configured to utilize a voice codec based on voice encoding technology to complement the voice itself and restore it to normal voice when packet loss occurs due to degradation of the mobile network environment.

[0036] Since the present invention uses a mobile network, the voice transmitted from the speaker to the receiver may be degraded due to packet loss caused by degradation of the mobile network environment. In such a case, the voice itself can be complemented and restored to normal voice by using a voice codec based on voice encoding technology. This improves the accuracy when the speaker's voice is converted into text and translated into multiple languages.

[0037] In the multilingual audio distribution method of the present invention, the audio distribution unit can select either individual distribution to individual receiving terminals, group distribution to multiple receiving terminals set to belong to a specific group, or simultaneous distribution to all receiving terminals.

[0038] Since individual delivery, group delivery, or simultaneous delivery can be selected, calls can be made that suit the situation according to the recipient's language attributes and call type attributes.

[0039] In the multilingual audio distribution method of the present invention, the selection of the individual distribution, the group distribution, or the simultaneous distribution can be determined based on individual information, group information, and simultaneous information stored in a database installed in the audio distribution unit.

[0040] Since information on individual information, group information, and simultaneous information is stored in a database installed in the audio distribution unit, it is possible to select individual delivery, group delivery, or simultaneous delivery by performing mode selection, group selection, etc. based on this stored data. Information on each receiving terminal that makes up a group of receiving terminals that belong to a specific group is stored as group information.

[0041] In the multilingual voice distribution method of the present invention, language switching can be performed using dial knobs provided on the speaking terminal and the receiving terminal for switching translation languages, and the selected language can be announced by voice from the speaking terminal and the receiving terminal.

[0042] The language is switched using dial knobs on the speaking terminal and the receiving terminal, and the selected language is announced by voice, so that even if the setting allows translation into multiple languages, it is easy to know which language has been selected.

[0043] In the multilingual voice distribution method of the present invention, voice emitted by a listener and received at the listening terminal can be transmitted to a speaking terminal held by the speaker or to a listening terminal held by another listener, enabling two-way communication.

[0044] By enabling two-way communication, all receiving terminals can have the same functions as the speaking terminal, making it possible for all users who have terminals to share information. Effect of the Invention

[0045] According to the present invention, it is possible to realize a multilingual voice distribution method that can translate the speech of a speaker into other languages, minimize the time lag associated with translation, and transmit the speech to multiple listeners without delay. [Brief description of the drawings]

[0046] [Figure 1] 1 is a block diagram for implementing a multilingual audio distribution method according to an embodiment of the present invention; [Diagram 2] FIG. 1 is a diagram showing the timing of a PTT server and speech translation. [Diagram 3] FIG. 11 is a diagram for explaining a method for reducing the load on a translation engine. [Figure 4] FIG. 13 is a diagram showing a mechanism for improving speech translation accuracy by the speech adjustment means of the speech filter section. [Diagram 5] FIG. 13 is a flowchart showing a process for selecting one of individual delivery, group delivery, and simultaneous delivery. [Figure 6] FIG. 2 is a diagram illustrating an example of a database structure. [Figure 7] 3A and 3B are diagrams showing the screens and buttons of a speaking terminal and a receiving terminal. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0047] Hereinafter, a multilingual audio distribution method of the present invention will be described based on an embodiment thereof. FIG. 1 shows a block diagram for implementing a multilingual audio distribution method according to an embodiment of the present invention. The voice of the speaker is input to the speech terminal 1 held by the speaker, and is transmitted with the PTT (Press to Talk) button held down until the speaker ends the call. The speech language input to the speech terminal 1 is preset in a setting menu. The speech terminal 1 and the receiving terminal at this time may be dedicated terminals with a program mounted on a dedicated board, or may be commercially available smartphones.

[0048] The speaker's voice transmitted from the speaking terminal 1 is input to the voice distribution server 2, where noise is removed by the voice filter unit 2a, and then input to the translation engine 3, where it is converted to text by the text conversion unit 3a. The voice distribution server 2 functions as a PTT server. The text converted from the voice is translated into multiple languages ​​by the multilingual translation unit 3b. The multilingual translated text is converted into voice in each language by the voice synthesis engine unit 3c for each language. The NICT translation engine can be used as an example of the translation engine 3.

[0049] In the voice filter unit 2a, the speaker's voice, including environmental noise, transmitted from the speaking terminal 1 is adjusted by a voice adjustment means for the purpose of improving the accuracy of the voice translation. Among the voice adjustment means, those that use electrical processing are performed in the voice filter unit 2a, while those that perform voice adjustment without electrical processing are performed by their own unique methods. These voice adjustment means will be described in detail later.

[0050] The converted multilingual voice is input to voice distribution server 4, where the multilingual voice is distributed in a preselected language by voice distribution unit 4a, and this translated and distributed voice is received by each of the receiving terminals constituting receiving terminal group 10 belonging to translation group 1 and each of the receiving terminals constituting receiving terminal group 20 belonging to translation group 2, and transmitted to the listener. A different translation language can be set for each translation group, and the same language can be set within the same translation group. The speech language output to each of receiving terminal groups 10, 20 is set in advance in a setting menu. Note that voice distribution server 2 and voice distribution server 4 can be used as one server, but for convenience of explanation, they are treated as separate servers.

[0051] When the voice of a speaker is not translated and is sent to the receiving terminal in the same language as the speaker, the voice of the speaker is input as it is to the voice distribution server 4 without going through the translation engine 3, and is distributed to each receiving terminal constituting the receiving terminal group 30 belonging to the call group 3.

[0052] The voice distribution unit 4a of the voice distribution server 4 is equipped with a sampling frequency changeover switch 4b. This sampling frequency changeover switch 4b switches between the frequency used for communication with the receiver terminal that receives the translated voice and the frequency used for communication with the receiver terminal that receives the speaker's voice as it is without translation. If the sampling frequency for voice translation is not 16 kHz, the sound quality will deteriorate and accurate voice input will be impossible, but conventional wireless devices have a frequency of 8 kHz to reduce the number of packets used. Because of this, it is usually not possible to have a conversation using the same service, so a sampling frequency converter is installed in the voice distribution server 4 to enable conversation with conventional wireless devices.

[0053] The audio distribution unit 4a of the audio distribution server 4 selects one of the following: individual distribution, which transmits to individual receiving terminals; group distribution, which transmits to multiple receiving terminals that are set to belong to a specific group; and simultaneous distribution, which transmits to all receiving terminals.

[0054] The selection of individual distribution, group distribution, or simultaneous distribution is determined based on individual information, group information, and simultaneous information stored in database 5 installed in audio distribution unit 4a. Individual information, group information, and language selection information are stored in database 5, and when individual information is output from database 5, audio by individual distribution is distributed to individual receiving terminals. When group information is output from database 5, audio by group distribution is distributed to multiple receiving terminals receiving the same translation language.

[0055] Based on FIG. 2, the PTT server and the timing of speech translation will be described. When the speaker's voice is sent from the speaking terminal 1 with the PTT ON, (1) 1 2 3 4, followed by (2) 5 6 7 after a period of time, and then the PTT is turned OFF, the PTT server sequentially transmits the voice to the translation server functioning as a translation engine. In this state, the voice is transferred directly to the translation engine without any text conversion, so translation begins without waiting for the PTT to be turned OFF. The translated synthetic voice is sent to the receiving terminal, so there is almost no time lag between the speaker and the voice being sent to the receiver, and the voice can be transmitted to multiple receivers without delay.

[0056] A method for reducing the load on the translation engine will be described with reference to FIG. Figure 3 shows how the translation engine can be distributed and integrated into cloud-based PCs in conjunction with the use of web-connected speech recognition and text conversion services, in contrast to the case where the translation engine is installed on a local physical server.

[0057] If the translation engine is installed on a local physical server, 50% of the GPU RAM is used for speech recognition and speech-to-text conversion, 30% for translating text into multiple languages, and 20% for synthesizing speech from the translated text. In this way, if one translation engine is configured in one PC, the load on the GPU RAM is high, and therefore a large capacity GPU RAM must be prepared, and the cost of the PC equipped with the translation engine becomes expensive.

[0058] To solve this problem, the speaker's voice transmitted from the speech terminal 1 is recognized and the voice is converted into text by using a web-connected speech recognition and text conversion service, which can reduce the load on the GPU RAM by 50%.

[0059] This reduces the RAM capacity of the translation engine, and makes it possible to incorporate the translation engine into cloud-type PCs, such as cloud 1 and cloud 2, as shown in Figure 3, making it possible to distribute the installation. This allows the number of users to increase, and speeds up translation processing. Also, maintenance such as updates and modifications can be controlled via Internet access, making it easier than with physical servers. Also, if the translation engine goes down, the system itself will not function, so by distributing the engine, system failures can be prevented.

[0060] The significance and mechanism of improving speech translation accuracy by using a speech adjustment means will be explained with reference to FIG. In the prior art document, Japanese Patent No. 6318292, the focus is on accurate text conversion, and although there is an advantage in this respect, by converting the speech into text, it takes time to transmit the language to the translation engine, and a time lag in information transmission inevitably occurs. In contrast, when the basic configuration of the present invention is used, a major advantage is that there is almost no time lag between when the speech is transmitted from the speaker to the listener via translation. Under these circumstances, in order to ensure the accuracy of information transmission, it is important that the speech of the speaker is accurately converted into text and translated, which becomes a new problem. In order to solve this new problem, the speech is adjusted by a speech adjustment means in order to improve the accuracy of speech translation, and then input to the translation engine.

[0061] As a means of adjusting the sound, a method can be adopted in which speaker sound and microphone sound are monitored via the Internet and the speaker sound is subtracted from the microphone sound. This allows the echo sound caused by the microphone and speaker being close to each other to be removed by a program via the Internet using the AEC (Acoustic Echo Cancellation) function, improving the accuracy when the speaker's voice, including environmental noise, is converted into text and translated into multiple languages. As an example, AEC processing can be performed using WebRTC.

[0062] As an audio adjustment means, a method can be adopted in which an aperture is provided at the installation location of an omnidirectional condenser microphone mounted on the speaking terminal, so that the direction of the speaker's voice is made to have a single directivity as a physical guide.

[0063] Condenser microphones have the advantage of being small and lightweight, making them easy to install in small radios, but because they are omnidirectional, they tend to pick up echoes and ambient noise. Therefore, by providing an aperture at the installation site of the condenser microphone and making the direction of the speaker's voice unidirectional, it is possible to prevent the microphone from picking up the voice from the speaker and prevent the generation of echoes. It is also possible to prevent the microphone from picking up ambient noise. This improves the accuracy of converting the speaker's voice, including environmental noise, into text and translating it into multiple languages.

[0064] As a sound adjustment method, a method can be adopted in which an inverse phase sound is generated to cancel out the external noise, thereby eliminating the noise. This is a method called ANC (Active Noise Cancellation).

[0065] When a speaker speaks at a workplace or the like, mechanical noise may be generated from devices installed at the workplace, and such mechanical noise often has a specific frequency and a unique spectrum. Therefore, by generating an inverse phase sound of external noise such as mechanical noise to cancel the external noise, it is possible to eliminate noise even at a workplace or the like, and improve the accuracy when the speaker's voice, including environmental noise, is converted into text and translated into multiple languages.

[0066] As a sound adjustment means, a method can be adopted in which, when the sound level is low, the volume is automatically increased to adjust the sound to a level that is easy to translate.

[0067] When the speaker's voice level is low and buried in the surrounding noise, it is effective to use the AGC (Automatic Gain Control) function to raise the speaker's voice level so that the speaker can be input at a specified sound pressure level. At this time, a filter can be used to improve the signal-to-noise ratio. This improves the accuracy when the speaker's voice, including environmental noise, is converted into text and translated into multiple languages. AGC processing can be performed using WebRTC, for example, or can be performed by configuring an electrical circuit.

[0068] As a sound adjustment means, a method of modulating the pitch of the sound by an audio equalizer can be adopted.

[0069] This method uses an audio equalizer that can emphasize high or low tones, and shifts the tone to higher or lower tones in response to changes in the surrounding environment, thereby improving the probability that the translation server will accurately recognize the voice. By modulating the pitch of the voice, the speaker's voice can be easily understood, improving the accuracy of multilingual translation. In normal voice transmission, if the speaker's voice is modulated and the pitch changes, the receiver may feel uncomfortable. However, in the voice distribution method of the present invention, the receiver receives an automated voice that has been translated, so no discomfort caused by modulation occurs.

[0070] As an audio adjustment method, when packet loss occurs due to degradation of the mobile network environment, a method can be adopted in which an audio codec based on audio encoding technology is used to complement the audio itself and restore it to normal audio.

[0071] When using a mobile network, the voice transmitted from the speaker to the receiver may be degraded due to packet loss caused by degradation of the mobile network environment. In such cases, the voice itself can be complemented and restored to normal voice by using a voice codec based on voice encoding technology. As an example, this process can be performed using WebRTC. This can improve the accuracy when the speaker's voice is converted into text and translated into multiple languages.

[0072] The process of selecting one of individual delivery, group delivery, and simultaneous delivery will be described with reference to Figures 5 and 6. Figure 5(a) shows the mode selection on the side of the speaking terminal 1. When Service (simultaneous) is selected in the mode selection, the speaking terminal 1 turns on the PTT and starts speaking, and simultaneous delivery is made to all the receiving terminals in the selected language.

[0073] When Member (individual) is selected in the mode selection, a member is selected, the speaking terminal 1 turns PTT ON and starts speaking, and individual delivery is made in the selected language to the receiving terminal held by the selected Member.

[0074] When Group is selected in the mode selection, the speaking terminal 1 turns on the PTT and starts speaking, and group distribution is performed in the selected language to the listening terminals carried by the listeners who belong to the selected group.

[0075] FIG. 5(b) shows the process in the translation server. According to the mode selection on the speaking terminal 1 side, the translation server receives the voice, recognizes the voice, converts it into text, translates it into multiple languages, and performs voice synthesis in each language.

[0076] Fig. 6 shows an example of a database structure, Fig. 6(a) shows an example of specific data in the database, and Fig. 6(b) shows a schematic example of the database structure. In this example, Members 1, 2, 3, 4, 5, and 6 belong to 100Group, and Members 7, 8, 9, 10, 11, 12, and 13 belong to 200Group. In this way, information on each of the receiving terminals that make up a group of receiving terminals that belong to a specific group is stored.

[0077] The screens and buttons of the speaking terminal and the receiving terminal will be described with reference to FIG. The language is switched using a dial knob 6 provided on the speaking terminal and the receiving terminal for switching the translation language, and the selected language is announced by voice. This makes it easy to know which language has been selected, even if the setting allows translation into multiple languages.

[0078] A PTT button 7 and a mode change button 8 are attached to the sides of the speaking terminal and the receiving terminal. The PTT button 7 remains pressed while the speaker is transmitting voice. The mode change button 8 is a button for selecting individual distribution, group distribution, or simultaneous distribution, and when one of the modes is selected, the corresponding mode is clearly displayed in the M (Member), G (Group), and S (Service) displays on the front of the screen.

[0079] Currently, the translation devices that are commonly used are mostly app-type, and in the case of app-type, it is necessary to operate while looking at the screen. Therefore, especially when you are doing some work or operating a device, you cannot see the screen, and it is difficult to translate. In particular, in the case of a device with multilingual capabilities, it is necessary to operate while looking at the screen when changing the language being spoken.

[0080] However, by making it possible to switch languages ​​using the dial knob 6, even if the language being spoken changes or the person in charge is changed, the language can be switched without looking at the screen by operating the dial knob 6. Also, by having a function that pronounces "English" immediately after switching, for example, to English, it is possible to know that the language has been switched to English without even looking at the screen.

[0081] The voice emitted by the listener and received at the listening terminal is transmitted to the listening terminal held by the speaker or to the listening terminal held by another listener, enabling two-way communication. Therefore, all listening terminals can have the same functions as the listening terminal, making it possible to share information among all users holding terminals.

[0082] When a handheld type radio terminal is used as the speaking terminal and the receiving terminal, when you speak in Japanese from the speaking terminal, the voice is translated into the language you set, for example, English, Korean, Vietnamese, or Indonesian, and heard from the receiving terminal. Because a radio-type terminal is used as the terminal after voice conversion, the user feels as if the radio is doing the translation. Therefore, the translated voice can be transmitted to multiple receivers without delay, which is very convenient. In addition, by using IP radio, the audio can be transmitted from the audio distribution server to the receiving terminal, making it possible to distribute translated audio to a large number of devices, up to about 300 at a time.

[0083] Comparing the features of the present invention with app translation, the following advantages are available: In the case of app translation, one device must be shared and the speaker and the listener must be in close proximity to each other, but with this invention, translation is possible even if the speaker and the listener are far apart. In the case of app translation, translation is difficult when the speaker and the recipient are in Japan and overseas, respectively. However, in the case of this invention, translation is possible even if the speakers are in different countries. In the case of this invention, the speaker can translate by simply pressing the PTT button. The PTT button can be felt without needing to see it, so there is no need to touch the screen. Languages ​​can be switched with the dial knob, so even if you hand the device over to someone else, they can start using it right away. · Audio can be distributed to a group at once, and translation results can be distributed to the group. · Language can be switched for each device, so audio distributed at once can be distributed in the language set for each device. The audio distribution server and translation server (capable of simultaneous translation) work in conjunction with each other, allowing simultaneous distribution to multiple devices. [Industrial Applicability]

[0084] The present invention can be widely used as a multilingual voice distribution method that can translate a speaker's voice into other languages, minimize the time lag associated with translation, and transmit the voice to multiple listeners without delay. [Explanation of symbols]

[0085] 1. Speech terminal 2. Audio distribution server 2a Audio filter section 3. Translation Engine 3a Text conversion section 3b Multilingual Translation Department 3c Speech synthesis engine 4. Audio distribution server 4a Audio distribution section 4b Sampling frequency changeover switch 5. Database 6 Dial knob 7 PTT button 8 Mode Change Button 10, 20, 30 Receiver terminal group

Claims

1. A voice uttered by a speaker is transmitted from the speaking terminal while a PTT button on the speaking terminal is kept pressed, the speaker's voice transmitted from the speaking terminal is input to a translation engine and converted into text, the converted text is translated into multiple languages, the multilingual translated text is converted into voice in each language by a voice synthesis engine for each language, the converted multilingual voice is input to a voice distribution server, the multilingual voice is distributed by a voice distribution unit in a language previously selected and set by the listener, and this translated and distributed voice is received by a listening terminal and transmitted to the listener, When the speech uttered by the speaker is to be transmitted to the receiving terminal in the same language as the speaker without translation, the speech of the speaker is input to the speech distribution server and distributed to the receiving terminal without going through the translation engine. A multilingual voice distribution method characterized in that a sampling frequency changeover switch provided in the voice distribution unit changes between a frequency used for communication to a receiving terminal that receives the translated voice and a frequency used for communication to a receiving terminal that receives the speaker's voice as it is, without being translated.

2. The multilingual voice distribution method according to claim 1, characterized in that recognizing the speaker's voice transmitted from the speech terminal and converting the voice to text is performed by using a web-connected voice recognition and text conversion service, and translating the converted text into multiple languages ​​and converting the multilingual translated text into voice in each language are performed by translation engines distributed and installed in cloud-type PCs.

3. 2. The multilingual speech distribution method according to claim 1, wherein the speaker's speech, including environmental noise, transmitted from the speaking terminal is adjusted by a speech adjustment means for the purpose of improving speech translation accuracy, and then input to the translation engine.

4. 4. The multilingual voice distribution method according to claim 3, wherein said voice adjustment means monitors speaker sound and microphone sound via the Internet and subtracts the speaker sound from the microphone sound.

5. The multilingual voice distribution method according to claim 3, characterized in that the voice adjustment means functions as a physical guide to make the direction of the voice emitted by the speaker unidirectional by providing an aperture at the installation section of an omnidirectional condenser microphone mounted on the speaking terminal.

6. 4. The multilingual voice distribution method according to claim 3, wherein said voice adjustment means generates a sound in an inverse phase to the external noise to cancel the external noise and eliminate the noise.

7. 4. The multilingual voice distribution method according to claim 3, wherein said voice adjustment means automatically increases the volume when the voice level is low, thereby adjusting the voice to a level that is easy to translate.

8. 4. The multilingual voice distribution method according to claim 3, wherein said voice adjustment means modulates the pitch of the voice by an audio equalizer.

9. The multilingual voice distribution method according to claim 3, characterized in that the voice adjustment means uses a voice codec based on voice encoding technology to complement the voice itself and restore it to normal voice when packet loss occurs due to deterioration of the mobile network environment.

10. 2. The multilingual audio distribution method according to claim 1, wherein the audio distribution unit selects one of individual distribution, which transmits to individual receiving terminals, group distribution, which transmits to a plurality of receiving terminals set to belong to a specific group, and simultaneous distribution, which transmits to all receiving terminals.

11. The multilingual audio distribution method according to claim 10, characterized in that the selection of the individual distribution, the group distribution, or the simultaneous distribution is determined based on individual information, group information, and simultaneous information stored in a database installed in the audio distribution unit.

12. 2. The multilingual voice distribution method according to claim 1, wherein the language is switched by a dial knob provided on the speaking terminal and the receiving terminal for switching the translation language, and the selected language is announced by voice from the speaking terminal and the receiving terminal.

13. The multilingual voice distribution method according to claim 1, characterized in that the voice uttered by a listener and received at the listening terminal is transmitted to a listening terminal held by the listener or to a listening terminal held by another listener, thereby enabling two-way communication.

Images