Translation server, translation system, and translation method

Abstract

The present invention provides a translation server, a translation system, and a translation method which make it possible to perform translation corresponding to each of a plurality of users who use differing languages.　A translation system 1 is configured to comprise: a plurality of terminals 2 each having an audio input unit 20 which receives input of audio and a communication unit 26 which outputs audio via a network 4; and a translation server 3 having a communication unit 32 which communicates with the plurality of terminals 2 via the network 4 and acquires audio from each of the plurality of terminals 2, and a control unit 33 which translates audio to be translated into a respective language used on each terminal 2 other than the terminal 2 that output the audio to be translated, which generates synthesized audio, and which outputs the synthesized audio to each other terminal 2 via the network 4 and the communication unit 32.

Description

Translation server, translation system, and translation method

[0001] The present invention relates to a translation server, a translation system, and a translation method.

[0002] As a conventional technique, a conference server that establishes a network conference among a plurality of terminals is known (see, for example, Patent Document 1).

[0003] When this conference server receives voice data from multiple terminals, it stores the voice data received later as a voice memo. When a speaker speaks for a long time and participants add comments such as supplements, rebuttals, or questions, the conference server can record the comments as a voice memo, thereby improving convenience.

[0004] JP 2023-72720 A

[0005] Conventional conference servers have the problem that if participants speak different languages, the conference must be conducted through an interpreter, which is costly and time-consuming, making it difficult to hold international conferences.

[0006] Therefore, an object of the present invention is to provide a translation server, a translation system, and a translation method that can perform translations that correspond to each of a plurality of users who use different languages.

[0007] One aspect of the present invention provides a translation server that includes a communication unit that communicates with multiple terminals via a network and acquires speech from each of the multiple terminals, and a translation unit that translates the speech to be translated into the language used by each terminal other than the terminal that output the speech to be translated, to generate synthetic speech, and outputs the synthetic speech to each of the other terminals via the communication unit and the network.

[0008] Another aspect of the present invention provides a translation system comprising: a plurality of terminals each having a speech input unit for inputting speech and a terminal communication unit for outputting speech via a network; a communication unit that communicates with the plurality of terminals via the network and acquires speech from each of the plurality of terminals; and a translation unit that translates the speech to be translated into a language used by each terminal other than the terminal that output the speech to be translated, to generate synthetic speech, and outputs the synthetic speech to each of the other terminals via the communication unit and the network.

[0009] Another aspect of the present invention provides a translation method that communicates with multiple terminals via a network, acquires speech from each of the multiple terminals, translates the speech to be translated into the language used by each terminal other than the terminal that output the speech to be translated to generate synthetic speech, and outputs the synthetic speech to each of the other terminals via the network.

[0010] According to the present invention, it is possible to perform translations that correspond to each of a plurality of users who use different languages.

[0011] FIG. 1 is a diagram showing an example of a translation system. FIG. 2A is an example of a block diagram of a terminal. FIG. 2B is an example of a block diagram of a translation server. FIG. 2C is a diagram for explaining an example of processing of a translation API. FIG. 3 is an example of a two-dimensional code for joining a room of a translation server. FIG. 4 is a diagram showing an example of a language selection screen displayed on a terminal. FIG. 5 is a diagram showing an example of a translation screen displayed on a terminal when translated from Japanese to English. FIG. 6 is a diagram showing an example of a translation screen displayed on a terminal when translated from Japanese to French. FIG. 7 is a diagram showing an example of a control screen displayed on a tourist guide terminal. FIG. 8 is a diagram for explaining an example of operation of the translation system. FIG. 9 is a flowchart showing an example of operation of the translation system.

[0012] [Overview of the Embodiment] As an example, a translation system with AI (Artificial Intelligence) interpretation according to an embodiment is generally configured by combining multiple terminals and an AI interpretation device. The translation system according to this embodiment recognizes speech from the terminal using the AI ​​interpretation device, and transmits translated speech and translated text to multiple terminals through streaming processing. The translation system according to the embodiment allows even people who do not speak a foreign language to guide foreigners from various countries, or to hold international seminars and international conferences without the need for an interpreter. Note that the AI ​​interpretation is a translation that combines multiple Web Application Programming Interfaces (APIs) that exist in the cloud, as will be described later.

[0013] [Embodiment] (Overview of Translation System 1) FIG. 1 is a diagram illustrating an example of a translation system according to an embodiment. FIG. 2A is an example of a block diagram of a terminal according to an embodiment, FIG. 2B is an example of a block diagram of a translation server, and FIG. 2C is a diagram illustrating an example of processing by a translation API. Note that in each of the figures according to the embodiment described below, the ratios and shapes between figures may differ from the actual ratios and shapes. Also, in FIG. 2C and FIG. 8 described below, arrows indicate the flow of main speech, synthesized speech, etc.

[0014] In the translation system 1 of this embodiment, multiple terminals 2 are connected to a translation server 3 via a network 4. The translation system 1 is configured to output synthetic speech according to a preset language used for each of multiple terminals 2 participating in a WebRTC (Web Real Time Communication) room. This synthetic speech is a translation of the speech to be translated according to the language used. The translation system 1 forms a room in which terminals communicate with each other via P2P (Peer to Peer). Terminals 2 that have been registered in advance can participate in this room.

[0015] The terminal 2 of this embodiment is, for example, a personal computer, a multi-function mobile phone, a phablet terminal, a tablet terminal, or the like.

[0016] 1 to 2B, the translation system 1 is generally configured to include a plurality of terminals 2 each having a speech input unit 20 for inputting speech and a communication unit 26 as a terminal communication unit for outputting speech via a network 4, a communication unit 32 that communicates with the plurality of terminals 2 via the network 4 and acquires speech from each of the plurality of terminals 2, and a translation server 3 having a translation unit that translates the speech to be translated into the language used by each of the terminals 2 other than the terminal 2 that output the speech to be translated, generates synthetic speech, and outputs the synthetic speech to each of the other terminals 2 via the communication unit 32 and the network 4. The network 4 is, for example, the Internet.

[0017] As an example, as shown in FIG. 2A, the terminal 2 further includes an output unit 21, a display unit 22, an input unit 23, an imaging unit 24, a memory unit 25, and a terminal control unit 27.

[0018] 2B , the translation server 3 further includes a storage unit 31 and a control unit 33. The control unit 33 functions as a translation unit by using a translation API 6 of the cloud 5. The translation API 6 of this embodiment performs translation processing using multiple cloud APIs of the cloud 5.

[0019] The translation API 6 is configured to perform speech recognition processing to recognize segments to be translated from the speech to be translated, converting the speech that has been recognized into text, translating the text obtained from the text conversion processing into the language to be translated, and converting the translated text obtained from the translation processing into speech, thereby generating synthetic speech for each of the other terminals.

[0020] 2C , the translation API 6 is generally configured to include a speech recognition processing unit 60 that performs speech recognition processing, a text processing unit 61 that performs text processing, a translation processing unit 62 that performs translation processing, and a voice processing unit 63 that performs voice processing. The speech recognition processing unit 60, the text processing unit 61, the translation processing unit 62, and the voice processing unit 63 are, for example, a Web API configured using AI technology.

[0021] 1, the translation API 6 is configured using a plurality of Web APIs, such as a speech recognition processing unit 60. Therefore, when there are a plurality of Web APIs that can perform the same processing, the speech recognition processing unit 60, the text processing unit 61, the translation processing unit 62, and the speech processing unit 63 are configured to have other Web APIs perform the processing depending on the processing speed, congestion, number of languages ​​to be translated, etc.

[0022] (Configuration of Terminal 2) The audio input unit 20 of the terminal 2 is a microphone. The output unit 21 is a speaker, but is not limited to this, and may be configured to be connected to earphones or the like via a wired or wireless connection. The display unit 22 is, for example, a liquid crystal display or an organic electroluminescence (EL) display. The input unit 23 is, for example, a keyboard, a mouse, or a touchpad. In the terminal 2 of this embodiment, the display unit 22 and the input unit 23 are configured as a touch panel in which the input unit 23 is superimposed on the display unit 22. The imaging unit 24 is, for example, a camera equipped with a semiconductor imaging element. The storage unit 25 is, for example, a storage device such as a hard disk drive (HDD) or a solid state drive (SSD). The storage unit 25 stores a browser application 250. The communication unit 26 is configured to exchange information and signals with the network 4 and the cloud 5.

[0023] The terminal control unit 27 is a microcomputer that includes, for example, a CPU (Central Processing Unit) that performs calculations and processing on acquired data in accordance with stored applications, and semiconductor memories such as RAM (Random Access Memory) and ROM (Read Only Memory). The ROM stores, for example, programs for operating the terminal control unit 27. The RAM is used, for example, as a storage area for temporarily storing calculation results and the like.

[0024] The terminal control unit 27 reads the browser application 250 from the memory unit 25 and accesses the website 310 of the translation server 3 described later, thereby separating the audio from the video and outputting the audio via WebRTC.

[0025] (Translation Server 3) Fig. 3 shows an example of a two-dimensional code for joining a room of a translation server according to an embodiment. The two-dimensional code 7 is, for example, a QR code (registered trademark), but is not limited to this and may be a barcode or the like.

[0026] The translation server 3 is connected to a network 4. The translation server 3 has a website 310 and a registrant DB (Data Base) 311 in a storage unit 31. The website 310 is configured so that only registered registrants can log in and participate in a room.

[0027] When a user photographs the two-dimensional code 7 issued to join the room with the terminal 2, the user can log in to the website 310 of the translation server 3. Note that the user can also log in by directly entering the URL (Uniform Resource Locator) displayed below the two-dimensional code 7 into the browser application 250 of the launched terminal 2, if the URL is registered.

[0028] As shown in Figure 2C, the control unit 33 connects to the translation API 6 via the network 4 and causes the translation API 6 to perform speech recognition processing, text conversion processing, translation processing, and speech conversion processing on the speech to be translated 600, thereby generating synthetic speech.

[0029] A user who uses translation system 1 pre-registers their service number, name, telephone number, language used, etc. in registrant DB 311. Control unit 33 recognizes the language used based on this registrant DB 311 and causes translation API 6 to execute translation. Note that if the user is not pre-registered in registrant DB 311, control unit 33 causes translation API 6 to execute translation based on the language used selected by the user on website 310, as an example.

[0030] As shown in Figure 2C, when translation target speech 600, which is the speech to be translated, is input, the speech recognition processing unit 60 of the translation API 6 recognizes segments in order to convert it into text. Next, when post-recognition processing speech 601 output from the speech recognition processing unit 60 is input, a text conversion processing unit 61 performs text conversion processing. Next, a translation processing unit 62 translates the text 602 output from the text conversion processing unit 61 into the user's language or the other party's language. Next, a voice conversion processing unit 63 performs voice conversion processing on the translation text 603 output from the text conversion processing unit 61, generating synthetic voice 604.

[0031] The translation API 6 generates translation text 603 and synthetic speech 604 from the speech to be translated 600 based on the working languages ​​of the terminals 2 other than the terminal 2 that output the speech to be translated 600. For example, if the speech to be translated 600 is in Japanese and the working languages ​​of the other terminals 2 are English and French, the translation API 6 translates the Japanese speech to be translated 600 into English and French, and generates translation text 603 and synthetic speech 604 in English, as well as translation text 603 and synthetic speech 604 in French.

[0032] The control unit 33 pauses the speech recognition process while the synthetic speech 604 is being output. The pause of the speech recognition process is performed to prevent a loop in which the output synthetic speech 604 is input from the speech input unit and processed again. This pause is performed when the synthetic speech 604 is output, after a break is recognized and a sentence is confirmed, but is not limited to this. Furthermore, when the pause is performed, the next synthetic speech to be played may be, for example, subjected to translation processing and then stored in the storage unit 31.

[0033] The control unit 33 superimposes the translation target speech 600 and the synthetic speech 604 and outputs the superimposed speech to the target terminal 2, and the translation target speech 600 and the synthetic speech 604 are superimposed and output from the output unit 21 of the terminal 2. This gives the user of the terminal 2 the feeling that simultaneous interpretation is being performed. The translation target speech 600 and the synthetic speech 604 may be superimposed at the same volume, or one may be at a lower volume.

[0034] (Regarding the display screen of terminal 2) Fig. 4 is a diagram showing an example of a language selection screen displayed on a terminal according to an embodiment. Fig. 5 is a diagram showing an example of a translation screen displayed on a terminal according to an embodiment when Japanese is translated into English. Fig. 6 is a diagram showing an example of a translation screen displayed on a terminal according to an embodiment when Japanese is translated into French. Fig. 7 is a diagram showing an example of a control screen displayed on a terminal for tour guides according to an embodiment. Figs. 4 to 7 show a case where the translation system 1 is used in the tour guide business. In this translation system 1, one terminal 2 is used as the terminal 2 for tour guides.

[0035] The language selection screen, translation screen, and control screen shown in FIGS. 4 to 7 are examples of screens that are displayed when the user launches the browser application 250 on the terminal 2 and opens the website 310.

[0036] 4 is a screen that displays, as an example, a plurality of language icons 220a and a scroll bar 220b. The language icons 220a are selected and determined by a touch operation on the input unit 23. When the user wants to select language D, the user can set the language to be used to language D by performing a touch operation on the language icon 220a that displays "language D."

[0037] The scroll bar 220b is operated when a user wishes to select a language other than "Language A" to "Language E" shown in FIG. 4, for example. The user can display languages ​​other than the displayed languages ​​by moving the black portion up and down in the figure. The translation system 1 of this embodiment supports languages ​​from 130 countries and regions, for example, but by using multiple translation APIs 6, it is possible to translate languages ​​from even more countries and regions.

[0038] 5 is an example of a screen that is displayed when the speech (speech to be translated 600) from the tourist guide terminal 2 is translated into English. This display screen 221 is a screen that displays, for example, a language icon 221a, a translation start icon 221b, a status display area 221c, a translated text display area 221d, a pre-translation text display area 221e, a speech output radio button 221f, an end icon 221g, and a URL input area 221h.

[0039] The language icon 221a is, for example, an icon indicating the language used by the user who owns the terminal 2. The speech to be translated 600 is translated into the language indicated by the language icon 221a.

[0040] The translation start icon 221b is, for example, an icon that starts translation of the speech to be translated 600.

[0041] The status display area 221c is an area that displays, for example, a status such as "translating" or "translation stopped."

[0042] As an example, the translated text display area 221d is an area where a display is made based on translated text 603 translated into English, which is the language used by the user of the terminal 2. In Fig. 5, since the language used is English, text is displayed that is based on translated text 603 obtained by translating the speech to be translated 600 into English.

[0043] The pre-translation text display area 221e is an area that displays text based on the pre-translation text 602, for example.

[0044] The audio output radio button 221f is, for example, a radio button for selecting whether or not to output audio.

[0045] The exit icon 221g is, for example, an icon for ending participation in the room, i.e., logging out. Terminal 2 is configured to connect to the Public Switched Telephone Network (PSTN) and relay audio between the two terminals via streaming. The exit icon 221g is an icon operated when exiting the WebRTC room.

[0046] The URL input field 221h is, for example, a field for inputting the URL of the website 310.

[0047] 6 is an example of a screen that is displayed when the speech (speech to be translated 600) of the tourist guide terminal 2 is translated into French. A translated text display area 221d of this display screen 221 displays a translation text 603 that has been translated into French, which is the language used by the user of the terminal 2.

[0048] 7 shows, as an example, a control screen of the tourist guide terminal 2. This display screen 222 displays, as an example, a control area 222a, a language icon 222b, a microphone icon 222c, a translation stop icon 222d, a status display area 222e, a translated text display area 222f, a pre-translation text display area 222g, a voice output radio button 222h, and an end icon 222i.

[0049] The control area 222a is an area where information such as whether or not the text is being saved and the name setting is displayed.

[0050] The language icon 222b, status display area 222e, translated text display area 222f, pre-translation text display area 222g, voice output radio button 222h, and end icon 222i are similar to the language icon 221a, status display area 221c, translated text display area 221d, pre-translation text display area 221e, voice output radio button 221f, and end icon 221g on the display screen 221.

[0051] The microphone icon 222c is, for example, an icon for switching the microphone on and off.

[0052] The translation stop icon 222d is, for example, an icon that stops translation when operated.

[0053] Since this terminal 2 is used by a tour guide, the translated text display area 222f and the pre-translation text display area 222g display text based on the pre-translation text 602 of the speech to be translated 600. This text has the same content as the text displayed in the pre-translation text display area 221e in Figures 5 and 6.

[0054] (Operation) Fig. 8 is a diagram for explaining an example of the operation of the translation system according to the embodiment. Fig. 9 is a flowchart showing an example of the operation of the translation system according to the embodiment. Here, the tourist guide terminal 2a translates the speech to be translated S １ and outputs the synthesized speech to the terminal 2b using language A as the working language, the terminal 2c using language B as the working language, and the terminal 2d using language C as the working language, which are all participants in the room. １ may be output from any of the terminals 2a to 2d, and will be translated in the order of the voices input to the translation server 3 unless translation is paused.

[0055] As shown in FIG. 8, the control unit 33 of the translation server 3 receives the speech to be translated S from the tourist guide terminal 2a. １ When the user inputs the speech to be translated (Step 1), the user confirms the languages ​​used by the connected terminals 2b, 2c, and 2d based on the registered user DB 311 and transmits the speech to be translated S to the translation API 6. １ The terminals 2b, 2c, and 2d output the speech to be translated (the speech to be translated S １ ) corresponds to a terminal other than the terminal (terminal 2a) that output the message.

[0056] The translation API 6 receives the speech to be translated S １ Synthesized speech S translated into language A, language B, and language C Ａ ~Synthetic voice S Ｃ , the text before translation S２ , and the translated text S translated into each language ａ ~Translation Text S ｃ Generate.

[0057] The control unit 33 generates the synthesized voice S Ａ ~Synthetic voice S Ｃ , Text S ２ , and the translated text S ａ ~Translation Text S ｃ When the translation API 6 obtains the target speech S １ The control unit 33 outputs the translation target speech S １ , synthesized speech S Ａ , Text S ２ and translated text S ａ is output to the terminal 2b, and the translation target speech S １ , synthesized speech S Ｂ , Text S ２ and translated text S ｂ is output to the terminal 2c, and the translation target speech S １ , synthesized speech S Ｃ , Text S ２ and translated text S ｃ is output to the terminal 2d.

[0058] The control unit 33 controls the terminal 2b to translate the speech to be translated S １ and Synthetic Voice S Ａ While the voice to be translated S is being played back with the voice to be translated superimposed on it, the terminal 2c １ and Synthetic Voice S Ｂ While the voice to be translated S is being played back with the voice to be translated superimposed on it, and while the terminal 2d is playing back the voice to be translated S １ and Synthetic Voice S Ｃ While the above-mentioned sentences are being played back with the original text superimposed, new translation processing is temporarily stopped (Step 4).

[0059] When the playback from terminal 2b to terminal 2d is completed, control unit 33 resumes the translation process (Step 5). Note that control unit 33 may be configured to proceed with the translation process and store the translation process in storage unit 31 if speech is input to another terminal while the translation process is paused.

[0060] The order of translation is arbitrary, and the same user's speech may be translated consecutively.

[0061] Effect of the embodiment The translation system 1 according to the present embodiment can perform translations that correspond to each of multiple users who speak different languages. Specifically, since the translation system 1 performs translations according to the language spoken by each terminal 2, interpreters are not required for each language at international conferences, international seminars, and other events attended by users who speak different languages, compared to when this configuration is not adopted, and the cost and effort required for arranging interpreters can be reduced. Furthermore, the translation system 1 can provide synthesized speech and text translated into the appropriate language for multiple users who speak different languages, such as in tour guides, thereby eliminating the need for users to search for an interpreter who understands their language and allowing for smooth communication.

[0062] The translation system 1 enables two-way translation instead of one-way translation to a specific terminal 2, and also simultaneously translates into a plurality of languages ​​used in other terminals 2, so that the time required for translation for each language used is reduced compared to when this configuration is not adopted, and smooth communication can be achieved.

[0063] The translation system 1 continuously recognizes and translates without requiring any operations other than those required when participating in a room, so the user can obtain translated text and synthesized speech without being aware of it, compared to when button operations or the like are required for translation.

[0064] The translation system 1 uses a translation API 6 present in the cloud 5 for the speech recognition unit used for translation, and therefore can use a Web API that has superior update frequency, number of languages ​​that can be translated, translation accuracy, and translation speed depending on the situation, compared to when it is built into the device.

[0065] The translation system 1 can pause the translation process while synthetic speech is being output, thereby preventing the speech for the call and the synthetic speech from being mixed together in the translation, compared to when the process is not paused.

[0066] Since the translation system 1 outputs the original speech and the translated synthetic speech in a superimposed manner, it is easier to capture nuances that cannot be grasped from the synthetic speech alone compared to when only the synthetic speech is output.

[0067] Another embodiment may be provided as a program that executes a translation method that communicates with multiple terminals via a network, acquires speech from each of the multiple terminals, translates the speech to be translated into the language used by each terminal other than the terminal that output the speech to be translated to generate synthetic speech, and outputs the synthetic speech to at least each of the other terminals via the network, or as a computer-readable recording medium on which this program is recorded.

[0068] The translation method is further configured to suspend speech recognition processing while the synthesized speech is being output.

[0069] The translation server 3, translation system 1, and translation method according to the above-described embodiments may be partially realized by a computer-executed program, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like, depending on the intended use.

[0070] Although several embodiments of the present invention have been described above, these embodiments are merely examples and do not limit the scope of the invention as claimed. These novel embodiments may be embodied in various other forms, and various omissions, substitutions, modifications, etc. may be made without departing from the spirit of the present invention. Furthermore, not all combinations of features described in these embodiments are necessarily essential to solving the problems of the invention. Furthermore, these embodiments are included within the scope and spirit of the invention, and are also included in the scope of the inventions and their equivalents as defined in the claims.

[0071] According to the present invention, it is possible to perform translations that correspond to each of a plurality of users who use different languages.

[0072] 1...translation system, 2, 2a to 2d...terminal, 3...translation server, 4...network, 5...cloud, 6...translation API, 7...two-dimensional code, 20...voice input unit, 21...output unit, 22...display unit, 23...input unit, 24...imaging unit, 25...storage unit, 26...communication unit, 27...terminal control unit, 31...storage unit, 32...communication unit, 33...control unit, 60...speech recognition processing unit, 61...text conversion processing unit, 62...translation processing unit, 63...speech conversion processing unit, 310...website, 311...registrant DB, 600...speech to be translated, 601...speech after recognition processing, 602...text, 603...translated text, 604...synthesized speech, S １ ...Speech to be translated, S ２ ...Text, S Ａ ~S Ｃ ...Synthetic voice, S ａ ~S ｃ …translated text

Claims

1. A translation server comprising: a communication unit that communicates with a plurality of terminals via a network and acquires speech from each of the plurality of terminals; and a translation unit that translates the speech to be translated into a language used by each terminal other than the terminal that outputs the speech to be translated, to generate synthetic speech, and outputs the synthetic speech to each of the other terminals via the communication unit and the network.

2. The translation server of claim 1, wherein the translation unit performs speech recognition processing to recognize segments to be translated from the speech to be translated, converts the speech that has been subjected to the speech recognition processing into text, translates the text obtained from the text conversion processing into the language to be translated, and converts the translated text obtained from the translation processing into speech, thereby generating the synthetic speech for each of the other terminals.

3. The translation server according to claim 2, wherein the translation unit causes the translated text to be displayed on each of the other terminals.

4. The translation server according to claim 2 or 3, wherein the translation unit suspends the speech recognition process while the synthetic speech is being output.

5. A translation system comprising: a plurality of terminals each having a voice input unit for inputting voice and a terminal communication unit for outputting the voice via a network; and a translation server having a communication unit for communicating with the plurality of terminals via the network and acquiring voice from each of the plurality of terminals, and a translation unit for translating the voice to be translated into a language used by each terminal other than the terminal that outputted the voice to be translated, to generate synthetic voice, and outputting the synthetic voice to each of the other terminals via the communication unit and the network.

6. A translation method comprising: communicating with a plurality of terminals via a network; acquiring speech from each of the plurality of terminals; translating the speech to be translated into a language used by each of the other terminals other than the terminal that output the speech to be translated to generate synthetic speech; and outputting the synthetic speech to each of the other terminals via the network.

Images