Translation server, translation system, and translation method

Abstract

To provide a translation server, a translation system, and a translation method that can perform translation corresponding to each of a plurality of users who use different languages.SOLUTION: A translation system 1 is generally configured to include: a plurality of terminals 2 having a voice input unit 20 that inputs voice and a communication unit 26 that outputs voice via a network 4; and a translation server 3 having a communication unit 32 that communicates with the plurality of terminals 2 via the network 4 and acquires voice for each of the plurality of terminals 2, and a control unit 33 that generates synthetic voice by translating voice to be translated into a language used by each of other terminals 2 other than a terminal 2 that outputs the voice to be translated, and outputs the synthetic voice to each of the other terminals 2 via the communication unit 32 and the network 4.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a translation server, a translation system, and a translation method. [Background technology]

[0002] 2. Description of the Related Art As a conventional technique, a conference server that establishes a network conference among a plurality of terminals is known (see, for example, Patent Document 1).

[0003] When the conference server receives voice data from multiple terminals, it stores the voice data received later as a voice memo. When a speaker speaks for a long time and a participant makes a comment such as an addendum, a rebuttal, or a question, the conference server can record the comment as a voice memo, thereby improving convenience. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2023-72720 Summary of the Invention [Problem to be solved by the invention]

[0005] Conventional conference servers have the problem that if participants use different languages, the conference must be conducted through an interpreter, which is costly and time-consuming, making it difficult to hold international conferences.

[0006] SUMMARY OF THE PRESENT EMBODIMENT An object of the present invention is to provide a translation server, a translation system, and a translation method that can perform translations suited to each of a plurality of users who use different languages. [Means for solving the problem]

[0007] One aspect of the present invention provides a translation server that includes a communication unit that communicates with a plurality of terminals via a network and acquires speech from each of the plurality of terminals, and a translation unit that translates the speech to be translated into a language used by each terminal other than the terminal that outputs the speech to be translated, to generate synthetic speech, and outputs the synthetic speech to each of the other terminals via the communication unit and the network.

[0008] Another aspect of the present invention provides a translation system comprising: a plurality of terminals each having a voice input unit for inputting voice and a terminal communication unit for outputting voice via a network; a communication unit for communicating with the plurality of terminals via the network and acquiring voice from each of the plurality of terminals; and a translation unit for translating the voice to be translated into a language used by each terminal other than the terminal that outputted the voice to be translated, to generate synthetic voice, and outputting the synthetic voice for each of the other terminals via the communication unit and the network.

[0009] Another aspect of the present invention provides a translation method which communicates with a plurality of terminals via a network, acquires speech from each of the plurality of terminals, translates the speech to be translated into a language used by each terminal other than the terminal which outputs the speech to be translated to generate synthetic speech, and outputs the synthetic speech to each of the other terminals via the network. Effect of the Invention

[0010] According to the present invention, it is possible to perform translations that correspond to each of a plurality of users who use different languages. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an example of a translation system. [Diagram 2] FIG. 2(a) is an example of a block diagram of a terminal, FIG. 2(b) is an example of a block diagram of a translation server, and FIG. 2(c) is a diagram for explaining an example of processing of a translation API. [Diagram 3] Figure 3 shows an example of a two-dimensional code for joining a room on a translation server. [Figure 4] FIG. 4 is a diagram showing an example of a language selection screen displayed on the terminal. [Diagram 5] FIG. 5 is a diagram showing an example of a translation screen displayed on a terminal when Japanese is translated into English. [Figure 6] FIG. 6 is a diagram showing an example of a translation screen displayed on a terminal when Japanese is translated into French. [Figure 7] FIG. 7 is a diagram showing an example of a control screen displayed on a tour guide terminal. [Figure 8] FIG. 8 is a diagram for explaining an example of the operation of the translation system. [Figure 9] FIG. 9 is a flowchart showing an example of the operation of the translation system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] [Outline of the embodiment] The translation system with AI (Artificial Intelligence) interpretation according to the embodiment is, as an example, generally configured by combining multiple terminals and an AI interpretation device. The translation system according to the embodiment recognizes the voice of the terminal with the AI ​​interpretation device, and transmits the translated voice and translated text by streaming processing to multiple terminals. The translation system according to the embodiment allows even those who cannot speak a foreign language to guide foreigners from various countries, hold international seminars, and hold international conferences without the need for an interpreter. Note that the AI ​​interpretation is a translation that combines multiple WebAPIs (Web Application Programming Interfaces) that exist in the cloud, as will be described later.

[0013] [Embodiment Mode] (Overview of Translation System 1) FIG. 1 is a diagram showing an example of a translation system according to an embodiment. FIG. 2(a) is a block diagram of a terminal according to an embodiment, FIG. 2(b) is a block diagram of a translation server, and FIG. 2(c) is a diagram for explaining an example of a translation API process. In the figures according to the embodiments described below, the ratios and shapes between figures may differ from the actual ratios and shapes. In FIG. 2(c) and FIG. 8 described later, the flow of main voices, synthetic voices, etc. are indicated by arrows.

[0014] In the translation system 1 of this embodiment, multiple terminals 2 are connected to a translation server 3 via a network 4. The translation system 1 is configured to output synthetic speech according to a preset language used for each of multiple terminals 2 participating in a WebRTC (Web Real Time Communication) room. This synthetic speech is a translation of the speech to be translated according to the language used. The translation system 1 forms a room in which terminals communicate with each other via P2P (Peer to Peer). Terminals 2 that have been registered in advance can participate in this room.

[0015] The terminal 2 in this embodiment is, for example, a personal computer, a multi-function mobile phone, a phablet terminal, a tablet terminal, or the like.

[0016] 1 to 2(b), the translation system 1 is generally configured to include a plurality of terminals 2 having a voice input unit 20 for inputting voice and a communication unit 26 as a terminal communication unit for outputting voice via a network 4, a communication unit 32 for communicating with the plurality of terminals 2 via the network 4 and acquiring voice from each of the plurality of terminals 2, and a translation server 3 having a translation unit for translating the voice to be translated into a language used by each of the other terminals 2 other than the terminal 2 that output the voice to be translated, generating synthetic voice, and outputting the synthetic voice to each of the other terminals 2 via the communication unit 32 and the network 4. The network 4 is, for example, the Internet.

[0017] As an example, the terminal 2 further includes an output unit 21, a display unit 22, an input unit 23, an imaging unit 24, a storage unit 25, and a terminal control unit 27, as shown in FIG. 2(a).

[0018] 2(b), the translation server 3 further includes a storage unit 31 and a control unit 33. The control unit 33 functions as a translation unit by using a translation API 6 of the cloud 5. The translation API 6 of this embodiment performs translation processing by using a plurality of cloud APIs of the cloud 5.

[0019] The translation API 6 is configured to perform speech recognition processing to recognize segments to be translated from the speech to be translated, text conversion processing of the speech that has been recognized, translation processing of the text obtained from the text conversion processing into the language to be translated, and speech conversion processing of the translated text obtained from the translation processing, and to generate synthetic speech for each other terminal.

[0020] Specifically, as an example, the translation API 6 is roughly configured to include a voice recognition processing unit 60 that performs voice recognition processing, a text conversion processing unit 61 that performs text conversion processing, a translation processing unit 62 that performs translation processing, and a voice conversion processing unit 63 that performs voice conversion processing, as shown in Fig. 2(c). The voice recognition processing unit 60, the text conversion processing unit 61, the translation processing unit 62, and the voice conversion processing unit 63 are, as an example, WebAPIs configured using AI technology.

[0021] 1, the translation API 6 is configured using a plurality of WebAPIs such as a voice recognition processing unit 60. Therefore, when there are a plurality of WebAPIs capable of executing similar processes, the voice recognition processing unit 60, the text processing unit 61, the translation processing unit 62, and the voice processing unit 63 are configured to cause other WebAPIs to execute the processes depending on the processing speed, congestion, number of languages ​​to be translated, etc.

[0022] (Terminal 2 Configuration) The voice input unit 20 of the terminal 2 is a microphone. The output unit 21 is a speaker, but is not limited thereto, and may be configured to be connected to earphones or the like by wire or wirelessly. The display unit 22 is, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display. The input unit 23 is, for example, a keyboard, a mouse, a touch pad, or the like. In the terminal 2 of this embodiment, the display unit 22 and the input unit 23 are configured as a touch panel in which the input unit 23 is superimposed on the display unit 22. The imaging unit 24 is, for example, a camera equipped with a semiconductor imaging element. The storage unit 25 is, for example, a storage device such as a hard disk drive (HDD) or a solid state drive (SSD). The storage unit 25 stores a browser application 250. The communication unit 26 is configured to exchange information and signals with the network 4 and the cloud 5.

[0023] The terminal control unit 27 is a microcomputer including a CPU (Central Processing Unit) that performs calculations and processing on acquired data according to a stored application, a RAM (Random Access Memory) that is a semiconductor memory, and a ROM (Read Only Memory). The ROM stores, for example, a program for operating the terminal control unit 27. The RAM is used, for example, as a storage area for temporarily storing calculation results.

[0024] This terminal control unit 27 reads the browser application 250 from the memory unit 25 and accesses a website 310 of the translation server 3 described later, thereby making it possible to separate audio from video and output the audio via WebRTC.

[0025] (Translation Server 3) 3 shows an example of a two-dimensional code for joining a room of a translation server according to an embodiment. The two-dimensional code 7 is, for example, a QR code (registered trademark), but is not limited to this and may be a barcode or the like.

[0026] The translation server 3 is connected to a network 4. The translation server 3 has a website 310 and a registrant DB (Data Base) 311 in a storage unit 31. The website 310 is configured so that only registered registrants can log in and participate in a room.

[0027] When a user photographs the two-dimensional code 7 issued to join the room with the terminal 2, the user can log in to the website 310 of the translation server 3. Note that the user can also log in by directly entering the URL (Uniform Resource Locator) displayed below the two-dimensional code 7 into the browser application 250 of the launched terminal 2, if the URL is registered.

[0028] As shown in Figure 2 (c), the control unit 33 connects to the translation API 6 via the network 4 and has the translation API 6 perform speech recognition processing, text conversion processing, translation processing, and speech conversion processing on the speech to be translated 600, thereby generating synthetic speech.

[0029] A user who uses the translation system 1 registers in advance their service number, name, telephone number, language used, etc. in the registrant DB 311. The control unit 33 recognizes the language used based on this registrant DB 311 and causes the translation API 6 to execute the translation. If the user is not registered in advance in the registrant DB 311, the control unit 33 causes the translation API 6 to execute the translation based on the language used selected by the user on the website 310, as one example.

[0030] As shown in Fig. 2(c), when speech to be translated 600 is input, the speech recognition processing unit 60 of the translation API 6 recognizes segments in order to convert it to text. Next, when post-recognition processing speech 601 output from the speech recognition processing unit 60 is input, a text conversion processing unit 61 performs text conversion processing. Next, a translation processing unit 62 translates the text 602 output from the text conversion processing unit 61 into the user's language or the other party's language. Next, a voice conversion processing unit 63 performs voice conversion processing on the translation text 603 output from the text conversion processing unit 61 to generate synthetic voice 604.

[0031] The translation API 6 generates translation text 603 and synthetic speech 604 from the speech to be translated 600 based on the language used by each of the terminals 2 other than the terminal 2 that output the speech to be translated 600. For example, if the speech to be translated 600 is in Japanese and the languages ​​used by the other terminals 2 are English and French, the translation API 6 translates the Japanese speech to be translated 600 into English and French, and generates translation text 603 and synthetic speech 604 in English, as well as translation text 603 and synthetic speech 604 in French.

[0032] The control unit 33 suspends the speech recognition process while the synthetic speech 604 is being output. The suspension of the speech recognition process is performed to prevent a loop in which the output synthetic speech 604 is input from the speech input unit and processed. This suspension is performed when the synthetic speech 604 is output, after a break is recognized and a sentence is confirmed, but is not limited to this. In addition, when the speech recognition process is suspended, the synthetic speech to be played next may be stored in the storage unit 31 after a translation process is performed, for example.

[0033] The control unit 33 superimposes the translation target speech 600 and the synthetic speech 604 and outputs the superimposed speech to the target terminal 2, and the translation target speech 600 and the synthetic speech 604 are superimposed and output from the output unit 21 of the terminal 2. This gives the user of the terminal 2 the feeling that simultaneous interpretation is being performed. Note that the translation target speech 600 and the synthetic speech 604 may be superimposed at the same volume, or one of them may be at a lower volume.

[0034] (Display screen of terminal 2) FIG. 4 is a diagram showing an example of a language selection screen displayed on a terminal according to an embodiment. FIG. 5 is a diagram showing an example of a translation screen displayed on a terminal according to an embodiment when Japanese is translated into English. FIG. 6 is a diagram showing an example of a translation screen displayed on a terminal according to an embodiment when Japanese is translated into French. FIG. 7 is a diagram showing an example of a control screen displayed on a terminal for tour guides according to an embodiment. FIGS. 4 to 7 show a case where the translation system 1 is used in the tour guide business. In this translation system 1, one terminal 2 is used as the terminal 2 for tour guides.

[0035] The language selection screen, translation screen, and control screen shown in FIGS. 4 to 7 are examples of screens that are displayed when the user launches the browser application 250 of the terminal 2 and opens the website 310.

[0036] 4 is, as an example, a screen that displays a plurality of language icons 220a and a scroll bar 220b. The language icon 220a is selected and determined by a touch operation on the input unit 23. When the user wishes to select the D language, the user can set the language to be used to the D language by performing a touch operation on the language icon 220a that displays "D language."

[0037] Scroll bar 220b is operated when a user wishes to select a language other than "Language A" to "Language E" displayed in FIG. 4, for example. A user can display a language other than the displayed language by moving the black portion up and down in the figure. Translation system 1 of this embodiment corresponds to languages ​​of 130 countries and regions, for example, but by using multiple translation APIs 6, it is possible to translate languages ​​of more countries and regions.

[0038] 5 is, for example, a screen displayed when the voice of the tourist guide terminal 2 (the voice to be translated 600) is translated into English. This display screen 221 is, for example, a screen displaying a used language icon 221a, a translation start icon 221b, a status display area 221c, a translated text display area 221d, a pre-translation text display area 221e, a voice output radio button 221f, an end icon 221g, and a URL input area 221h.

[0039] The language icon 221a is, for example, an icon indicating the language used by the user who owns the terminal 2. The translation target speech 600 is translated into the language indicated by the language icon 221a.

[0040] The translation start icon 221b is, for example, an icon for starting translation of the speech 600 to be translated.

[0041] The status display area 221c is an area for displaying a status such as "translation in progress" or "translation stopped", for example.

[0042] As an example, the translated text display area 221d is an area where a display is made based on a translated text 603 translated into English, which is the language used by the user of the terminal 2. In Fig. 5, since the language used is English, text is displayed that is based on the translated text 603 obtained by translating the translation target speech 600 into English.

[0043] The pre-translation text display area 221e is an area that displays text based on the pre-translation text 602, for example.

[0044] The audio output radio button 221f is, for example, a radio button for selecting whether or not to output audio.

[0045] The exit icon 221g is, for example, an icon for ending participation in the room, that is, logging out. The terminal 2 is configured to connect a PSTN (Public Switched Telephone Network) to WebRTC and stream audio between the terminals. The exit icon 221g is an icon operated when exiting the WebRTC room.

[0046] The URL input field 221h is, for example, a field for inputting the URL of the website 310.

[0047] 6 is an example of a screen displayed when the speech of the tourist guide terminal 2 (speech to be translated 600) is translated into French. The translation text display area 221d of this display screen 221 displays a translation text 603 translated into French, which is the language used by the user of the terminal 2.

[0048] 7 shows, as an example, a control screen of the tourist guide terminal 2. This display screen 222 displays, as an example, a control area 222a, a language icon 222b, a microphone icon 222c, a translation stop icon 222d, a status display area 222e, a translated text display area 222f, a pre-translation text display area 222g, a voice output radio button 222h, and an end icon 222i.

[0049] The control area 222a is an area where whether or not the text is being saved, the name setting, and the like are displayed.

[0050] The language used icon 222b, status display area 222e, translated text display area 222f, untranslated text display area 222g, voice output radio button 222h, and exit icon 222i are similar to the language used icon 221a, status display area 221c, translated text display area 221d, untranslated text display area 221e, voice output radio button 221f, and exit icon 221g on the display screen 221.

[0051] The microphone icon 222c is, for example, an icon for switching the microphone on and off.

[0052] The translation stop icon 222d is, for example, an icon that stops translation when an operation is performed.

[0053] Since this terminal 2 is used by a tour guide, the translated text display area 222f and the pre-translation text display area 222g display text based on the pre-translation text 602 of the speech to be translated 600. This text has the same content as the text displayed in the pre-translation text display area 221e in Figs. 5 and 6.

[0054] (operation) FIG. 8 is a diagram for explaining an example of the operation of the translation system according to the embodiment. FIG. 9 is a flowchart showing an example of the operation of the translation system according to the embodiment. Here, an example of the operation will be described in which a tour guide terminal 2a outputs a speech to be translated S1, and outputs a synthetic speech to a terminal 2b that uses A language, a terminal 2c that uses B language, and a terminal 2d that uses C language, which are participating in a room. Note that the speech to be translated S1 may be output from any of the terminals 2a to 2d, and is translated in the order of the speech input to the translation server 3 unless translation is paused.

[0055] 8, when the translation target speech S1 is input from the tourist guide terminal 2a (Step 1), the control unit 33 of the translation server 3 checks the languages ​​used by the connected terminals 2b, 2c, and 2d based on the registrant DB 311 and outputs the translation target speech S1 to the translation API 6 (Step 2). The terminals 2b, 2c, and 2d correspond to terminals other than the terminal (terminal 2a) that outputs the speech to be translated (translation target speech S1).

[0056] The translation API 6 generates a synthesized speech S1 translated into a language A, a language B, and a language C. A ~Synthetic voice SC , the original text S2, and the translated text S a ~Translated Text S c Generate.

[0057] The control unit 33 generates the synthetic voice S A ~Synthetic voice S C , text S2, and translated text S a ~Translated Text S c When the translation API 6 obtains the translation target speech S1, the translation target speech S1 and the synthetic speech S2 are output to the corresponding terminal 2 (Step 3). A , text S2 and translated text S a is output to terminal 2b, and the translation target speech S1 and the synthetic speech S B , text S2 and translated text S b is output to terminal 2c, and the translation target speech S1 and the synthetic speech S C , text S2 and translated text S c is output to terminal 2d.

[0058] The control unit 33 controls the terminal 2b to translate the speech to be translated S1 and the synthesized speech S A While the two are being played back, the terminal 2c is playing back the translation target speech S1 and the synthetic speech S B While the translation target speech S1 and the synthetic speech S C While the text is being played back with the original text superimposed, new translation processing is temporarily halted (Step 4).

[0059] When the playback from terminal 2b to terminal 2d ends, control unit 33 resumes the translation process (Step 5). Note that control unit 33 may be configured to proceed with the translation process and store it in storage unit 31 when voice is input to another terminal while the translation process is paused.

[0060] The order of translation is arbitrary, and the speech of the same user may be translated consecutively.

[0061] (Effects of the embodiment) The translation system 1 according to the present embodiment can perform translation corresponding to each of a plurality of users who use different languages. Specifically, since the translation system 1 performs translation according to the language used by each terminal 2, in comparison with a case where this configuration is not adopted, an interpreter is not required for each language used in an international conference or international seminar attended by users who use different languages, and the cost of interpreters and the effort of arranging them can be reduced. Furthermore, the translation system 1 can provide synthetic speech or text translated into an appropriate language for a plurality of users who use different languages, such as in a tourist guide, and therefore the user does not need to search for an interpreter who can understand the language used by the user, and can communicate smoothly.

[0062] The translation system 1 enables two-way translation rather than one-way translation for a specific terminal 2, and also simultaneously translates into a plurality of languages ​​used at other terminals 2, so that the time required for translation for each language is reduced compared to a case where this configuration is not adopted, and smoother communication can be achieved.

[0063] The translation system 1 continuously recognizes and translates without requiring any operations other than those required when joining a room, so the user can obtain translated text and synthesized voice without being aware of it, compared to when button operations are required for translation.

[0064] The translation system 1 uses a translation API 6 in the cloud 5 for the speech recognition unit and other parts used for translation, so that it is possible to use WebAPIs that have superior update frequency, number of languages ​​that can be translated, translation accuracy, and translation speed depending on the situation, compared to when they are built into the device.

[0065] The translation system 1 can pause the translation process while the synthetic voice is being output, thereby preventing the voice for the call and the synthetic voice from being mixed together and translated, compared to when the process is not paused.

[0066] Since translation system 1 outputs the original speech and the translated synthetic speech in a superimposed manner, it is easier to capture nuances that cannot be grasped from the synthetic speech alone, compared to when only the synthetic speech is output.

[0067] Here, another embodiment may be provided as a program that executes a translation method that communicates with multiple terminals via a network, acquires speech from each of the multiple terminals, translates the speech to be translated into a language used by each terminal other than the terminal that outputs the speech to be translated to generate synthetic speech, and outputs the synthetic speech to at least each of the other terminals via the network, or as a computer-readable recording medium on which this program is recorded.

[0068] The translation method is further configured to suspend the speech recognition process while the synthetic speech is being output.

[0069] The translation server 3, translation system 1, and translation method according to the above-described embodiments may be realized, for example, in part by a program executed by a computer, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like, depending on the application.

[0070] Although several embodiments of the present invention have been described above, these embodiments are merely examples and do not limit the invention according to the claims. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, etc. can be made without departing from the gist of the present invention. In addition, not all combinations of features described in these embodiments are necessarily essential to the means for solving the problems of the invention. Furthermore, these embodiments are included within the scope and gist of the invention, and are included within the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0071] 1...translation system, 2, 2a to 2d...terminal, 3...translation server, 4...network, 5...cloud, 6...translation API, 7...2D code, 20...voice input unit, 21...output unit, 22...display unit, 23...input unit, 24...imaging unit, 25...storage unit, 26...communication unit, 27...terminal control unit, 31...storage unit, 32...communication unit, 33...control unit, 60...voice recognition processing unit, 61...text conversion processing unit, 62...translation processing unit, 63...voice conversion processing unit, 310...website, 311...registered user DB, 600...speech to be translated, 601...speech after recognition processing, 602...text, 603...translated text, 604...synthesized speech, S1...speech to be translated, S2...text, S A ~S C ...Synthetic voice, S a ~S c …Translated text

Claims

1. a communication unit that communicates with a plurality of terminals via a network and acquires voice from each of the plurality of terminals; a translation unit that translates the speech to be translated into a language used by each of the terminals other than the terminal that outputs the speech to be translated, generates synthetic speech, and outputs the synthetic speech to each of the terminals via the communication unit and the network; A translation server equipped with

2. the translation unit performs a speech recognition process for recognizing segments to be translated from the speech to be translated, a text conversion process for the speech that has been subjected to the speech recognition process, a translation process for the text obtained by the text conversion process into the language to be translated, and a voice conversion process for the translated text obtained by the translation process, and generates the synthetic speech for each of the other terminals. The translation server of claim 1.

3. The translation unit causes the translated text to be displayed on each of the other terminals. The translation server of claim 2.

4. The translation unit suspends the speech recognition process while the synthetic speech is being output.

4. The translation server according to claim 2 or 3.

5. A plurality of terminals each having a voice input unit for inputting voice and a terminal communication unit for outputting the voice via a network; a translation server having a communication unit that communicates with the plurality of terminals via the network and acquires speech from each of the plurality of terminals, and a translation unit that translates the speech to be translated into a language used by each of the terminals other than the terminal that outputs the speech to be translated, to generate synthetic speech, and outputs the synthetic speech to each of the other terminals via the communication unit and the network; A translation system with

6. Communicating with a plurality of terminals via a network and acquiring voice from each of the plurality of terminals; Translating the speech to be translated into a language used by each terminal other than the terminal that outputs the speech to be translated to generate synthetic speech; outputting the synthetic voice to each of the other terminals via the network; How to translate.

Images