First user-side device, translation system, and translation method

Abstract

Provided are a first user-side device, a translation system, and a translation method with which it is possible to smoothly perform a call even when different languages are used.　A user-side device 2 schematically comprises: a speech input unit 20 to which a user-side speech S1 is inputted; an output unit 21 that outputs a partner-side synthesized speech S32; a communication unit 25 that performs communication with a partner-side device 3 via a network 4; and a user-side control unit 26 that generates a user-side synthesized speech S31 obtained by translating the user-side speech S1 inputted to the speech input unit 20 into a partner-side language, generates the partner-side synthesized speech S32 obtained by translating a partner-side speech S11 inputted via the communication unit 25 into a user-side language, and causes the partner-side synthesized speech S32 to be outputted from the output unit 21 and causes the user-side synthesized speech S31 to be outputted to the partner-side device 3.

Description

First user-side device, translation system, and translation method

[0001] The present invention relates to a first user-side device, a translation system, and a translation method.

[0002] As a conventional technology, an IP network translation system is known for Internet Protocol (IP) telephones that use the Internet, which automatically identifies the language used by the caller and the language used by the recipient based on the IP number at the time of IP telephone connection, and performs automatic translation of telephone calls that pass through an IP network (see, for example, Patent Document 1).

[0003] This IP network translation method allows a translation server provided on the IP network to automatically translate between the language used by the caller and the language used by the recipient, allowing voice calls to be made while automatically translating voices between IP telephones.

[0004] Japanese Patent Application Laid-Open No. 2004-260430

[0005] Conventional IP network translation methods have the problem that when identifying the language used by the caller and the language used by the recipient based on the IP number, the language used by both IP phones within the same country is treated as the same even if the languages ​​used are different.

[0006] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide a first user device, a translation system and a translation method that enable smooth communication even when the languages ​​used by the parties are different.

[0007] One aspect of the present invention provides a first user-side device comprising: a first user-side voice input unit that inputs a first user-side voice; a first user-side output unit that outputs a second user-side synthesized voice; a first user-side communication unit that communicates with a second user-side device via a network; and a translation unit that generates a first user-side synthesized voice by translating the first user-side voice input to the first user-side voice input unit into a second user-side language, generates a second user-side synthesized voice by translating the second user-side voice input via the first user-side communication unit into the first user-side language, outputs the second user-side synthesized voice from the first user-side output unit, and outputs the second user-side synthesized voice to the second user-side device.

[0008] Another aspect of the present invention is a system including a first user-side device having a first user-side voice input unit for inputting a first user-side voice, a first user-side output unit for outputting a second user-side synthetic voice, a first user-side communication unit for communicating with a second user-side device via a network, and a translation unit for generating a first user-side synthetic voice by translating the first user-side voice inputted to the first user-side voice input unit into a second user-side language, generating a second user-side synthetic voice by translating the second user-side voice inputted via the first user-side communication unit into the first user-side language, outputting the second user-side synthetic voice from the first user-side output unit, and outputting the second user-side synthetic voice to the second user-side device via the network; and a translation unit connected to the network and configured to translate the first user-side voice and Provided is a translation system comprising: a translation mechanism that performs speech recognition processing to recognize segments to be translated from a second user-side speech, text conversion processing of the speech that has undergone speech recognition processing, translation processing of the text obtained from the text conversion processing into the language of the other party, and speech conversion processing of the translated text obtained from the translation processing to generate a first user-side synthetic speech and a second user-side synthetic speech; and a second user-side device having a second user-side speech input unit that inputs the second user-side speech, a second user-side output unit that outputs the first user-side synthetic speech, a second user-side communication unit that communicates with the first user-side device via a network, and a second user-side control unit that outputs the first user-side synthetic speech obtained via the second user-side communication unit from the second user-side output unit.

[0009] Another aspect of the present invention provides a translation method that inputs a first user-side speech and a second user-side speech, performs a speech recognition process to recognize segments to be translated from the input first user-side speech and second user-side speech, converts the speech that has been recognized into text, translates the text obtained from the text conversion process into a target language, converts the translated text obtained from the translation process into speech to generate a first user-side synthetic speech and a second user-side synthetic speech, outputs the second user-side synthetic speech on a first user-side device, and outputs the first user-side synthetic speech on a second user-side device.

[0010] According to the present invention, communication can be smoothly carried out even when the languages ​​used by the parties are different.

[0011] FIG. 1 is a diagram showing an example of a translation system. FIG. 2A is an example of a block diagram of a user-side device. FIG. 2B is an example of a block diagram of a counterpart-side device. FIG. 2C is a diagram for explaining an example of processing of a translation API. FIG. 3 is a diagram showing an example of a language selection screen displayed on a user-side device. FIG. 4 is a diagram showing an example of a standby screen displayed on a user-side device. FIG. 5 is a diagram showing an example of a phone book screen displayed on a user-side device. FIG. 6 is a diagram showing an example of a control screen displayed on a user-side device. FIG. 7 is a timing chart showing an example of the operation of the translation system.

[0012] [Overview of the Embodiment] A translation system with AI (Artificial Intelligence) interpretation according to an embodiment is generally configured by combining a multifunction mobile phone or an IP telephone terminal with an AI interpretation device, as an example. The translation system according to this embodiment uses the AI ​​interpretation device to recognize speech during a conversation on the multifunction mobile phone or IP telephone terminal, and outputs speech translated by streaming processing from the multifunction mobile phone or IP telephone terminal. This allows the translation system according to the embodiment to hold telephone conferences or calls with users who speak different languages ​​without the need for an interpreter. Note that the AI ​​interpretation is a translation that combines multiple Web APIs (Web Application Programming Interfaces) that exist in the cloud, as described below.

[0013] [First embodiment] (Overview of translation system 1) FIG. 1 is a diagram illustrating an example of a translation system according to a first embodiment. FIG. 2A is an example of a block diagram of a user-side device according to the first embodiment, FIG. 2B is an example of a block diagram of a destination-side device, and FIG. 2C is a diagram illustrating an example of processing by a translation API. Note that in each diagram according to the embodiments described below, the ratios and shapes between figures may differ from the actual ratios and shapes. Also, in FIGS. 2A to 2C, arrows indicate the flow of main speech and synthesized speech.

[0014] The translation system 1 of this embodiment enables a first user using a user device 2 (a first user device) to communicate with a second user using a remote device 3 (a second user device) via Web Real Time Communication (WebRTC) via P2P (Peer to Peer) communication. The white arrows in FIG. 1 indicate the P2P communication between the user device 2 and the remote device 3. Furthermore, when the first user speaks a first language and the second user speaks a second language different from the first language, the translation system 1 translates the first language into the second language and the second language into the first language using a translation process described below, enabling smooth communication even when the users do not understand each other's languages. If at least one of the user device 2 and the remote device 3 has an imaging unit, audio is separated from the video and output to WebRTC.

[0015] This embodiment describes a case where a first user makes a call to a second user, and the first user's user device 2 has a translation unit (described later). In this embodiment, the first user making the call is referred to as the user, and the second user receiving the call is referred to as the other party. The user device 2 and the other party device 3 are, for example, personal computers, multi-function mobile phones, phablet terminals, and tablet terminals.

[0016] As shown in Fig. 1, the translation system 1 is generally configured to include a user-side device 2, a counterpart-side device 3, and a translation API 6 that functions as a translation mechanism. The translation mechanism may be a translation device connected to a network 4. The translation API 6 of this embodiment performs processing as a translation mechanism using multiple cloud APIs in a cloud 5.

[0017] As shown in FIG. 2A, the user device 2 receives a user voice (first user voice) S １ A voice input unit (first user side voice input unit) 20 for inputting a voice from the other side (second user side synthesized voice) S ３２an output unit (first user side output unit) 21 that outputs the user side voice S input to the voice input unit 20; a communication unit (first user side communication unit) 25 that communicates with the other side device 3 via the network 4; １ A user-side synthesized speech (first user-side synthesized speech) S translated into the other party's language (second user-side language) ３１ and the voice of the other party (second user's voice) S １１ The other party's synthesized speech S is translated into the user's language (first user's language). ３２ and the other party's synthesized voice S ３２ is output from the output unit 21, and the user-side synthesized voice S ３１ to the other device 3. The network 4 is, for example, the Internet.

[0018] As an example, the user device 2 further includes a display unit 22, an input unit 23, and a storage unit 24, as shown in FIG. 2A.

[0019] As shown in FIG. 2B, the other device 3 receives the other device voice S １１ a voice input unit (second user-side voice input unit) 30 for inputting a user-side synthesized voice S ３１ an output unit (second user-side output unit) 31 that outputs the user-side voice signal S; a communication unit (second user-side communication unit) 35 that communicates with the user-side device 2 via the network 4; and a user-side synthesized voice signal S ３１ and a counterpart control unit (second user side control unit) 36 that causes the output unit 31 to output the above.

[0020] The user device 2 further includes, for example, a display unit 32, an input unit 33, and a storage unit .

[0021] As shown in FIG. 2C, the translation API 6 is connected to the network 4 and receives the user's voice S １ and the other party's voice S １１A speech recognition process is performed to recognize segments to be translated from the speech, a text conversion process is performed on the speech that has been recognized, a translation process is performed on the text obtained by the text conversion process into the language of the other party, and a speech conversion process is performed on the translated text obtained by the translation process to generate a synthesized speech S ３１ and the other party's synthesized voice S ３２ The apparatus is generally configured to generate:

[0022] Specifically, the translation API 6 is generally configured to include, for example, a speech recognition processing unit 60 that performs speech recognition processing, a text processing unit 61 that performs text processing, a translation processing unit 62 that performs translation processing, and a voice processing unit 63 that performs voice processing. The speech recognition processing unit 60, the text processing unit 61, the translation processing unit 62, and the voice processing unit 63 are, for example, a Web API configured using AI technology.

[0023] 1, the translation API 6 is configured using a plurality of APIs, such as a speech recognition processing unit 60. Therefore, when there are a plurality of APIs capable of executing the same process, the speech recognition processing unit 60, the text processing unit 61, the translation processing unit 62, and the speech processing unit 63 are configured to have another API execute the process depending on the processing speed, congestion, etc.

[0024] (Configuration of User Device 2) The voice input unit 20 of the user device 2 is a microphone. The output unit 21 is a speaker, but is not limited to this, and may be configured to be connected to earphones or the like via a wired or wireless connection. The display unit 22 is, for example, a liquid crystal display or an organic electroluminescence (EL) display. The input unit 23 is, for example, a keyboard, a mouse, or a touchpad. In the user device 2 of this embodiment, the display unit 22 and the input unit 23 are configured as a touch panel in which the input unit 23 is superimposed on the display unit 22. The memory unit 24 is, for example, a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The memory unit 24 stores a browser application 240. The communication unit 25 is configured to exchange information and signals with the network 4 and the cloud 5.

[0025] The user-side control unit 26 is a microcomputer that includes, for example, a CPU (Central Processing Unit) that performs calculations and processing on acquired data in accordance with stored applications, and semiconductor memories such as RAM (Random Access Memory) and ROM (Read Only Memory). The ROM stores, for example, programs for operating the user-side control unit 26. The RAM is used, for example, as a storage area for temporarily storing calculation results and the like.

[0026] The user-side control unit 26 functions as a translation unit by reading the browser application 240 from the storage unit 24 and accessing a website 70 on the web server 7, which will be described later. The website 70 enables two-way communication between the user-side device 2 and the other-side device 3.

[0027] The user-side control unit 26 connects to the translation API 6 via the network 4 and converts the user-side voice S １ and the other party's voice S １１ The translation API 6 executes speech recognition processing, text conversion processing, translation processing, and speech conversion processing, and the user-side synthesized speech S ３１ and the other party's synthesized voice S ３２ Generate.

[0028] A user of translation system 1 pre-registers their service number, name, telephone number, language used, etc. in a telephone directory DB (Data Base) 71 (described later). User-side control unit 26 recognizes the language used based on this telephone directory DB 71 and causes translation API 6 to execute translation. If the user is not pre-registered in telephone directory DB 71, user-side control unit 26 causes translation API 6 to execute translation based on the language used selected by the user on website 70.

[0029] As shown in FIG. 2C, the voice recognition processing unit 60 recognizes the user's voice S １ and the other party's voice S １１When speech 600 is input, it recognizes segments to convert it into text. Next, when the post-recognition-processed speech 601 output from the speech recognition processing unit 60 is input, the text conversion processing unit 61 performs text conversion processing. Next, the translation processing unit 62 translates the text 602 output from the text conversion processing unit 61 into the user's language or the other party's language. Next, the voice conversion processing unit 63 performs voice conversion processing on the translated text 603 output from the text conversion processing unit 61 to generate synthetic speech 604. This synthetic speech 604 is the user's synthetic speech S ３１ and the other party's synthesized voice S ３２ is.

[0030] The translated text 603 is translated into the user's language and output to the user-side device 2, and translated into the other party's language and output to the other party's device 3. １１ The other device 3 can also display the user's voice S translated into the user's own language. １ It will be possible to display text translated into your own language.

[0031] The user-side control unit 26 generates the user-side synthesized voice S ３１ and the other party's synthesized voice S ３２ is being output, the speech recognition process is paused. The pause of the speech recognition process is performed to prevent a loop in which the output synthetic speech is input from each speech input unit and the process proceeds. This pause is performed when synthetic speech is being output, after a break is recognized and a sentence is confirmed, but is not limited to this. Furthermore, when the pause is performed, the next synthetic speech to be played may be, for example, subjected to translation processing and then stored in the storage unit 24.

[0032] The user-side control unit 26 controls the user-side voice S １ and user-side synthesized voice S ３１ and output to the other device 3, and the other device's voice S １１ and the other party's synthesized voice S ３２The voice and the synthesized voice are superimposed and output from the output unit 21 on the user side. This allows the user and the other party to communicate as if simultaneous interpretation were being performed. The voice and the synthesized voice may be superimposed at the same volume, or one of them may be made lower in volume before being superimposed.

[0033] (Configuration of the other device 3) The audio input unit 30 of the other device 3 is a microphone. The output unit 31 is a speaker, but is not limited to this, and may be configured to be connected to earphones or the like via a wired or wireless connection. The display unit 32 is, for example, a liquid crystal display or an organic EL display. The input unit 33 is, for example, a keyboard, a mouse, or a touchpad. As an example, the other device 3 may be configured as a touch panel in which the display unit 32 and the input unit 33 are superimposed on the display unit 32. The memory unit 34 is, for example, a memory device such as an HDD or SSD. The communication unit 35 is configured to exchange information and signals with the network 4 and the cloud 5.

[0034] The other-side control unit 36 ​​is, for example, a microcomputer including a CPU, a RAM, a ROM, and the like.

[0035] (Web Server 7) The web server 7 is connected to the network 4. This web server 7 has a website 70 and a telephone directory DB 71. The website 70 enables P2P communication with registered telephone numbers. These telephone numbers are registered in the telephone directory DB 71 of the web server 7.

[0036] The user operates the user side device 2 to display this website 70, and can make a call by selecting and deciding the telephone number of the person he / she wants to call from the displayed telephone numbers.

[0037] (Regarding the display screen of the user-side device 2) Fig. 3 is a diagram showing an example of a language selection screen displayed on the user-side device according to the first embodiment. Fig. 4 is a diagram showing an example of a standby screen displayed on the user-side device according to the first embodiment. Fig. 5 is a diagram showing an example of a phone book screen displayed on the user-side device according to the first embodiment. Fig. 6 is a diagram showing an example of a control screen displayed on the user-side device according to the first embodiment.

[0038] The language selection screen, standby screen, phone book screen, and control screen shown in Figures 3 to 6 are examples of screens that are displayed when the user launches the browser application 240 of the user-side device 2 and opens the website 70.

[0039] 3 is a screen that displays, as an example, a plurality of language icons 220a and a scroll bar 220b. The language icons 220a are selected and determined by a touch operation on the input unit 23. When the user wants to select language D, the user can set the language to be used to language D by performing a touch operation on the language icon 220a that displays "language D."

[0040] The scroll bar 220b is operated when a user wishes to select a language other than "Language A" to "Language E" shown in FIG. 3, for example. The user can display languages ​​other than the displayed languages ​​by moving the black portion up and down in the figure. The translation system 1 of this embodiment supports languages ​​from 130 countries and regions, for example, but by using multiple translation APIs 6, it is possible to translate languages ​​from even more countries and regions.

[0041] 4 is, for example, a screen displayed when making a call from the user-side device 2 to the other-side device 3. This display screen 221 is a screen that displays, for example, a caller display area 221a, a selection area 221b, a status display area 221c, a translation display 221d, a microphone display 221e, a translation stop icon 221f, a language switch icon 221g, a text save icon 221h, an end icon 221i, a phone number selection icon 221j, a phone number display area 221k, a call start icon 221l, an enter icon 221m, an exit icon 221n, and a text display area 221o.

[0042] The other party display area 221a is an area where, for example, an image of the other party operating the other party's device 3, the start and time of video or audio recording, etc. are displayed.

[0043] As an example, the selection area 221b is an area where a "mic" radio button that allows the audio input unit 20 to be turned on or off and a "voice" radio button that allows the user to select whether or not to mix audio and synthesized audio are displayed.

[0044] The status display area 221c is an area that displays the current status, for example, by displaying "Waiting" while waiting for the other party to answer a call after making a call, and by displaying "Call in progress" once the call has started.

[0045] The translation display 221d displays the language to be translated, for example, in Fig. 4, Japanese to English and English to Japanese.

[0046] The microphone display 221e shows, as an example, whether the microphone is on or off.

[0047] As an example, the translation stop icon 221f is an icon that stops translation when a touch operation is received during translation, and starts translation when another touch operation is received.

[0048] The language switching icon 221g is, for example, an icon that enables switching of the language to be translated.

[0049] The text save icon 221h is, for example, an icon for saving the text displayed in the text display area 221o in the storage unit 24.

[0050] The end icon 221i is, for example, an icon for ending a call.

[0051] The telephone number selection icon 221j is, for example, an icon that is operated when selecting a party to call from a telephone directory. This telephone directory is displayed based on the telephone directory DB 71.

[0052] The telephone number display area 221k is an area that displays a selected telephone number, for example.

[0053] The call start icon 221l is, for example, an icon that is operated when making a call to a selected phone number.

[0054] The user device 2 and the remote device 3 are configured to connect to the PSTN (Public Switched Telephone Network) and stream audio between each other. The enter icon 221m is an icon operated when entering a WebRTC room. The exit icon 221n is an icon operated when leaving a WebRTC room. In Figure 4, the exit icon 221n is grayed out because the state is before entry.

[0055] The text display area 221o displays, for example, the user's voice S １ and the other party's voice S １１ This is the area where the translated text is displayed.

[0056] 5 shows an example of a display screen 222 in which a phone book screen 222a, which is displayed when selecting a telephone number to call, is superimposed on the display screen 221. The phone book screen 222a displays, for example, a close icon 222b, an add icon 222c, an edit icon 222d, a delete icon 222e, a search box 222f, a refine icon 222g, a reset icon 222h, a telephone number display area 222i, a select icon 222j, and a cancel icon 222k. In FIG. 5, the same icons as those in the display screen 221 shown in FIG. 4 are denoted by the same reference numerals as in FIG. 4.

[0057] The close icon 222b is, for example, an icon that is operated when closing the phone book screen 222a.

[0058] The add icon 222c is, for example, an icon that is operated when adding a phone number.

[0059] The edit icon 222d is, for example, an icon operated when editing a telephone number or the like.

[0060] The delete icon 222e is, for example, an icon that is operated when deleting a registered telephone number or the like.

[0061] The search box 222f is, for example, an area for inputting a search word for searching a registered telephone number or the like.

[0062] The narrowing down icon 222g is, for example, an icon that is operated when searching the telephone directory using the search word entered in the search box 222f.

[0063] The reset icon 222h is, for example, an icon that is operated when erasing the search word entered in the search box 222f.

[0064] The telephone number display area 222i is an area that displays, for example, a "checkbox," a "selected name," a "telephone number," and a "language." When a user checks the "checkbox" next to the "selected name" of the person they want to call, they can call the corresponding telephone number. The "language" indicates the language used by the person displayed in the selected name.

[0065] The select icon 222j is, for example, an icon that is operated when calling the telephone number selected in the telephone number display area 222i.

[0066] The cancel icon 222k is, for example, an icon that is operated when ending editing or selection of a telephone number or the like, or when closing the phone book screen 222a.

[0067] The display screen 223 shown in Fig. 6 shows, as an example, a control screen displayed during a call. As an example, a new control area 223a is displayed on this display screen 223, and the other party display area 221a and the like have been moved downward. Note that in Fig. 6, the same icons and the like as those on the display screen 221 shown in Fig. 4 are assigned the same reference numerals as in Fig. 4.

[0068] The control area 223a is an area where information such as whether or not the text is being saved and the other party's telephone number are displayed.

[0069] (Operation) An example of the operation of the translation system 1 according to this embodiment will be described below with reference to the timing chart of Fig. 7. In Fig. 7, for ease of explanation, the order of communication is from the user to the other party.

[0070] The translation system 1 receives the user's voice S １ and the other party's voice S １１ and input the user's voice S １ and the other party's voice S １１ A speech recognition process is performed to recognize segments to be translated from the speech, a text conversion process is performed for the speech that has been recognized, a translation process is performed for the text obtained by the text conversion process into the language of the other party, and a speech conversion process is performed for the translated text obtained by the translation process to produce a synthesized speech S ３１ and the other party's synthesized voice S ３２ and the other party's synthesized voice S ３２ is output by the user-side device 2, and the user-side synthesized voice S ３１ is output by the other device 3, and an operation is performed based on the translation method.

[0071] Specifically, the user device 2 starts up the browser application 240 based on the user's operation, and displays the website 70 on the display unit 22 (Step 1).

[0072] The user device 2 starts calling the telephone number of the party selected by the user from the website 70 displayed on the display unit 22 (Step 2).

[0073] When the other device 3 receives the call from the user device 2, a call is initiated (Step 3).

[0074] After the call starts, the user device 2 receives the user voice signal S １ is input to the voice input unit 20, the user's voice S １ is output to the translation API 6 via the communication unit 25 and the network 4 (Step 4).

[0075] The translation API 6 translates the input user's voice S １ The user-side synthesized speech S is processed by speech recognition, text conversion, translation, and voice conversion. ３１ (Step 5). The translation API 6 generates the user's voice S １ and user-side synthesized voice S ３１ The translated text is then output to the user device 2 via the network 4 (Step 6).

[0076] The user device 2 receives the superimposed user voice S via the communication unit 25. １ and user-side synthesized voice S ３１ The user-side device 2 outputs the translated text to the other-side device 3 via the communication unit 25 (Step 7). １ and user-side synthesized voice S ３１ While the translation API 6 is being output, new translation is temporarily stopped (Step 8).

[0077] The other device 3 receives the user's voice S input via the network 4 and the communication unit 35. １ and user-side synthesized voice S ３１ is output from the output unit 31, and the translated text is displayed on the display unit 32 (Step 9).

[0078] The user side device 2 receives the user side voice S １ and user-side synthesized voice S ３１ After the command is output, the translation pause is resumed (Step 10). The user-side device 2 resumes the translation pause based on the voice input from the voice input unit 30 of the other-side device 3, for example.

[0079] The other device 3 receives the other device's voice S １１ When the voice is input to the voice input unit 30, it is output to the user device 2 via the communication unit 35 and the network 4 (Step 11).

[0080] The user device 2 receives the voice of the other party S input via the network 4 and the communication unit 25. １１ is output to the translation API 6 via the communication unit 35 and the network 4 (Step 12).

[0081] The translation API 6 translates the input voice of the other party S １１ The voice recognition process, the text conversion process, the translation process and the voice conversion process are performed on the other side of the synthesized voice S ３２ The translation API 6 generates the speech S １１ and the other party's synthesized voice S ３２ and then output the translated text to the user device 2 (Step 14).

[0082] The user device 2 receives the superimposed voice of the other party S via the network 4 and the communication unit 25. １１ and the other party's synthesized voice S ３２ is output from the output unit 21, and the translated text is displayed on the display unit 22 (Step 15).

[0083] The translation system 1 repeats these operations until the call ends, and ends the translation operation when the call ends.

[0084] The order of translation is not limited to the order from the user to the other party, but is arbitrary, and the speech of the user or the other party is translated consecutively.

[0085] (Effects of the First Embodiment) The translation system 1 according to this embodiment can smoothly carry out telephone conversations even when the languages ​​used are different. Specifically, the translation system 1 allows a user-side device 2 and a remote-side device 3 to communicate via P2P communication, and each device's speech is automatically translated into the user's language and the remote-side language. This allows a person who does not speak the remote language to smoothly and easily hold a telephone conference or talk without an interpreter, compared to cases where the users take turns translating or only one device translates.

[0086] The translation system 1 continuously recognizes and translates without requiring any operation other than that required for making a phone call, so that a user can make a call without being aware of it, compared to a case where a button operation or the like is required for translation.

[0087] Because the translation system 1 uses a translation API 6 residing in the cloud 5 for speech recognition and other functions used for translation, it can use a web API that offers superior update frequency, number of languages ​​that can be translated, translation accuracy, and translation speed, depending on the situation, compared to systems built into the device. For example, if the other device 3 is installed in a hospital to receive calls from patients, the translation system 1 can accurately translate medical terms and other terms, allowing the hospital to accurately understand the patient's symptoms and provide appropriate responses and effective treatment if the other party speaks a different language. The translation system 1 can also be used in a variety of situations, including business and tourism.

[0088] The translation system 1 can pause the translation process while synthetic speech is being output, thereby preventing the speech for the call and the synthetic speech from being mixed together in the translation, compared to when the process is not paused.

[0089] Since the translation system 1 outputs the original speech and the translated synthetic speech in a superimposed manner, it is easier to capture nuances that cannot be grasped from the synthetic speech alone compared to when only the synthetic speech is output.

[0090] Second Embodiment The second embodiment differs from the first embodiment in that the first user device is the counterpart device.

[0091] In the following embodiments, parts having the same functions and configurations as those in the first embodiment are given the same reference numerals as those in the first embodiment, and the description thereof will be omitted.

[0092] In the translation system 1 of this embodiment, the first user-side device is the other-side device, and the second user-side device is the user-side device.

[0093] In other words, in the translation system 1, the second user device is configured as an IP telephone terminal. When a user calls the first user device from the second user device, which is also an IP telephone terminal, the first user device performs translation processing into each other's language. If the language used by the user is unknown, the AI ​​language recognition of the translation API 6 automatically recognizes the language and performs translation.

[0094] (Effects of the Second Embodiment) The translation system 1 of this embodiment allows smooth communication between people who speak different languages ​​without the need for an interpreter, even when the caller is using an IP telephone terminal.

[0095] Here, another embodiment may be provided as a program for executing a translation method, or as a computer-readable recording medium on which this program is recorded, which includes inputting a first user-side speech and a second user-side speech, performing a speech recognition process to recognize segments to be translated from the input first user-side speech and second user-side speech, converting the speech that has been recognized into text, translating the text obtained from the text conversion process into the language of the other party, converting the translated text obtained from the translation process into speech to generate a first user-side synthetic speech and a second user-side synthetic speech, outputting the second user-side synthetic speech on the first user-side device, and outputting the first user-side synthetic speech on the second user-side device.

[0096] The translation method is further configured to suspend the speech recognition process while the first user-side synthesized speech and the second user-side synthesized speech are being output.

[0097] The first user-side device, translation system 1, and translation method according to the above-described embodiment may be partially realized by a computer-executed program, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like, depending on the intended use.

[0098] Although several embodiments of the present invention have been described above, these embodiments are merely examples and do not limit the scope of the invention as claimed. These novel embodiments may be embodied in various other forms, and various omissions, substitutions, modifications, etc. may be made without departing from the spirit of the present invention. Furthermore, not all combinations of features described in these embodiments are necessarily essential to solving the problems of the invention. Furthermore, these embodiments are included within the scope and spirit of the invention, and are also included in the scope of the inventions and their equivalents as defined in the claims.

[0099] According to the present invention, communication can be smoothly carried out even when the languages ​​used by the parties are different.

[0100] 1...translation system, 2...user-side device, 3...other-side device, 4...network, 5...cloud, 6...translation API, 7...web server, 20...voice input unit, 21...output unit, 22...display unit, 23...input unit, 24...storage unit, 25...communication unit, 26...user-side control unit, 30...voice input unit, 31...output unit, 32...display unit, 33...input unit, 34...storage unit, 35...communication unit, 36...other-side control unit, 60...voice recognition processing unit, 61...text processing unit, 62...translation processing unit, 63...voice processing unit, 70...website, S １ ...User voice, S １１ ...Other party voice, S ３１ ...User-side synthesized voice, S ３２ … Synthesized voice from the other party

Claims

1. A first user-side device comprising: a first user-side voice input unit for inputting a first user-side voice; a first user-side output unit for outputting a second user-side synthetic voice; a first user-side communication unit for communicating with a second user-side device via a network; and a translation unit for generating a first user-side synthetic voice by translating the first user-side voice inputted to the first user-side voice input unit into a second user-side language, generating the second user-side synthetic voice by translating the second user-side voice inputted via the first user-side communication unit into the first user-side language, outputting the second user-side synthetic voice from the first user-side output unit, and outputting the second user-side synthetic voice to the second user-side device.

2. The first user-side device described in claim 1, wherein the translation unit is connected to a translation mechanism via the network, and causes the translation mechanism to perform a speech recognition process for recognizing segments to be translated from the first user-side speech and the second user-side speech, a text conversion process for the speech that has been subjected to the speech recognition process, a translation process for the text obtained by the text conversion process into the language of the other party, and a speech conversion process for the translated text obtained by the translation process, thereby generating the first user-side synthetic speech and the second user-side synthetic speech.

3. The first user-side device according to claim 2, wherein the translation unit temporarily suspends the speech recognition process while the first user-side synthetic speech and the second user-side synthetic speech are being output.

4. A first user-side device having a first user-side voice input unit for inputting a first user-side voice, a first user-side output unit for outputting a second user-side synthetic voice, a first user-side communication unit for communicating with a second user-side device via a network, and a translation unit for generating a first user-side synthetic voice by translating the first user-side voice inputted to the first user-side voice input unit into a second user-side language, generating the second user-side synthetic voice by translating the second user-side voice inputted via the first user-side communication unit into the first user-side language, outputting the second user-side synthetic voice from the first user-side output unit, and outputting the second user-side synthetic voice to the second user-side device via the network; a translation mechanism connected to the network, which performs a speech recognition process to recognize segments to be translated from the first user-side speech and the second user-side speech, a text conversion process of the speech that has been subjected to the speech recognition process, a translation process into a partner language of the text obtained by the text conversion process, and a text conversion process of the translated text obtained by the translation process to generate the first user-side synthetic speech and the second user-side synthetic speech; and a second user-side device having a second user-side speech input unit that inputs the second user-side speech, a second user-side output unit that outputs the first user-side synthetic speech, a second user-side communication unit that communicates with the first user-side device via the network, and a second user-side control unit that outputs the first user-side synthetic speech obtained via the second user-side communication unit from the second user-side output unit.

5. A translation method comprising: inputting a first user-side voice and a second user-side voice; performing a speech recognition process to recognize segments to be translated from the input first user-side voice and the second user-side voice; converting the speech that has been subjected to the speech recognition process into text; performing a translation process into a partner language of the text obtained by the text conversion process; performing a speech conversion process on the translated text obtained by the translation process to generate a first user-side synthetic voice and a second user-side synthetic voice; outputting the second user-side synthetic voice on a first user-side device; and outputting the first user-side synthetic voice on a second user-side device.

Images