First user-side device, translation system, and translation method

Abstract

To provide a first user-side device, a translation system, and a translation method for enabling smooth calls even when languages used by each other are different.SOLUTION: A user-side device 2 is generally configured to include a voice input unit 20 that inputs a user-side voice S1, an output unit 21 that outputs an other-side synthetic voice S32, a communication unit 25 that communicates with an other-side device 3 via a network 4, and a user-side control unit 26 that generates a user-side synthetic voice S31 by translating the user-side voice S1 inputted to the voice input unit 20 into an other-side language, generates the other-side synthetic voice S32 by translating an other-side voice S11 inputted via the communication unit 25 into a user-side language, outputs the other-side synthetic voice S32 from the output unit 21, and outputs the user-side synthetic voice S31 to the other-side device 3.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a first user-side device, a translation system, and a translation method. [Background technology]

[0002] As a conventional technology, there is known an IP network translation method for Internet Protocol (IP) telephones that use the Internet, which automatically identifies the language used by the caller and the language used by the recipient based on the IP number at the time of IP telephone connection, and performs automatic translation for telephone calls that go through an IP network (see, for example, Patent Document 1).

[0003] This IP network translation method uses a translation server installed on the IP network to automatically translate between the language used by the caller and the language used by the recipient, allowing voice calls to be made while the voice between IP phones is automatically translated. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2004-260430 A Summary of the Invention [Problem to be solved by the invention]

[0005] Conventional IP network translation methods have the problem that when identifying the language used by the caller and the language used by the recipient based on the IP number, the language used by both IP phones within the same country is treated as the same language even if the languages ​​used are different.

[0006] Therefore, an object of the present invention is to provide a first user side device, a translation system and a translation method that enable smooth communication even when different languages ​​are used. [Means for solving the problem]

[0007] One aspect of the present invention provides a first user-side device comprising a first user-side voice input unit for inputting a first user-side voice, a first user-side output unit for outputting a second user-side synthetic voice, a first user-side communication unit for communicating with a second user-side device via a network, and a translation unit for generating a first user-side synthetic voice by translating the first user-side voice inputted to the first user-side voice input unit into a second user-side language, generating a second user-side synthetic voice by translating the second user-side voice inputted via the first user-side communication unit into the first user-side language, outputting the second user-side synthetic voice from the first user-side output unit, and outputting the second user-side synthetic voice to the second user-side device.

[0008] Another aspect of the present invention relates to a first user-side device having a first user-side voice input unit for inputting a first user-side voice, a first user-side output unit for outputting a second user-side synthetic voice, a first user-side communication unit for communicating with a second user-side device via a network, and a translation unit for generating a first user-side synthetic voice by translating the first user-side voice inputted to the first user-side voice input unit into a second user-side language, generating a second user-side synthetic voice by translating the second user-side voice inputted via the first user-side communication unit into the first user-side language, outputting the second user-side synthetic voice from the first user-side output unit, and outputting the second user-side synthetic voice to the second user-side device via the network; and A translation system is provided which performs a speech recognition process to recognize segments to be translated from a second user-side speech, a text conversion process of the speech that has been subjected to the speech recognition process, a translation process into a partner language of the text obtained by the text conversion process, and a speech conversion process of the translated text obtained by the translation process to generate a first user-side synthetic speech and a second user-side synthetic speech, and a second user-side device which has a second user-side speech input unit that inputs the second user-side speech, a second user-side output unit that outputs the first user-side synthetic speech, a second user-side communication unit that communicates with the first user-side device via a network, and a second user-side control unit that outputs the first user-side synthetic speech obtained via the second user-side communication unit from the second user-side output unit.

[0009] Another aspect of the present invention provides a translation method which includes inputting a first user-side speech and a second user-side speech, performing a speech recognition process to recognize segments to be translated from the input first user-side speech and second user-side speech, converting the speech that has been recognized into text, performing a translation process into a partner language for the text obtained by the text conversion process, and converting the translated text obtained by the translation process into text to generate a first user-side synthetic speech and a second user-side synthetic speech, outputting the second user-side synthetic speech on a first user-side device, and outputting the first user-side synthetic speech on a second user-side device. Effect of the Invention

[0010] According to the present invention, a telephone conversation can be smoothly carried out even if the two parties use different languages. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an example of a translation system. [Diagram 2] FIG. 2(a) is an example of a block diagram of a user-side device, FIG. 2(b) is an example of a block diagram of a counterpart-side device, and FIG. 2(c) is a diagram for explaining an example of a translation API process. [Diagram 3] FIG. 3 is a diagram showing an example of a language selection screen displayed on the user side device. [Figure 4] FIG. 4 is a diagram showing an example of a standby screen displayed on the user side device. [Diagram 5] FIG. 5 is a diagram showing an example of a phone book screen displayed on the user side device. [Figure 6] FIG. 6 is a diagram showing an example of a control screen displayed on the user side device. [Figure 7] FIG. 7 is a timing chart showing an example of the operation of the translation system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] [Outline of the embodiment] The translation system with AI (Artificial Intelligence) interpretation according to the embodiment is generally configured by combining a multi-function mobile phone or an IP telephone terminal with an AI interpretation device, as an example. The translation system according to the embodiment recognizes the voice of a conversation taking place on a multi-function mobile phone or an IP telephone terminal with an AI interpretation device, and outputs the voice translated by streaming processing from the multi-function mobile phone or the IP telephone terminal. This allows the translation system according to the embodiment to hold telephone conferences or calls with users who speak different languages ​​without the need for an interpreter. The AI ​​interpretation is a translation that combines multiple WebAPIs (Web Application Programming Interfaces) that exist in the cloud, as will be described later.

[0013] [First embodiment] (Overview of Translation System 1) FIG. 1 is a diagram showing an example of a translation system according to a first embodiment. FIG. 2(a) is an example of a block diagram of a user-side device according to the first embodiment, FIG. 2(b) is an example of a block diagram of a destination-side device, and FIG. 2(c) is a diagram for explaining an example of a translation API process. In each diagram according to the embodiments described below, the ratios and shapes between figures may differ from the actual ratios and shapes. In addition, in FIG. 2(a) to FIG. 2(c), the flow of main voices and synthetic voices is indicated by arrows.

[0014] The translation system 1 of this embodiment is a system that enables a first user using a user-side device 2 as a first user-side device to make a call with a second user using a counterpart device 3 as a second user-side device by P2P (Peer to Peer) communication via WebRTC (Web Real Time Communication). The white arrow in FIG. 1 indicates a state in which the user-side device 2 and the counterpart device 3 make a call by P2P communication. In addition, when the first user uses a first language and the second user uses a second language different from the first language, the translation system 1 translates the first language into the second language and the second language into the first language by a translation process described later, making it possible to make a smooth call even if the two parties do not understand each other's languages. Note that if at least one of the user-side device 2 and the counterpart device 3 has an imaging unit, the audio is separated from the video and output to WebRTC.

[0015] In this embodiment, a case will be described in which a first user makes a call to a second user, and the user-side device 2 of the first user further has a translation unit described later. In this embodiment, the first user who makes the call will be described as the user, and the second user who receives the call will be described as the other party. The user-side device 2 and the other party-side device 3 are, for example, a personal computer, a multi-function mobile phone, a phablet terminal, a tablet terminal, etc.

[0016] As shown in Fig. 1, the translation system 1 is generally configured to include a user device 2, a partner device 3, and a translation API 6 that functions as a translation mechanism. The translation mechanism may be a translation device connected to a network 4. The translation API 6 of this embodiment performs processing as a translation mechanism using multiple cloud APIs of a cloud 5.

[0017] As shown in FIG. 2(a), the user-side device 2 includes a voice input unit (first user-side voice input unit) 20 for inputting a user-side voice (first user-side voice) S1, and a counterpart-side synthetic voice (second user-side synthetic voice) S2. 32a communication unit (first user side communication unit) 25 that communicates with the other device 3 via a network 4; and a user side synthetic voice (first user side synthetic voice) S that is obtained by translating the user side voice S1 input to the voice input unit 20 into a language of the other device (second user side language). 31 and converts the voice of the other party (voice of the second user) S 11 Synthesized speech S on the other side translated into the user's language (first user's language) 32 Generate the synthetic voice S 32 is output from the output unit 21, and the user-side synthetic voice S 31 and a user-side control unit 26 serving as a translation unit that outputs the message to the other device 3. The network 4 is, for example, the Internet.

[0018] As an example, the user side device 2 further includes a display unit 22, an input unit 23, and a storage unit 24, as shown in FIG. 2(a).

[0019] As shown in FIG. 2(b), the other device 3 receives the other party's voice S 11 A voice input unit (second user-side voice input unit) 30 for inputting a user-side synthetic voice S 31 a communication unit (second user side communication unit) 35 that communicates with the user side device 2 via a network 4; and a user side synthetic voice S acquired via the communication unit 35. 31 and a counterpart control unit (second user side control unit) 36 that causes the output unit 31 to output the above.

[0020] The user side device 2 further includes a display unit 32, an input unit 33, and a storage unit , as an example.

[0021] As shown in FIG. 2(c), the translation API 6 is connected to the network 4 and receives the user's voice S1 and the other party's voice S2. 11A speech recognition process is performed to recognize segments to be translated from the speech, a text conversion process is performed for the speech obtained by the speech recognition process, a translation process is performed for the text obtained by the text conversion process into the language of the other party, and a speech conversion process is performed for the translated text obtained by the translation process to generate a synthetic speech S on the user side. 31 and the other party's synthesized voice S 32 The present invention is generally configured to generate

[0022] Specifically, the translation API 6 is generally configured to include, for example, a voice recognition processing unit 60 that performs voice recognition processing, a text conversion processing unit 61 that performs text conversion processing, a translation processing unit 62 that performs translation processing, and a voice conversion processing unit 63 that performs voice conversion processing. For example, the voice recognition processing unit 60, the text conversion processing unit 61, the translation processing unit 62, and the voice conversion processing unit 63 are WebAPIs configured using AI technology.

[0023] 1, the translation API 6 is configured using a plurality of APIs such as a voice recognition processing unit 60. Therefore, when there are a plurality of APIs capable of executing the same process, the voice recognition processing unit 60, the text processing unit 61, the translation processing unit 62, and the voice processing unit 63 are configured to have the other APIs execute the process depending on the processing speed, congestion, etc.

[0024] (Configuration of user device 2) The voice input unit 20 of the user side device 2 is a microphone. The output unit 21 is a speaker, but is not limited thereto, and may be configured to be connected to earphones or the like by wire or wirelessly. The display unit 22 is, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display. The input unit 23 is, for example, a keyboard, a mouse, a touch pad, or the like. In the user side device 2 of this embodiment, the display unit 22 and the input unit 23 are configured as a touch panel in which the input unit 23 is superimposed on the display unit 22. The storage unit 24 is, for example, a storage device such as a HDD (Hard Disk Drive) or an SSD (Solid State Drive). The storage unit 24 stores a browser application 240. The communication unit 25 is configured to exchange information and signals with the network 4 and the cloud 5.

[0025] The user-side control unit 26 is a microcomputer that includes a CPU (Central Processing Unit) that performs calculations and processing on acquired data according to stored applications, a RAM (Random Access Memory) that is a semiconductor memory, and a ROM (Read Only Memory). The ROM stores, for example, a program for operating the user-side control unit 26. The RAM is used, for example, as a storage area for temporarily storing calculation results.

[0026] The user-side control unit 26 functions as a translation unit by reading the browser application 240 from the storage unit 24 and accessing a website 70 of the web server 7, which will be described later. The website 70 enables two-way communication between the user-side device 2 and the other-side device 3.

[0027] The user-side control unit 26 is connected to the translation API 6 via the network 4, and transmits the user-side voice S1 and the other-side voice S 11 The translation API 6 executes speech recognition processing, text conversion processing, translation processing, and speech conversion processing, and the user's side synthetic speech S 31 and the other party's synthesized voice S 32Generate.

[0028] A user who uses the translation system 1 pre-registers his / her service number, name, telephone number, language used, etc. in a telephone directory DB (Data Base) 71 described later. The user-side control unit 26 recognizes the language used based on this telephone directory DB 71 and causes the translation API 6 to execute the translation. If the user is not pre-registered in the telephone directory DB 71, the user-side control unit 26 causes the translation API 6 to execute the translation based on the language used that the user selected on the website 70.

[0029] As shown in FIG. 2(c), the voice recognition processing unit 60 recognizes the user's voice S1 and the other party's voice S 11 When speech 600, which is the speech in the form of a single word, is input, it recognizes segments in order to convert it into text. Next, when post-recognition processing speech 601 output from the speech recognition processing unit 60 is input, a text conversion processing unit 61 performs text conversion processing. Next, a translation processing unit 62 translates the text 602 output from the text conversion processing unit 61 into the user's language or the recipient's language. Next, a voice conversion processing unit 63 performs voice conversion processing on the translation text 603 output from the text conversion processing unit 61, to generate synthetic voice 604. This synthetic voice 604 is called the user's synthetic voice S 31 and the other party's synthesized voice S 32 It is.

[0030] The translation text 603 is translated into the user's language and output to the user-side device 2, and translated into the other party's language and output to the other party's device 3. 11 The other device 3 can display text obtained by translating the user's voice S1 into the user's own language.

[0031] The user-side control unit 26 generates the user-side synthetic voice S 31 and the other party's synthesized voice S 32is output, the speech recognition process is paused. The pause of the speech recognition process is performed to prevent a loop in which the output synthetic speech is input from each speech input unit and the process proceeds. This pause is performed when the synthetic speech is output, after a break is recognized and a sentence is confirmed, but is not limited to this. In addition, when the speech recognition process is paused, the synthetic speech to be played next may be stored in the storage unit 24 after a translation process is performed, for example.

[0032] The user-side control unit 26 synthesizes the user-side voice S1 and the user-side synthesized voice S 31 and output to the other device 3, and the other device's voice S 11 and the other party's synthesized voice S 32 The voice and the synthetic voice are superimposed and output from the output unit 21 on the user side. This allows the user and the other party to talk as if simultaneous interpretation were being performed. The voice and the synthetic voice may be superimposed at the same volume, or one of them may be made lower in volume before being superimposed.

[0033] (Configuration of remote device 3) The voice input unit 30 of the other device 3 is a microphone. The output unit 31 is a speaker, but is not limited thereto, and may be configured to be connected to earphones or the like by wire or wirelessly. The display unit 32 is, for example, a liquid crystal display or an organic EL display. The input unit 33 is, for example, a keyboard, a mouse, or a touch pad. As an example, the other device 2 may be configured as a touch panel in which the display unit 32 and the input unit 33 are superimposed on the display unit 32. The storage unit 34 is, for example, a storage device such as an HDD or SSD. The communication unit 35 is configured to exchange information and signals with the network 4 and the cloud 5.

[0034] The other-side control unit 36 ​​is, for example, a microcomputer including a CPU, a RAM, and a ROM.

[0035] (Web Server 7) The Web server 7 is connected to the network 4. The Web server 7 has a Web site 70 and a telephone directory DB 71. The Web site 70 enables P2P communication to a registered telephone number. The telephone number is registered in the telephone directory DB 71 that the Web server 7 has.

[0036] The user can operate the user device 2 to display this Web site 70, and can select and confirm the telephone number of the person he or she wishes to call from the displayed telephone numbers, thereby making it possible to call the other person.

[0037] (Display screen of user device 2) Fig. 3 is a diagram showing an example of a language selection screen displayed on the user-side device according to the first embodiment. Fig. 4 is a diagram showing an example of a standby screen displayed on the user-side device according to the first embodiment. Fig. 5 is a diagram showing an example of a phone book screen displayed on the user-side device according to the first embodiment. Fig. 6 is a diagram showing an example of a control screen displayed on the user-side device according to the first embodiment.

[0038] The language selection screen, the standby screen, the phone book screen, and the control screen shown in FIGS. 3 to 6 are examples of screens that are displayed when the user launches the browser application 240 of the user-side device 2 and opens the Web site 70.

[0039] 3 is, as an example, a screen that displays a plurality of language icons 220a and a scroll bar 220b. The language icon 220a is selected and determined by a touch operation on the input unit 23. When the user wishes to select the D language, the user can set the language to be used to the D language by performing a touch operation on the language icon 220a that displays "D language."

[0040] Scroll bar 220b is operated when a user wishes to select a language other than "Language A" to "Language E" displayed in FIG. 3, for example. A user can display a language other than the displayed language by moving the black portion in the figure up and down. Translation system 1 of this embodiment supports languages ​​from 130 countries and regions, for example, but by using multiple translation APIs 6, it is possible to translate languages ​​from more countries and regions.

[0041] 4 is, for example, a screen displayed when making a call from the user-side device 2 to the other-side device 3. This display screen 221 is, for example, a screen displaying a other party display area 221a, a selection area 221b, a status display area 221c, a translation display 221d, a microphone display 221e, a translation stop icon 221f, a language switching icon 221g, a text save icon 221h, an end icon 221i, a phone number selection icon 221j, a phone number display area 221k, a call start icon 221l, an entry icon 221m, an exit icon 221n, and a text display area 221o.

[0042] The other party display area 221a is an area where, for example, an image of the other party operating the other party's device 3, the start and time of video or audio recording, etc. are displayed.

[0043] Selection area 221b is, for example, an area displaying a "mic" radio button that allows the voice input unit 20 to be turned on or off, and a "voice" radio button that allows the user to select whether or not to mix voice and synthetic voice.

[0044] The status display area 221c is an area that displays the current status, for example. The status display area 221c displays "Waiting" while waiting for the other party to answer a call, and displays "Call in progress" when the call starts.

[0045] The translation display 221d displays, for example, the language to be translated into. In Fig. 4, it shows that Japanese is translated into English and English is translated into Japanese.

[0046] The microphone display 221e, as an example, indicates the on / off state of the microphone.

[0047] As one example, the translation stop icon 221f is an icon that stops translation when a touch operation is received during translation, and starts translation when another touch operation is received.

[0048] The language switching icon 221g is, for example, an icon that allows switching of the language to be translated.

[0049] The text save icon 221h is, for example, an icon for saving the text displayed in the text display area 221o in the storage unit 24.

[0050] The end icon 221i is, for example, an icon for ending a call.

[0051] The telephone number selection icon 221j is, for example, an icon that is operated when selecting a party to call from a telephone directory. This telephone directory is displayed based on the telephone directory DB71.

[0052] The telephone number display area 221k is, for example, an area for displaying a selected telephone number.

[0053] The call start icon 221l is, for example, an icon that is operated when making a call to a selected phone number.

[0054] The user-side device 2 and the other-side device 3 are configured to connect the PSTN (Public Switched Telephone Network) to WebRTC and stream audio between each other. The entry icon 221m is an icon operated when entering this WebRTC room. The exit icon 221n is an icon operated when leaving the WebRTC room. In FIG. 4, the state is before entering the room, so the exit icon 221n is displayed grayed out.

[0055] The text display area 221o displays, for example, the user's voice S1 and the other party's voice S 11 This is the area where the translated text is displayed.

[0056] In the display screen 222 shown in Fig. 5, for example, a phone book screen 222a displayed when selecting a phone number to call is superimposed on the display screen 221. In the phone book screen 222a, for example, a close icon 222b, an add icon 222c, an edit icon 222d, a delete icon 222e, a search window 222f, a refine icon 222g, a reset icon 222h, a phone number display area 222i, a select icon 222j, and a cancel icon 222k are displayed. In Fig. 5, the same icons as those in the display screen 221 shown in Fig. 4 are denoted by the same reference numerals as those in Fig. 4.

[0057] The close icon 222b is, for example, an icon that is operated when closing the phone book screen 222a.

[0058] The add icon 222c is, for example, an icon that is operated when adding a phone number.

[0059] The edit icon 222d is, for example, an icon that is operated when editing a phone number or the like.

[0060] The delete icon 222e is, for example, an icon that is operated when deleting a registered phone number or the like.

[0061] The search window 222f is, for example, an area for inputting a search word for searching a registered telephone number or the like.

[0062] The refinement icon 222g is, for example, an icon that is operated when searching the telephone directory with the search word entered in the search box 222f.

[0063] The reset icon 222h is, for example, an icon that is operated when erasing the search word inputted in the search box 222f.

[0064] The phone number display area 222i is, for example, an area in which a "check box," a "selected name," a "phone number," and a "language" are displayed. When a user checks the "check box" next to the "selected name" of the person they wish to call, they can call the corresponding phone number. The "language" indicates the language used by the person displayed in the selected name.

[0065] The select icon 222j is, for example, an icon that is operated when calling a phone number selected in the phone number display area 222i.

[0066] The cancel icon 222k is, for example, an icon that is operated when finishing editing or selecting a phone number or the like, or when closing the phone book screen 222a.

[0067] The display screen 223 shown in Fig. 6 shows, as an example, a control screen displayed during a phone call. As an example, a new control area 223a is displayed on this display screen 223, and the other party display area 221a and the like are moved downward. In Fig. 6, the same icons and the like as those in the display screen 221 shown in Fig. 4 are given the same reference numerals as in Fig. 4.

[0068] The control area 223a is an area where whether or not text is being saved, the other party's phone number, and the like are displayed.

[0069] (operation) An example of the operation of the translation system 1 according to the present embodiment will be described below with reference to the timing chart of Fig. 7. In Fig. 7, for ease of explanation, the order of communication is from the user to the other party.

[0070] The translation system 1 receives a user's voice S1 and a counterpart's voice S 11 The input user's voice S1 and the other party's voice S11 A speech recognition process is performed to recognize segments to be translated from the speech, the speech that has been recognized is converted into text, the text obtained by the text conversion process is translated into the language of the other party, and the translated text obtained by the translation process is converted into voice to produce a synthetic voice S 31 and the other party's synthesized voice S 32 Generate the synthetic voice S 32 is output by the user side device 2, and the user side synthetic voice S 31 is output by the other device 3, and an operation is performed based on the translation method.

[0071] Specifically, the user-side device 2 launches the browser application 240 based on the user's operation, and displays the website 70 on the display unit 22 (Step 1).

[0072] The user-side device 2 starts calling the telephone number of the other party selected by the user from the Web site 70 displayed on the display unit 22 (Step 2).

[0073] When the other party's device 3 receives the call from the user's device 2, a call is started (Step 3).

[0074] After starting a call, when a user's voice S1 is input to the voice input unit 20, the user's device 2 outputs the user's voice S1 to the translation API 6 via the communication unit 25 and the network 4 (Step 4).

[0075] The translation API 6 performs speech recognition processing, text conversion processing, translation processing, and voice conversion processing on the input user-side voice S1, and generates a user-side synthetic voice S 31 The translation API 6 generates the user-side speech S1 and the user-side synthetic speech S 31 The translated text is then output to the user device 2 via the network 4 (Step 6).

[0076] The user-side device 2 receives the superimposed user-side voice S1 and the user-side synthetic voice S 31The user-side device 2 outputs the translated text to the other-side device 3 via the communication unit 25 (Step 7). 31 While outputting the translation, the translation API 6 is controlled to suspend new translation (Step 8).

[0077] The other device 3 receives the user's voice S1 and the user's synthetic voice S2 via the network 4 and the communication unit 35. 31 is output from the output unit 31, and the translated text is displayed on the display unit 32 (Step 9).

[0078] The user-side device 2 receives the user-side voice S1 and the user-side synthesized voice S 31 After the command is output, the pause of the translation is released (Step 10). The user-side device 2 releases the pause of the translation based on the voice input from the voice input unit 30 of the other-side device 3, for example.

[0079] The other device 3 receives the other device's voice S 11 When the voice is input to the voice input unit 30, the voice is output to the user side device 2 via the communication unit 35 and the network 4 (Step 11).

[0080] The user device 2 receives the voice of the other party S input via the network 4 and the communication unit 25. 11 is output to the translation API 6 via the communication unit 35 and the network 4 (Step 12).

[0081] Translation API6 translates the input voice of the other party 11 The voice recognition process, text conversion process, translation process and voice conversion process are performed on the other party's synthetic voice S 32 (Step 13). The translation API 6 generates the voice of the other party S 11 and the other party's synthesized voice S 32 and then outputting the translated text to the user device 2 (Step 14).

[0082] The user device 2 receives the superimposed voice of the other party S input via the network 4 and the communication unit 25.11 and the other party's synthesized voice S 32 is output from the output unit 21, and the translated text is displayed on the display unit 22 (Step 15).

[0083] The translation system 1 repeats these operations until the telephone call is ended, at which point the translation operation is ended.

[0084] The order of translation is not limited to from the user to the other party, but is arbitrary, and the speech of the user or the other party is translated consecutively.

[0085] (Effects of the First Embodiment) The translation system 1 according to the present embodiment allows smooth telephone conversations even when different languages ​​are used. Specifically, the translation system 1 allows a user device 2 and a counterpart device 3 to communicate by P2P communication, and automatically translates each other's voice into the user's language and the counterpart's language, so that even a person who does not speak the counterpart's language can smoothly and easily hold a telephone conference or talk without an interpreter, compared to a case where each device takes turns translating or only one device translates.

[0086] The translation system 1 continuously recognizes and translates without requiring any operations other than those required for making a phone call, so the user can make a call without being aware of it, compared to when button operations are required for translation.

[0087] The translation system 1 uses a translation API 6 in the cloud 5 for the speech recognition unit used for translation, so it can use WebAPIs that have a higher update count, a higher number of languages ​​that can be translated, and a higher translation accuracy and speed, depending on the situation, compared to when the WebAPIs are built into the device. For example, if the other device 3 is installed in a hospital to receive calls from patients, the translation system 1 can translate medical terms and other terms with high accuracy, allowing the hospital to accurately understand the patient's symptoms and provide appropriate responses and effective treatment when the other party speaks a different language. The translation system 1 can also be used in a variety of situations, such as business and tourism.

[0088] The translation system 1 can pause the translation process while the synthetic voice is being output, thereby preventing the voice for the call and the synthetic voice from being mixed together and translated, compared to when the process is not paused.

[0089] Since translation system 1 outputs the original speech and the translated synthetic speech in a superimposed manner, it is easier to capture nuances that cannot be grasped from the synthetic speech alone, compared to when only the synthetic speech is output.

[0090] [Second embodiment] The second embodiment differs from the first embodiment in that the first user side device is the counterpart side device.

[0091] In the embodiments described below, parts having the same functions and configurations as those in the first embodiment are given the same reference numerals as those in the first embodiment, and descriptions thereof will be omitted.

[0092] In the translation system 1 of this embodiment, the first user-side device is the other party-side device, and the second user-side device is the user-side device.

[0093] In other words, in the translation system 1, the second user device is configured as an IP telephone terminal. When a user calls the first user device from the second user device, which is an IP telephone terminal, the first user device performs a translation process into each other's language. If the language used by the user is unknown, the AI ​​language recognition of the translation API 6 automatically recognizes the language and performs the translation.

[0094] (Advantages of the second embodiment) The translation system 1 of this embodiment enables smooth communication between people who speak different languages ​​without the need for an interpreter, even if the caller is an IP telephone terminal.

[0095] Here, another embodiment may be provided as a program for executing a translation method, or a computer-readable recording medium having this program recorded thereon, which includes inputting a first user-side speech and a second user-side speech, performing a speech recognition process to recognize segments to be translated from the input first user-side speech and second user-side speech, converting the speech that has been recognized into text, performing a translation process into the other party's language for the text obtained by the text conversion process, and converting the translated text obtained by the translation process into text to generate a first user-side synthetic speech and a second user-side synthetic speech, outputting the second user-side synthetic speech on a first user-side device, and outputting the first user-side synthetic speech on a second user-side device.

[0096] The translation method further includes pausing the speech recognition process while the first user-side synthetic speech and the second user-side synthetic speech are being output.

[0097] The first user-side device, translation system 1, and translation method according to the above-described embodiment may be realized, for example, in part by a program executed by a computer, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), etc., depending on the application.

[0098] Although several embodiments of the present invention have been described above, these embodiments are merely examples and do not limit the invention according to the claims. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, etc. can be made without departing from the gist of the present invention. In addition, not all combinations of features described in these embodiments are necessarily essential to the means for solving the problems of the invention. Furthermore, these embodiments are included within the scope and gist of the invention, and are included within the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0099] 1...translation system, 2...user device, 3...other device, 4...network, 5...cloud, 6...translation API, 7...web server, 20...voice input unit, 21...output unit, 22...display unit, 23...input unit, 24...storage unit, 25...communication unit, 26...user control unit, 30...voice input unit, 31...output unit, 32...display unit, 33...input unit, 34...storage unit, 35...communication unit, 36...other control unit, 60...voice recognition processing unit, 61...text processing unit, 62...translation processing unit, 63...voice processing unit, 70...website, S1...user voice, S 11 …Other party voice, S 31 ...User-side synthesized voice, S 32 … Synthetic voice from the other party

Claims

1. a first user side voice input unit for inputting a first user side voice; a first user-side output unit that outputs a second user-side synthetic speech; a first user communication unit that communicates with a second user device via a network; a translation unit that generates a first user-side synthetic voice by translating the first user-side voice inputted to the first user-side voice input unit into a second user-side language, generates the second user-side synthetic voice by translating the second user-side voice inputted via the first user-side communication unit into the first user-side language, outputs the second user-side synthetic voice from the first user-side output unit, and outputs the second user-side synthetic voice to a second user-side device; A first user device comprising:

2. the translation unit is connected to a translation mechanism via the network, and causes the translation mechanism to execute a speech recognition process for recognizing segments to be translated from the first user-side speech and the second user-side speech, a text conversion process for the speech that has been subjected to the speech recognition process, a translation process for the text obtained by the text conversion process into a partner language, and a speech conversion process for the translated text obtained by the translation process, thereby generating the first user-side synthetic speech and the second user-side synthetic speech.

2. A first user side device according to claim 1.

3. the translation unit suspends the speech recognition process while the first user-side synthetic speech and the second user-side synthetic speech are being output; 3. The first user side device according to claim 2.

4. a first user-side device having a first user-side voice input unit for inputting a first user-side voice, a first user-side output unit for outputting a second user-side synthetic voice, a first user-side communication unit for communicating with a second user-side device via a network, and a translation unit for generating a first user-side synthetic voice by translating the first user-side voice inputted to the first user-side voice input unit into a second user-side language, generating the second user-side synthetic voice by translating the second user-side voice inputted via the first user-side communication unit into the first user-side language, outputting the second user-side synthetic voice from the first user-side output unit, and outputting the second user-side synthetic voice to the second user-side device via the network; a translation mechanism connected to the network, which performs a speech recognition process for recognizing segments to be translated from the first user-side speech and the second user-side speech, a text conversion process for the speech that has been subjected to the speech recognition process, a translation process for the text obtained by the text conversion process into a partner language, and a speech conversion process for the translated text obtained by the translation process to generate the first user-side synthetic speech and the second user-side synthetic speech; a second user-side device having a second user-side voice input unit for inputting the second user-side voice, a second user-side output unit for outputting the first user-side synthetic voice, a second user-side communication unit for communicating with the first user-side device via the network, and a second user-side control unit for outputting the first user-side synthetic voice acquired via the second user-side communication unit from the second user-side output unit; A translation system with

5. A first user side voice and a second user side voice are input; performing a speech recognition process for recognizing segments to be translated from the input first user-side speech and the input second user-side speech; converting the speech that has been subjected to the speech recognition processing into text; A translation process is performed on the text obtained by the text conversion process into a target language; performing a speech conversion process on the translation text obtained by the translation process to generate a first user-side synthetic speech and a second user-side synthetic speech; outputting the second user-side synthetic speech on a first user-side device; outputting the first user-side synthetic speech on a second user-side device; How to translate.

Images