Speech translation program

Abstract

To provide means for stimulating communication between people who converse with each other.SOLUTION: A speech translation program executed on a portable information terminal (1) causes a control unit (2) of the information terminal to execute: a step (S4) of recognizing input speech of a speaker and outputting information on the recognition results; a step (S5) of translating a language spoken by a speaker into another language based on the output information on the recognition results, and outputting the information of the translation results; a step (S6) of speech-synthesizing words in the other language so as to be able to play back in speech made to resemble voice of the speaker based on the output information of the translation results; and steps (S8, S9) of playing back the words of the other language in the speech made to resemble voice of the speaker by the speech-synthesis.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a speech translation program, and more particularly to a speech translation program capable of invigorating communication between people having a conversation. [Background technology]

[0002] In recent years, with the progress of globalization, cross-border activities overseas are increasing in various industries and fields, including business. The number of individuals traveling abroad for tourism and other purposes is also increasing worldwide. When working or traveling abroad, people need to communicate using different languages, but many people are not good at foreign languages, and it remains difficult for foreigners to communicate adequately with each other.

[0003] Thus, for example, Patent Document 1 discloses an example of a convenient speech translation device that helps foreigners communicate with each other. This conventional speech translation device recognizes the speaker's voice, for example Japanese voice, input from a microphone. Next, it converts the recognized Japanese voice into, for example, English voice. It then has the function of outputting the converted English voice from a speaker. Therefore, by using this speech translation device, even a Japanese person who is not good at English can easily communicate with English-speaking foreigners.

[0004] However, although conventional speech translation devices can communicate with foreigners who speak different languages ​​through their speech translation functions, the speech emitted from the speaker is halting, has a monotonous intonation, and is unnatural. Furthermore, the emitted speech is immediately recognizable as a synthetic voice far from the speaker's voice, and the listener may feel a sense of incongruity between the voice of the speaker that he or she hears directly and the voice emitted by the speech translation device during the conversation. For this reason, with conventional speech translation devices, conversations are limited to single questions such as "Where is the station?" Therefore, even if there is more to talk about or to ask, the conversation does not get exciting and does not continue, and as a result, communication between the people speaking is not activated. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2004-212685 A Summary of the Invention [Problem to be solved by the invention]

[0006] The present invention is intended to solve the above problems, and has an object to provide a means for invigorating communication between people having a conversation. [Means for solving the problem]

[0007] The speech translation program of the present invention is a speech translation program executed on a portable information terminal, and is characterized in that it causes a control unit of the information terminal to execute the following steps: recognizing the input speech of a speaker and outputting information on the recognition result; translating the language spoken by the speaker into another language based on the output information on the recognition result and outputting information on the translation result; synthesizing speech based on the output information on the translation result so that words in the other language can be played in a voice that resembles the speaker's voice; and playing back the words in the other language in a voice that resembles the speaker's voice using the speech synthesis. Effect of the Invention

[0008] The speech translation program according to the present invention can provide a means for invigorating communication between people having a conversation. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a block diagram showing a hardware configuration of a mobile terminal. [Diagram 2] FIG. 2 is a diagram illustrating the software module configuration of a speech translation program. [Diagram 3] FIG. 2 is a diagram showing a network configuration of a mobile terminal and a server related to speech translation. [Figure 4] 13 is a flowchart showing the flow of a speech translation process in a mobile terminal. [Diagram 5] FIG. 13 is a sequence diagram showing exchanges relating to speech translation between a mobile terminal and a server (when a translation server is specified). [Figure 6] FIG. 13 is a sequence diagram showing exchanges relating to speech translation between a mobile terminal and a server (when a translation server is not specified). DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. 1 is a block diagram showing a hardware configuration of a mobile terminal 1 according to an embodiment of the present invention. The mobile terminal 1 of this embodiment is a portable information terminal that can be worn and carried around, such as a smartphone. The speech translation program according to the present invention corresponds to application software that is installed and operated on this mobile terminal 1, and is a program having a function of translating, for example, Japanese words spoken by a speaker into English and reproducing the English words in a voice that resembles the speaker's voice.

[0011] The mobile terminal 1 of this embodiment includes a control unit 2, a storage unit 3, a wireless communication unit 4, a baseband processing unit 5, a sound input / output unit 6, a display unit 7, and an operation unit 8. The control unit 2 has an MPU (Micro Processing Unit) and executes programs such as a predetermined basic OS (Operating System), middleware, etc. By executing these programs, the control unit 2 controls each part of the mobile terminal 1, constructs a native platform environment and an application execution environment in terms of software configuration, and performs various processes required for the mobile terminal 1.

[0012] The storage unit 3 includes a RAM (Random Access Memory) as a volatile memory and an internal memory storage as a non-volatile memory, so-called a ROM (Read Only Memory). A dynamic RAM or the like is used for the RAM of the storage unit 3, and a flash memory or the like is used for the ROM. The storage unit 3 stores an operating system program for controlling the mobile terminal 1, a driver program used for processing in each part of the mobile terminal 1, an application program related to speech translation, and the like in the ROM. In particular, the storage unit 3 stores basic OS such as Android (registered trademark) OS and iOS (registered trademark) as the operating system program. The storage unit 3 also stores various text data, voice data, image data, and the like, and temporarily stores in the RAM data of the calculation result generated in the process of calculation by a predetermined process executed by the control unit 2.

[0013] The wireless communication unit 4 functions as a wireless LAN communication means for communicating via a wireless LAN access point and as a mobile communication means for communicating via a cellular communication base station, and is composed of, for example, a synthesizer, a frequency converter, a high-frequency amplifier, an antenna, etc. The wireless communication unit 4 executes high-frequency signal processing for performing wireless communication with the access point by a predetermined wireless LAN communication method such as IEEE802.11. The wireless communication unit 4 also executes high-frequency signal processing for performing wireless communication with the base station by a predetermined cellular communication method such as 3G, LTE (Long Term Evolution), or 5G (fifth generation mobile phone system). The baseband processing unit 5 executes digital signal processing for performing voice communication and data communication with communication terminals such as other mobile phones and various servers. The baseband processing unit 5 and the wireless communication unit 4 are connected via a D / A converter and an A / D converter.

[0014] The sound input / output unit 6 is composed of a microphone, a speaker, a signal processing circuit, etc. The sound input / output unit 6 converts an analog signal of sound input from the microphone into a digital signal using the signal processing circuit, and outputs it to the control unit 2, the baseband processing unit 5, etc. The sound input / output unit 6 also converts a digital signal input from the control unit 2, the baseband processing unit 5, etc. into an analog signal using the signal processing circuit, and outputs it to the speaker. The speaker outputs voice, sound effects such as incoming mail, music, etc. to the outside in response to the input of this analog signal. The display unit 7 is configured with a liquid crystal display, an organic EL (Electro-Luminescence) display, or the like, and displays various images based on instructions from the control unit 2. The operation unit 8 is composed of a touch panel built into the display unit 7, various operation buttons provided on the mobile terminal 1, a power switch as a means for turning the power on and off, etc. This operation unit 8 is used when the user turns the power of the mobile terminal 1 on and off, starts and ends applications, selects menus, gives instructions for switching screens, etc., changes settings required for the operation of various software, inputs text information, etc.

[0015] The mobile terminal 1 also includes a GPS (Global Positioning System) unit 9 that acquires location information, a camera unit 10 that captures images, a sensor unit 11, a power supply unit 12 that supplies power to each unit of the mobile terminal 1, an I / O port 13, and a clock unit (not shown). The GPS unit 9 is composed of a GPS receiving module, a GPS antenna, etc., and receives radio waves (positioning signals) from multiple GPS satellites positioned around the Earth, and calculates data regarding the latitude, longitude and altitude of the current position of the mobile terminal 1 based on the reception results. The camera unit 10 is composed of a lens, an imaging device, etc., and is used to capture images of people, scenery, etc. As the imaging device, a CCD (Charge Coupled Device) image sensor or a CMOS image sensor is used. The sensor unit 11 includes, for example, an acceleration sensor, a gyro sensor, a geomagnetic sensor, etc. Based on the output of the sensor unit 11, data relating to the position, direction, attitude, and movement of the mobile terminal 1 can be calculated. The power supply unit 12 includes a rechargeable battery, a power supply circuit that converts the output of the battery into power of a predetermined voltage and supplies it to each unit of the mobile terminal 1, and a charging circuit for charging the battery. The I / O port 13 is a port for connecting a cable or the like equipped with a connector based on the USB (Universal Serial Bus) Type-C (trademark) or Lightning (trademark) standard. The control unit 2 can transmit and receive data to and from an external device via a cable connected to this I / O port 13 using the connector. In addition, the battery of the mobile terminal 1 can be charged by supplying power from the outside via a cable connected to this I / O port 13. In this case, the power supplied from the outside via the cable is sent to a charging circuit of the power supply unit 12. The charging circuit uses the power to charge the battery. The clock section (not shown) is composed of a clock circuit and the like, and accurately counts the date and time and generates time information for updating various types of information and the like.

[0016] Next, the module configuration of a speech translation program as application software executed by the control unit 2 of the mobile terminal 1 will be described with reference to FIG. As shown in FIG. 2, the speech translation program of this embodiment mainly comprises a speech recognition module 31, a translation module 32, a speech synthesis module 33, and an avatar control module . The voice recognition module 31 is a software module that recognizes the words spoken by a speaker from the digital signal (voice data) of the speaker's voice that is input from a microphone and converted by the sound input / output unit 6, and outputs the recognition result as text data. From the perspective of future advances in voice recognition in terms of recognition accuracy and speed, and of reducing the load on mobile terminals due to the execution of a voice translation program, the voice recognition module 31 uses a voice recognition function provided as a cloud service by an Internet server. Specifically, for example, any of the voice recognition functions provided by Microsoft (registered trademark), Google (registered trademark), Amazon (registered trademark), etc. is used by using the API of the Speech To Text function. In this case, as shown in Fig. 3, the voice recognition module 31 uses the voice recognition function by communicating with a voice recognition server 53 connected to the Internet 52 via a baseband processing unit and a wireless communication unit of the mobile terminal 1, for example, via wireless communication with a base station 51.

[0017] In using the voice recognition function, although REST (REpresentational State Transfer) API (Application Programming Interface) can be used, it is preferable to use APIs provided by software development kits (SDKs) distributed by various companies. The speaker's voice data output from the sound input / output unit 6 after microphone input is passed to these APIs, and text data as the result of voice recognition returned from each company's server is received from the API. In other words, by using the API in this way, the voice recognition module 31 uses an external server to recognize the speaker's voice, receives the recognition result from the server, and outputs the text data as the result of voice recognition. The REST API is an API that performs an external call procedure to use a WWW (World Wide Web) system / service using HTTP (HyperText Transfer Protocol).

[0018] The translation module 32 is a software module that translates the text data of the speech recognition result output from the speech recognition module 31 into another language and outputs the translated text data as a translation result. From the viewpoint of future advances in language translation in terms of translation accuracy and speed, and of reducing the load on mobile terminals due to the execution of a speech translation program, the translation module 32 uses a language translation function provided as a cloud service. Specifically, for example, the translation module uses language translation functions provided by Microsoft, Google, Amazon, DeepL, etc.

[0019] 3, the translation module 32 communicates with a translation front-end server 54 connected to the Internet 52 via wireless communication with, for example, a base station 51 of the mobile terminal 1, thereby utilizing a language translation function provided by an external translation server 55. In other words, the translation module 32 utilizes the language translation function provided as a cloud service by the translation server 55 connected to the Internet 52 via the translation front-end server 54. In this case, the translation module 32 exchanges information with the translation front-end server 54 via an API that it has designed itself.

[0020] Specifically, the translation module 32 passes the text data of the speech recognition result from the speech recognition module 31 to the translation front-end server 54 as translation request data into another language through its own API. After that, the translation module 32 receives the text data as the translation result from the translation front-end server 54 through its own API. Although it is a uniquely designed API, this API allows the speaker, as a user of the speech translation program, to specify which company's language translation function to use, for example, in terms of the most reliable translation result. Also, if there is no particular specification, the language translation function of the server that returns the translation result the fastest after the translation is requested among the language translation functions provided by the servers of each company is used. In this case, the translation result from the server that returns the result the fastest is adopted as the translation result. This is because, except for cases such as translating special terms used in the medical field, there is currently little difference in translation accuracy between the language translation functions provided by each company, and in such cases, the speed of translation is emphasized.

[0021] In this way, the translation module 32 uses the language translation function provided by the external translation server 55, with the translation front-end server 54 acting as a proxy. The advantage of doing so is that, for example, when using multiple language translation services, the translation process can be centrally controlled, and management of the addition, change, and discontinuation of language translation service providers can be performed separately from the speech translation program. In this case, the speech translation program can simplify the process of using the language translation service. Naturally, since the processes related to this are executed on the translation front-end server 54 side, the load caused by the execution of the processes can be reduced on the mobile terminal running the speech translation program.

[0022] When using a language translation function provided by an external server in the translation front-end server 54, a REST API can be used, as in the case of using the above-mentioned voice recognition function. If each company that provides the language translation function distributes an SDK, it is preferable to use the API provided by the SDK. The translation front-end server 54 passes text data to be translated to these APIs, and receives text data as the translation result into another language returned from each company's server from the API. In detail, the translation front-end server 54 determines the translation server to which the translation is requested (the service user) based on the designated user information of the language translation service (language translation function) received from the translation module 32 of the mobile terminal 1 via its own API. In this determination, if the designated user information of the language translation service is specified, the translation server corresponding to the designated user is determined as the translation server to which the translation is requested. On the other hand, if the designated user information is not specified, all translation servers that are managed as available by the translation front-end server 54 at that time are determined as the translation server to which the translation is requested. Then, the translation front-end server 54 passes the text data to be translated to the API corresponding to the translation server determined in this way. After that, when the translation server returns the translation result into another language, the translation front-end server 54 receives the text data of the translation result from the API.

[0023] In addition, when receiving the translation result, if there is no designation in the designated usage information of the language translation service, the translation result will be returned from all translation servers that were requested to translate as described above. In this case, as described above, the translation result that is returned the earliest after the translation is requested is adopted as the translation result, so the translation front-end server 54 receives the text data of the translation result that is returned the earliest from the server via the API. By using the API in this way, when the translation front-end server 54 receives text data of the speech recognition result from the translation module 32 via its own API, it uses an external server to translate that text data into another language. Then, when the translation front-end server 54 receives the text data of the translation result into another language returned from the external server, it returns that text data to the translation module 32 via its own API as the translation result.

[0024] The voice synthesis module 33 is a software module that synthesizes voice in another language based on the translation result output from the translation module 32, i.e., text data of the translation result obtained by translating the speaker's words into another language, and outputs the voice data (digital signal) of the synthesis result. The voice synthesis module 33 uses a voice synthesis function provided as a cloud service from the viewpoint of the future evolution of voice synthesis in terms of the quality of the synthesized voice and the speed required for synthesis, and from the viewpoint of reducing the load on the mobile terminal due to the execution of the voice translation program. Specifically, for example, any of the voice synthesis functions provided by Microsoft, Google, Amazon, etc. is used by using the API of the TTS (Text To Speech) function. In this case, the voice synthesis module 33 uses the voice synthesis function by communicating with a voice synthesis server 56 connected to the Internet 52 via wireless communication with, for example, a base station 51 of the mobile terminal 1, as shown in FIG. 3.

[0025] Although the speech synthesis function can be used using a REST API, it is preferable to use an API provided by an SDK distributed by each company. Text data of the translation result obtained by translating the speaker's words into another language is passed to these APIs, and speech data returned from each company's server as the speech synthesis result is received from the API. In other words, by using the API in this way, the speech synthesis module 33 uses an external server to synthesize the speaker's words translated into another language, i.e., the text data of the translation result output from the translation module 32. Then, the speech synthesis module 33 receives the speech synthesis result (speech data) from the server, and outputs the speech data (digital signal) as the speech synthesis result.

[0026] Incidentally, the speech translation program of this embodiment can also synthesize speech so that translated words (text data) in another language are reproduced and output from a speaker in a voice that resembles the speaker's own voice when synthesizing the translated words. That is, the program has a function of reproducing translated words in a voice that resembles the speaker's own voice. This function can be used by a user of the speech translation program as a speaker by setting the function to "enable" on, for example, a setting screen of a mobile terminal. When this function is enabled, the speech synthesis module 33 refers to the model ID of the custom voice for the speaker when using a cloud service speech synthesis function provided by, for example, Google, in a call to the API of the TTS section. As a result, the speech translation program of this embodiment reproduces translated words in a voice that resembles the speaker's own voice. Google's custom voice models are created by preparing voice data that contains the speaker's own voice in advance, and using that voice data for advance training (machine learning) on ​​a cloud server. For example, when training a Japanese voice, voice data is prepared that records the user's own voice reading out about 500 Japanese sentences (scripts) provided by Google, such as "Take several photos of stationary objects in a well-lit place."

[0027] Incidentally, for the reasons mentioned above, the voice synthesis module 33 is usually designed to use a voice synthesis function provided as a cloud service by an Internet server. However, when considering the case where the above-mentioned function of reproducing the translated words in a voice that resembles the original voice is enabled, it is not very preferable to use a voice synthesis function provided as a cloud service from the viewpoint of security, such as preventing someone from using that voice to impersonate the original person. For this reason, the speech translation program of this embodiment is provided with a user-selectable option that, when the above function is enabled, uses the speech synthesis function of the mobile terminal 1 to play back the translated words in a voice that resembles the speaker's own voice and output them from the speaker. This selection is made by the user setting the speech synthesis function of the mobile terminal 1 to "use" on, for example, the setting screen of the mobile terminal 1.

[0028] Specifically, when this option is used, the user of the speech translation program reads a specified sentence into the microphone of the mobile terminal 1 in advance, and the mobile terminal 1 learns the characteristics of the voice at the time of reading, and creates a voice model of the user as the learning result. When the OS of the mobile terminal 1 is, for example, iOS, the user uses the Personal Voice function to read randomly selected text displayed on the display unit 7 of the mobile terminal 1 into the microphone of the mobile terminal 1, and the mobile terminal 1 learns the characteristics of the user's voice. For example, when learning an English voice, text such as "Grabbing a cup of coffee this afternoon sounds great." is displayed on the display unit 7, and the user reads this sentence. Through this learning (machine learning), a voice model of the user is created on the mobile terminal 1. Then, in the speech synthesis module 33 of the speech translation program, speech synthesis is performed by calling the API of iOS, specifying the text data translated into another language by the translation module 32 and the name of the user's own voice model created above. In this case, the API provided by the Speech synthesis framework of iOS is used as the API. As a result, the speech translation program uses the user's own voice model created on the mobile terminal 1 to synthesize the words of the text data translated into another language on the mobile terminal 1, and this speech synthesis reproduces the translated words in a voice that resembles the user's own voice. Note that at the time of this application, iOS 17 is used for iOS.

[0029] The avatar control module 34 is a software module that displays an avatar as the user's alter ego on the screen of the display unit 7 according to an option selected by the user, and controls the actions, facial expressions, mouth movements, etc. of the avatar. This selection is made by the user setting the display to "ON" (display avatar) or "OFF" (do not display) on, for example, a setting screen of the mobile terminal 1. When "ON" (display avatar) is set, the user can select the avatar he or she wants to display from among multiple avatars on, for example, the same setting screen. In the voice translation program of this embodiment, in particular, in order to synchronize the movement of the avatar's mouth with the playback of the voice synthesized by the voice synthesis module 33, i.e., the reading of words, the game engine Unity (trademark) and its lip sync assets are used. For the assets, for example, uLipsync, a Unity asset (plug-in) that performs lip sync processing by identifying vowels, consonants, etc. by capturing the characteristics of phonemes based on Mel-Frequency Cepstrum Coefficients (MFCC), is used. Also, the avatar model data is imported into Unity in, for example, a VRM file format for VR (Virtual Reality) applications.

[0030] The speech translation process performed by the speech translation program of this embodiment will be described below with reference to the flowchart in Fig. 4 and the sequence diagrams in Fig. 5 and 6. Note that the process of the flowchart in Fig. 4 is executed by the control unit 2 when, for example, the user taps the icon of the speech translation program displayed on the home screen of the mobile terminal 1 to start the program (application). In step S1, the control unit 2 determines whether or not the avatar display is set to "ON." If the avatar display is set to "ON," the control unit 2 proceeds to step S2 (Yes side), and if the avatar display is not set to "ON" (i.e., if it is set to "OFF"), the control unit 2 proceeds to step S3 (No side). In step S2, the control unit 2 executes the program of the avatar control module 34 and displays an avatar on the screen of the display unit 7. At this time, if an avatar has been selected by the user, the selected avatar is displayed on the screen, and if no avatar has been selected, the avatar that should be displayed in that case is displayed.

[0031] In step S3, when the speaker's voice is input from the microphone, the analog signal from the microphone is converted into a digital signal (voice data) by the sound input / output unit 6, and the control unit 2 acquires the voice data. In step S4, the control unit 2 executes the program of the voice recognition module 31 and passes the voice data to an API for using the voice recognition function, thereby causing an external server to recognize the speaker's voice. After that, the control unit 2 receives text data of the recognition result returned by the server from the API. When passing the voice data to the API, information on a language code corresponding to a source language (e.g., Japanese) in the language translation set in advance by the user is also passed to the API as a recognition language in the voice recognition. For the language code information, a language code conforming to the international standard ISO-639-1 or BCP-47 (e.g., "ja-JP" for Japanese) is used. In addition, the source language in the language translation and the translation target language described later are set by the user via, for example, a setting screen of the mobile terminal 1. When the control unit 2 receives the text data of the recognition result from the API, the control unit 2 outputs the text data as the result of the speech recognition. This output is input to the translation module 32.

[0032] In step S5, the control unit 2 executes the program of the translation module 32, and passes the text data resulting from the speech recognition to the translation front-end server 54 via its own API as translation request data into another language. At that time, the previously mentioned designated usage information is also passed to the translation front-end server 54 via its own API. The designated usage information for specifying the translation request destination (service destination) can be specified (set) by the user on, for example, the setting screen of the mobile terminal 1 by selecting the language translation service (language translation function) that the user wishes to use for translation from one or more translation services displayed on the screen. Note that in the initial state where no specification has been made yet, information indicating "no specification" is set for this designated usage information.

[0033] The translation front-end server 54, which has received this information, then translates the text into another language using a language translation function provided by an external server. When a translation request is made to the translation front-end server 54 using a unique API, the language code information corresponding to the source language (e.g., Japanese) and the target language (e.g., English) in the language translation previously set by the user is also passed through the API. For the language code information, a language code conforming to the above international standard (e.g., "ja-JP" for Japanese, "en-US" for English / US) is used. As described above, the translation front-end server 54 determines the translation server to which the translation is requested (the service user) based on the designated user information of the language translation service received from the translation module 32 of the mobile terminal 1 via its own API. If the designated user information is specified, the translation server corresponding to the designated user is determined as the translation server to which the translation is requested. If there is no specification, all translation servers managed by the translation front-end server 54 as available are determined as the translation server to which the translation is requested. The translation front-end server 54 then passes the text data to be translated to the API corresponding to the translation server thus determined as the destination. At that time, the translation front-end server 54 also passes the language codes (e.g., "ja-JP" for Japanese and "en-US" for English / United States) corresponding to the source language (e.g., Japanese) and the destination language (e.g., English) received via its own API to the API corresponding to the destination translation server.

[0034] After that, when the translation result into another language is returned from the requested translation server, the translation front-end server 54 receives the text data of the translation result from the API. In this reception, if there is no designation in the designated user information, translation results are returned from all translation servers that were requested for translation, and the translation front-end server 54 receives the text data of the translation result that is returned the earliest after the request from the API. In particular, this point is shown in the sequence diagram of FIG. 5 when there is a designation in the designated user information, and in FIG. 6 when there is no designation in the designated user information. Then, when the translation front-end server 54 receives the text data of the translation result into another language returned from the external server, it returns the text data as the translation result to the control unit 2 executing the translation module 32 via its own API. Thereafter, when the control unit 2 receives text data as the translation result into another language from the translation front-end server 54 via its own API, the control unit 2 outputs the text data as the translation result. This output is input to the voice synthesis module 33.

[0035] In step S6, the control unit 2 executes the program of the voice synthesis module 33 and passes the text data of the translation result to an API for utilizing the voice synthesis function, thereby causing an external server to perform voice synthesis of the speaker's words translated into another language. After that, the control unit 2 receives the voice data of the synthesis result returned by the server from the API. When passing the text data of the translation result to the API, information on the language code ("en-US") corresponding to the translation target language (e.g., English / United States) set in advance by the user is also passed to the API as the synthesis language for voice synthesis.

[0036] As described above, the speech translation program of this embodiment has a function of reproducing translated words in a voice similar to that of the speaker. When this function is used, the control unit 2 executes the speech synthesis module 33, calling an API that uses the speech synthesis function of a cloud service to refer to a model ID of a custom voice for the speaker, and causes the server providing the service to perform speech synthesis. In this case, the control unit 2 receives the speech data of the synthesis result returned by the server from the API. On the other hand, when using the voice synthesis function of the mobile terminal for this function, the control unit 2 calls the API of iOS in the case of iOS, for example, when executing the voice synthesis module 33. In this case, the control unit 2 specifies the name of the voice model of the user himself / herself created on the mobile terminal 1 by prior learning when calling the iOS API, and performs voice synthesis only on the mobile terminal 1. Note that the voice synthesis by the cloud or the mobile terminal is performed using a voice model corresponding to the translation target language (e.g. English / US) in the language translation set in advance by the user.

[0037] In step S7, the control unit 2 determines whether or not the avatar display is set to "ON." If the avatar display is set to "ON," the control unit 2 proceeds to step S8 (Yes side), and if the avatar display is not set to "ON" (i.e., if it is set to "OFF"), the control unit 2 proceeds to step S9 (No side).

[0038] In step S8, the control unit 2 executes the program of the avatar control module 34, controls the actions, facial expressions, mouth movements, etc. of the avatar displayed on the screen of the display unit 7, plays the synthesized voice data received from the API through the sound input / output unit 6, and outputs the voice from the speaker. More specifically, in controlling the avatar, the control unit 2 creates an audio file (.WAV format, etc.) in which the synthesized voice data is recorded, and operates Unity to play the voice using this file. This executes lip sync processing by the Unity plug-in uLipsync. In this lip sync processing, uLipsync analyzes the waveform of the voice being played and changes the mouth movement of the avatar so that the mouth shape corresponds to the analysis result, so that the translated voice (words) are played as if the avatar were speaking, although this is a purely subjective view in this case. In other words, this lip sync processing changes the mouth movement of the avatar so that it matches the timing of the speech due to the playback of the translated words (voice) and so that the mouth shape corresponds to the content of the speech.

[0039] In addition, when the function of reproducing the translated words in a voice that resembles the person's own voice (using a voice synthesis function of the mobile terminal, for example, the voice synthesis function of iOS) is used, the control unit 2 performs the following process instead of the process described above. Incidentally, in consideration of the accuracy of lip sync (synchronization timing, etc.), it is desirable to perform lip sync processing based on synthesized voice data. Therefore, in this embodiment, the above-mentioned uLipsync, which analyzes voice data and performs lip sync processing, is adopted. In this case, for example, in iOS, the write(_:toBufferCallback:) method of the Speech synthesis framework is usually used to output the synthesized voice data to an audio file. In this case, the control unit 2 operates Unity to play back the voice using the audio file (.WAV format, etc.) that outputs the synthesized voice data in the control of the avatar. However, in iOS 17 provided at the time of filing this application, it seems that voice data synthesized using the user's own voice model created by machine learning with the Personal Voice function (Personal Voice) cannot be output as an audio file using the write() method. In other words, this is thought to be due to Apple (registered trademark)'s policy, but in iOS 17 at the time of filing this application, it seems that synthesized voice data using Personal Voice cannot be generated as an audio file. Therefore, in this case, lip sync processing using uLipsync cannot be performed, so another method is required for that purpose.

[0040] In this embodiment, as one solution to this problem, the speak(_:) method of the AVSpeechSynthesizer class of iOS is used to read out the text data of the translation result received in step S5 above, and the movement of the avatar's mouth is changed using information on the "characters to be read out" when reading out. Specifically, the information on the "characters to be read out (unit: word)" is Define AVSpeechSynthesizerDelegate functions, such as: speechSynthesizer(_ synthesizer: AVSpeechSynthesizer, willSpeakRangeOfSpeechString characterRange: NSRange, utterance: AVSpeechUtterance) { let reading UtteranceString = ( utterance.speechString as NSString).substring(with: characterRange) } In this example, the information about the "characters to be read" is obtained using readingUtteranceString.

[0041] Next, convert the acquired information about the characters to be read into the International Phonetic Alphabet (IPA), which is a phoneme representation. For example, to convert English words (US) into IPA, you can use a program such as English-to-IPA on GitHub shown at the following URL. https: / / github.com / mphilli / English-to-IPA Next, a viseme ID is obtained from the converted IPA. This can be done, for example, based on a correspondence table between an IPA and a viseme ID that is created in advance. Example of a conversion table (US English): https: / / github.com / MicrosoftDocs / azure-docs.ja-jp / blob / master / articles / cognitive-services / Speech-Service / how-to-speech-synthesis-viseme.md#map-phonemes-to-visemes Then, at a predetermined interval, the avatar's mouth movement is changed for each IPA phoneme so as to conform to the mouth shape indicated by the viseme ID. The above processes are created as Unity plugins or assets.

[0042] In the above method, since the duration of each phoneme on the pronunciation time axis and the time required to pronounce the entire word are unknown, the interval used between IPA phonemes is set to a fixed value (e.g., 200 ms, etc.). From the viewpoint of improving lip sync accuracy, this interval may be calculated using a correction method such as using a weighted value (e.g., multiplied by 0.7 or 1.1) that is a fixed value so that the duration can be changed according to the IPA phoneme. After performing the above processes, the control unit 2 proceeds to step S10.

[0043] In step S9, the control unit 2 plays the voice data of the synthesis result received from the API through the sound input / output unit 6, and outputs the voice from the speaker. When the function of playing back the translated words in a voice similar to the original voice (using a voice synthesis function of the mobile terminal, for example, the voice synthesis function of iOS) is used, the speak(_:) method of the AVSpeechSynthesizer class of iOS can be used to play back the synthesized voice similar to the original voice. In addition, at this time, the control unit 2 displays the text data of the translation result into another language received from the translation front-end server 54 through its own API in the above step S5 on the screen of the display unit 7.

[0044] In step S10, the control unit 2 determines whether an end event has been received. The end event is generated by the OS when, for example, a user instructs the end of the running voice translation program by swiping the screen of the mobile terminal. If an end event has been received, the control unit 2 ends the voice translation program (Yes side). On the other hand, if an end event has not been received, the control unit 2 proceeds to step S3 to continue the above processing of the voice translation program (No side). The speech translation process performed by the speech translation program of this embodiment has been described above.

[0045] The speech translation program of this embodiment has a function of reproducing and speaking translated words in a voice similar to that of the user as described above, as one of the means for activating communication between people who are conversing. When this function is used, the speech synthesis module 33 of the speech translation program uses a voice model of the user that has previously learned (machine learning) the characteristics of the user's voice when reading a given sentence, and performs speech synthesis so that the translated words (text) are spoken in a voice similar to that of the user. In this case, when the synthesized voice data is reproduced, the translated words are reproduced in a voice similar to that of the user who is the speaker, so that the listener will feel less discomfort between the voice of the speaker that the listener hears directly and the voice of the mobile terminal 1 during the conversation. Furthermore, when reading out a given "sentence" in this way and learning the user's voice in advance (machine learning), it is possible to learn not only characters and words, but also acoustic features such as accents and intonations. Therefore, by using a voice model created through this learning, it is possible to synthesize speech so that the translated words (text) are spoken in a relatively natural voice. Therefore, the speech translation program of this embodiment enables a conversation to continue without being limited to a single question, and allows a person to talk to the person he or she is speaking with about things they want to say or ask. This allows the conversation to continue and become more lively, which in turn activates communication between the people speaking.

[0046] Furthermore, the speech translation program of this embodiment has a function of displaying an avatar as described above as one of the means for stimulating communication between people conversing. When this function is used, the avatar control module 34 of the speech translation program displays an avatar on the display unit 7 of the mobile terminal 1 during a conversation, and changes the movement of the avatar's mouth so as to match the timing of the utterance of translated words reproduced and so as to form a mouth shape according to the content of the utterance. Recently, theme songs from Japanese animations (anime) have topped the US Billboard charts, trading cards with pictures or photos of anime characters are sold at high prices internationally, and many foreigners are dressing up as anime characters. In addition, in recent years, it has become commonplace to use fictional characters or anime characters as avatars in virtual reality, metaverses, virtual YouTubers, and the like. Furthermore, it is clear from television and internet news that Japanese anime has become familiar and loved by people in various countries, including Europe, America, and Asia, in recent years. For these reasons, it is certain that if communication is made through avatars of fictional characters or anime characters, the avatars will serve as a catalyst for more opportunities to start a conversation and continue the conversation. Therefore, according to the speech translation program of this embodiment, the conversation will not be limited to a single question, but will become lively through communication with the avatar displayed, thereby making it possible to stimulate communication between the people conversing.

[0047] In the above, it has been explained that the speech translation program according to the present invention is installed and operated on a portable information terminal that can be worn and carried around, such as a smartphone. However, it goes without saying that the speech translation program according to the present invention can also be operated on other general computers, such as tablet, notebook, and desktop (tower) PCs. However, considering use, for example, when traveling abroad, it is preferable to install and operate the speech translation program according to the present invention on a portable information terminal (mobile terminal) such as a smartphone. [Explanation of symbols]

[0048] 1 Mobile terminal (portable information terminal), 2 control section, 3 storage section, 4. Wireless communication unit, 5. Baseband processing section; 6 Sound input / output section, 7 Display section, 8 Operation section, 31 Voice recognition module, 32 Translation Modules, 33 Speech synthesis module, 34 Avatar control module, 51 base station, 52 Internet, 53 Voice recognition server, 54 Translation front-end server, 55 Translation Server, 56 Speech synthesis server

Claims

1. A speech translation program executed on a portable information terminal, A step of recognizing an inputted voice of a speaker and outputting information of the recognition result; a step of translating the language spoken by the speaker into another language based on the output information of the recognition result, and outputting information of the translation result; a step of synthesizing the words in the other language based on the output information of the translation result so that the words can be reproduced in a voice similar to the voice of the speaker; reproducing the words in the other language in a voice that resembles the voice of the speaker by the voice synthesis; causing a control unit of the information terminal to execute the above. A speech translation program comprising:

2. 2. The speech translation program according to claim 1, A step of displaying a predetermined sentence on a display unit of the information terminal and creating a voice model for the speaker on the information terminal by machine learning characteristics of the voice of the speaker who reads the sentence; performing the voice synthesis using the voice model created on the information terminal; causing a control unit of the information terminal to execute the above. A speech translation program comprising:

3. 3. The speech translation program according to claim 1, displaying an avatar on a display unit of the information terminal; changing the movement of the mouth of the avatar so as to match the timing of the speech produced by the reproduction of the words and so as to change the shape of the mouth according to the content of the speech; causing a control unit of the information terminal to execute the above. A speech translation program comprising:

4. 4. The speech translation program according to claim 3, The translation into the other language is performed by using one or more cloud services via an external server; When the speaker designates designated user information designating the use of one of the cloud services, transmitting the designated user information to the server to request the translation; receiving, from the server, translation information into the other language obtained by using a cloud service specified in the specified usage information, and outputting the received translation information into the other language as information of a translation result; causing a control unit of the information terminal to execute the above. A speech translation program comprising:

5. 4. The speech translation program according to claim 3, The translation into the other language is performed by using one or more cloud services via an external server; When the speaker does not specify designated user information that specifies the use of one of the cloud services, transmitting information indicating that there is no specification as the designated user information to the server and requesting the translation; receiving, from the server, translation information into the other language obtained from a cloud service that returns translation information into the other language earliest among all of the cloud services, and outputting the received translation information as information of a translation result; causing a control unit of the information terminal to execute the above. A speech translation program comprising:

Images