system

Abstract

We provide the system. [Solution] A means for acquiring voice data from the user, A means of sending the acquired audio data to the server, A means by which the server converts audio data into text data, A means for the server to detect the language of the text data, A means for the server to translate text data into the target language, A means by which the server converts translated text data into audio data, A means of sending translated audio data to the user's device, A system that includes means for a device to play translated audio data.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the modern global business environment, communication between multiple languages is becoming increasingly important. However, it is technically difficult to achieve real-time voice translation that supports multiple languages, and particularly, the translation accuracy and speed in real time are issues. In addition, stable service provision that is independent of the type of terminal used by the user and the network environment is also required. Existing systems cannot sufficiently meet these requirements, and smooth communication between users of multiple languages is hindered.

Means for Solving the Problems

[0005] This invention provides a system for real-time multilingual speech translation in real-world and web meetings using a PC or smartphone. The system includes means for acquiring speech data from a user, means for transmitting the acquired speech data to a server, means for the server to convert the speech data into text data, means for the server to detect the language of the text data, means for the server to translate the text data into a target language, means for the server to convert the translated text data back into speech data, means for transmitting the translated speech data to the user's terminal, and means for the terminal to play the translated speech data. As a result, users can enjoy real-time, highly accurate multilingual translation, enabling smooth communication. Furthermore, by utilizing external APIs for speech recognition and translation processing, the system's processing power and accuracy are improved, ensuring stable service provision.

[0006] A "user" refers to a person who operates or uses a system.

[0007] "Audio data" refers to digital data that includes audio signals acquired through audio input devices such as microphones.

[0008] "Device" refers to electronic devices such as PCs, smartphones, and tablets used by users.

[0009] A "server" refers to a central computing system that processes and manages data transmitted from terminals via a network.

[0010] A "speech recognition engine" refers to a collection of software and hardware used to convert speech data into text data.

[0011] "Text data" refers to character data converted from speech data by a speech recognition engine.

[0012] A "language detection algorithm" refers to software and computational methods for automatically identifying the language of text data.

[0013] A "translation engine" refers to a collection of software and hardware used to translate text data from one language into another.

[0014] A "speech synthesis engine" refers to a collection of software and hardware used to convert text data into speech data.

[0015] "Means of playback" refers to the function of outputting audio data acquired by the device as sound through speakers, headphones, or other means.

[0016] An "external API" refers to external speech recognition, translation, and speech synthesis functions that are used through program interfaces provided by other services or systems.

[0017] A "multilingual speech translation system" refers to a comprehensive system for translating speech in multiple different languages ​​in real time. [Brief explanation of the drawing]

[0018] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Mode for Carrying Out the Invention

[0019] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0020] First, the language used in the following description will be explained.

[0021] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of a plurality of arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of a plurality of types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0022] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0023] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0024] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0026] [First Embodiment]

[0027] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0028] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0031] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0034] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0038] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0039] This invention relates to a system that translates multilingual speech in real time using a PC or smartphone. This system can translate user voice input into another language in real time and output it back as speech. The program processing of this system is described below in natural language.

[0040] 1. Acquisition of voice input

[0041] The user uses the microphone on their PC or smartphone to perform voice input. For example, User A says, "Hello, nice to meet you."

[0042] 2. Sending audio data to the server

[0043] The terminal temporarily stores this audio data and sends it to the server. At this time, the audio data is converted to a specific format (for example, a WAV file).

[0044] 3. Perform speech recognition

[0045] The server inputs the received voice data to a speech recognition engine and converts the voice into text data. As the speech recognition engine, it is possible to use an external speech recognition API (for example, a general cloud-based service). Here, text data such as "Hello, nice to meet you" is generated.

[0046] 4. Language Detection

[0047] The server uses a language detection algorithm to automatically detect the language of this text data. This algorithm can also use an external language detection API. For example, it is detected that this text data is in Japanese.

[0048] 5. Execution of Translation

[0049] The server inputs the detected Japanese text to a translation engine to translate it into the target language. As the translation engine, an external translation API can be used. Here, the text "Hello, nice to meet you" is translated into Chinese as "你好,很高兴认识你".

[0050] 6. Speech Synthesis of Translated Text

[0051] The server inputs the translated Chinese text to a speech synthesis engine to convert it into voice data. As the speech synthesis engine, an external speech synthesis API can be used. Here, the translated text "你好,很高兴认识你" is converted into Chinese voice data.

[0052] 7. Transmission of Voice Data to the Terminal

[0053] The server converts the generated voice data into a certain format (for example, an MP3 file) and prepares to transmit it to the user's terminal. This voice data is transmitted to the terminal.

[0054] 8. Output of Translated Voice

[0055] The terminal receives the voice data sent from the server. The terminal plays back this voice data through a playback device (such as a speaker or headphones) and provides the translated voice to the user. In this way, the voice of user A, "Hello, nice to meet you for the first time," is transmitted to user B as the Chinese voice "Ni hao, hen gao xing ren shi ni."

[0056] As a specific example, when user A (a Japanese speaker) communicates with Chinese-speaking user B in a real meeting, by using this system, the speech of user A is quickly translated into Chinese and transmitted to user B. In this way, smooth communication among multiple languages is realized.

[0057] As described above, the system of the present invention enables smooth communication among multiple languages by translating the user's voice into multiple languages in real time and outputting it as voice again. This system features flexibility and high-precision processing by using external APIs and meets the needs of large-scale global communication.

[0058] The processing flow will be described below.

[0059] Step 1:

[0060] The user inputs voice towards the microphone of a PC or smartphone. For example, say "Hello, nice to meet you for the first time."

[0061] Step 2:

[0062] The terminal temporarily stores the voice signal obtained from the microphone as digital voice data.

[0063] Step 3:

[0064] The terminal converts the voice data recorded by the terminal into a certain format (for example, a WAV file).

[0065] Step 4:

[0066] The device sends voice data to the server over the network.

[0067] Step 5:

[0068] The server receives the audio data.

[0069] Step 6:

[0070] The server inputs the received audio data into a speech recognition engine, which then converts the audio data into text data. An external speech recognition API (such as a common cloud-based service) is used as the speech recognition engine.

[0071] Step 7:

[0072] The server retrieves the text data returned by the speech recognition engine (for example, "Hello, nice to meet you").

[0073] Step 8:

[0074] The server inputs text data into a language detection algorithm to detect the language of the text data. The language detection algorithm also utilizes an external language detection API.

[0075] Step 9:

[0076] The server detects that the language of the text data is Japanese.

[0077] Step 10:

[0078] The server inputs Japanese text data into a translation engine to translate it into a target language (e.g., Chinese). An external translation API is used as the translation engine.

[0079] Step 11:

[0080] The server obtains the translated text data returned from the translation engine (e.g., "Hello, nice to meet you").

[0081] Step 12:

[0082] The server inputs the translated text data into a speech synthesis engine and converts it into audio data. As the speech synthesis engine, an external speech synthesis API is utilized.

[0083] Step 13:

[0084] The server obtains the audio data returned from the speech synthesis engine (e.g., Chinese audio data).

[0085] Step 14:

[0086] The server converts the generated audio data into a certain format (e.g., MP3 file).

[0087] Step 15:

[0088] The server transmits the audio data to the terminal.

[0089] Step 16:

[0090] The terminal receives the audio data sent from the server.

[0091] Step 17:

[0092] The terminal plays the received audio data through a playback device (speaker or headphones) and provides the translated audio to the user.

[0093] Through the above steps, the user's Japanese voice input is translated into Chinese in real time with high precision and transmitted to other users as audio.

[0094] (Example 1)

[0095] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0096] In today's global society, facilitating smooth communication between people who speak different languages ​​is crucial. However, real-time translation and playback of speech across multiple languages ​​is technically complex, and building a user-friendly system is a particularly challenging task. To overcome this challenge, efficient and highly accurate speech recognition, translation, and speech synthesis are necessary, along with a flexible and reliable system to achieve these capabilities.

[0097] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0098] In this invention, the server includes means for acquiring audio data from a user, means for transmitting the acquired audio data to the server, means for the server to convert the audio data into text data, means for the server to detect the language of the text data, means for the server to translate the text data into a target language, means for the server to convert the translated text data into audio data, means for transmitting the translated audio data to the user's terminal, means for the terminal to play the translated audio data, means for the server to convert the format of the audio data using an external audio processing API, and means for the terminal to output the received audio data to a predetermined playback device. This enables real-time audio translation and playback between users using different languages.

[0099] "Audio data" refers to data that records and stores the voice spoken by a user in digital format.

[0100] A "server" is a computer system that provides various services over a network, and it plays a role in processing and converting audio data.

[0101] "Text data" refers to data obtained by converting audio data into written text.

[0102] "Language detection" is the process of identifying the language of text data.

[0103] The "target language" is the language that will be output as a result of the translation.

[0104] "Speech recognition" is a technology that analyzes speech data and converts it into corresponding text data.

[0105] "Translation" is the process of converting text data written in one language into another language.

[0106] "Speech synthesis" is a technology that generates speech data based on text data.

[0107] An "external speech processing API" is an interface that provides external services and functions for performing speech recognition and speech synthesis on speech data.

[0108] A "device" refers to a device used by a user, such as a PC or smartphone.

[0109] A "playback device" refers to a device such as a speaker or headphones used to output audio data as sound.

[0110] This invention is a system that allows users to translate multilingual speech in real time using a PC or smartphone. The program processing of this system is described below in natural language.

[0111] To implement this system, the following hardware and software will be used:

[0112] Hardware: PC, smartphone, microphone, speakers, headphones

[0113] Software: Google (registered trademark) Cloud Speech-to-Text API, Google Cloud Translation API, Google Cloud Text-to-Speech API

[0114] First, the user performs voice input using the microphone of a PC or smartphone. For example, User A utters the words "Hello, nice to meet you." The terminal temporarily stores this voice data, converts it into the WAV format, and sends it to the server.

[0115] The server inputs the received voice data into the Google Cloud Speech-to-Text API and converts it into text data "Hello, nice to meet you." Next, the server automatically detects the language of this text data using the Google Cloud Translation API and determines that it is Japanese.

[0116] Next, the server translates the Japanese text into Chinese using the Google Cloud Translation API. For example, the text "Hello, nice to meet you" is converted into the Chinese "你好,很高兴认识你". After that, the server inputs this translated text into the Google Cloud Text-to-Speech API to generate Chinese voice data "你好,很高兴认识你".

[0117] The server converts the generated voice data into the MP3 format and sends it to the terminal. The terminal outputs the received voice data to a playback device (speaker or headphones) to convey it to User B. In this way, User A's voice is conveyed to User B as Chinese voice.

[0118] As a specific example, when User A communicates with Chinese-speaking User B in a real meeting, by using this system, User A's speech is quickly translated into Chinese and conveyed to User B. Thereby, smooth communication among multiple languages is realized.

[0119] Examples of prompt sentences to be input into the AI model:

[0120] When implementing a program for speech recognition, language detection, translation, and speech synthesis, please follow the following steps:

[0121] 1. The user spoke in Japanese, "Hello, nice to meet you for the first time." The terminal temporarily stores this voice data, converts it into the WAV file format, and sends it to the server.

[0122] 2. The server receives the WAV file and uses the Google Cloud Speech-to-Text API to convert it into the text "Hello, nice to meet you for the first time."

[0123] 3. The server automatically detects the language of this text data using the Google Cloud Translation API and determines that it is Japanese.

[0124] 4. The server translates the text data into Chinese using the Google Cloud Translation API.

[0125] 5. The translated text "Hello, nice to meet you" is converted into voice data using the Google Cloud Text-to-Speech API.

[0126] 6. The generated voice data is converted into the MP3 format and sent to the terminal.

[0127] 7. The terminal outputs the received voice data through a playback device and provides it to User B.

[0128] As described above, the system of the present invention enables smooth communication between multiple languages by translating the user's voice into multiple languages in real time and outputting it as voice again. This system features flexibility and high-precision processing by using external APIs, and meets the needs of large-scale global communication.

[0129] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0130] Step 1:

[0131] The user uses the microphone on their PC or smartphone to perform voice input. The subject is the user. The specific actions in this step are as follows:

[0132] Input: The user says, "Hello, nice to meet you."

[0133] Output: Input to the terminal as audio data.

[0134] Operation: The user picks up their smartphone and speaks into the microphone, or speaks into the microphone on their PC. The voice is captured by the microphone and temporarily stored as digital audio data.

[0135] Step 2:

[0136] The terminal temporarily stores the audio data, converts it to WAV format, and sends it to the server. The subject is the terminal. The specific actions in this step are as follows:

[0137] Input: Temporarily stored audio data.

[0138] Output: WAV format audio data sent to the server.

[0139] Operation: The device converts temporarily stored audio data into a WAV file format and sends it to the server.

[0140] Step 3:

[0141] The server inputs the received audio data into the Google Cloud Speech-to-Text API, converting the audio data into text data. The subject is the server. The specific actions in this step are as follows:

[0142] Input: WAV format audio data received from the terminal.

[0143] Output: Text data "Hello, nice to meet you for the first time".

[0144] Operation: The server sends the WAV format audio data to the Google Cloud Speech-to-Text API and obtains the returned text data.

[0145] Step 4:

[0146] The server automatically detects the language of the text data. The subject is the server. The specific operations in this step are as follows.

[0147] Input: Text data "Hello, nice to meet you for the first time".

[0148] Output: A determination that it is Japanese.

[0149] Operation: The server inputs the text data into the Google Cloud Translation API and receives the language detection result.

[0150] Step 5:

[0151] The server translates the Japanese text using the Google Cloud Translation API. The subject is the server. The specific operations in this step are as follows.

[0152] Input: Japanese text data "Hello, nice to meet you for the first time".

[0153] Output: Chinese text data "你好,很高兴认识你".

[0154] Action: The server sends Japanese text to the Google Cloud Translation API and receives the translated Chinese text.

[0155] Step 6:

[0156] The server converts the translated Chinese text into audio data using the Google Cloud Text-to-Speech API. The subject is the server. The specific actions in this step are as follows.

[0157] Input: Chinese text data "你好,很高兴认识你".

[0158] Output: Chinese audio data.

[0159] Action: The server sends the Chinese text to the Google Cloud Text-to-Speech API and obtains the returned audio data.

[0160] Step 7:

[0161] The server converts the generated audio data into the MP3 format and sends it to the user's terminal. The subject is the server. The specific actions in this step are as follows.

[0162] Input: The translated audio data.

[0163] Output: The audio data in MP3 format sent to the user terminal.

[0164] Action: The server converts the audio data into the MP3 format and prepares to send it to the terminal. The converted audio data is sent to the user's terminal.

[0165] Step 8:

[0166] The terminal receives the audio data sent from the server and plays it through the playback device. The subject is the terminal. The specific actions in this step are as follows.

[0167] Input: MP3 audio data received from the server.

[0168] Output: The translated audio "Hello, I'm happy to have met you" is played.

[0169] Operation: The device receives audio data in MP3 format and outputs it through its built-in playback device (speaker or headphones). Communication is established when User B listens to the audio.

[0170] (Application Example 1)

[0171] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0172] In autonomous vehicles, it is difficult to facilitate rapid and accurate communication between passengers who speak multiple languages, as well as between passengers and the system. The present invention aims to provide a system that translates speech in real time and enables smooth communication between passengers who speak different languages.

[0173] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0174] In this invention, the server includes means for acquiring voice data from a user, means for transmitting the acquired voice data to the server, means for the server to convert the voice data into text data, means for the server to detect the language of the text data, means for the server to translate the text data into a target language, means for the server to convert the translated text data back into voice data, means for transmitting the translated voice data to the user's terminal, means for the terminal to play the translated voice data, and means integrated into the vehicle system to translate passenger voices into other languages ​​in real time and output them as voice. This enables smooth multilingual communication within an autonomous vehicle.

[0175] A "user" is an individual or group that uses the system.

[0176] "Audio data" refers to information recorded in digital format from the voice spoken by a user.

[0177] A "server" is a computer system that provides functions such as processing, converting, and transmitting audio data.

[0178] "Text data" refers to data in a format where audio data has been converted into written text.

[0179] "Language detection" is the process of determining which language text data is written in.

[0180] The "target language" is the target language to which the translated text will be converted.

[0181] Translation is the process of converting text written in one language into another language.

[0182] "Speech synthesis" is a technology that converts text data into speech data.

[0183] A "device" refers to a computer, smartphone, tablet, or other device used by a user.

[0184] "Audio output" is the process of making audio data playable to the user through a playback device such as a speaker.

[0185] A "vehicle system" refers to a system that provides various functions, including the operation of autonomous vehicles and infotainment systems.

[0186] "Real-time" refers to a state where processing and output occur almost instantly, with virtually no delay.

[0187] This invention is a system that translates user voice data into other languages ​​in real time, facilitating smooth multilingual communication within autonomous vehicles.

[0188] The server acquires the user's voice data and converts it into text data. Specifically, it collects voice data spoken by the user through a smartphone or in-car microphone. The collected voice data is sent to the server and converted into text data by a speech recognition engine on the server. A common speech recognition API (e.g., Google Speech-to-Text API) can be used as this speech recognition engine.

[0189] Next, the server detects the language of the converted text data and translates it into the target language. This translation process can utilize an external translation API (e.g., Google Translate API). The translated text data is then input into a speech synthesis engine (e.g., Google Text-to-Speech API) and converted into speech data in the target language.

[0190] The translated audio data generated on the server is sent to the user's device. The device then plays the received translated audio data through speakers or headphones. This system enables users to communicate in different languages ​​in real time.

[0191] As a concrete example, consider a scenario where passenger A, a Japanese speaker, is in an autonomous vehicle and asks passenger B, an English speaker, "Where is this car going?" Passenger A's Japanese voice is captured by a smartphone or the car's microphone and sent to a server. The server converts the voice data into text data, which is then translated into English using a translation engine. This text is then converted back into English voice data and output to passenger B as "Where is this car going?". Similarly, passenger B's English response is also translated into Japanese and conveyed to passenger A.

[0192] An example of a prompt given so far is, "Please translate 'Hello, nice to meet you' into English and output it as speech."

[0193] This system enables smooth multilingual communication within autonomous vehicles.

[0194] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0195] Step 1:

[0196] The user uses their smartphone or the in-car microphone to input voice data. The voice data is acquired digitally and temporarily stored on the device. This is how the voice data is collected.

[0197] Step 2:

[0198] The device sends the acquired audio data to the server. The audio data is converted to an appropriate format (e.g., WAV file format) and uploaded to the server. This transfers the audio data to the server.

[0199] Step 3:

[0200] The server inputs the received audio data into a speech recognition engine and converts the audio data into text data. Specifically, it uses an external speech recognition service such as the Google Speech-to-Text API to analyze the audio waveform and generate the corresponding text data. The input is audio data, and the output is text data.

[0201] Step 4:

[0202] The server uses an external language detection API to determine the language of the generated text data. For example, it can determine whether the text data is in Japanese or English. The input is the text data, and the output is language information. Based on the language information, the processing of the text data proceeds to the next step.

[0203] Step 5:

[0204] The server uses an external translation engine, such as the Google Translate API, to translate the detected text data into the target language. This is the process of converting text data into another language. The input is text data and language information, and the output is translated text data.

[0205] Step 6:

[0206] The server inputs the translated text data into a speech synthesis engine and converts it into speech data. External speech synthesis services, such as the Google Text-to-Speech API, are used to convert the text data into speech output. The input is translated text data, and the output is speech data.

[0207] Step 7:

[0208] The server converts the generated audio data into an appropriate format (e.g., MP3 file format) and sends it to the user's device. This transfers the translated audio data to the device.

[0209] Step 8:

[0210] The terminal plays the received translated audio data through a playback device (speaker or headphones) and provides it to the user. The input is the audio data received from the server, and the output is the played audio. Specifically, this includes a process of decoding and playing the audio data.

[0211] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0212] This invention provides a system for real-time and web-based meetings using PCs or smartphones that translates multilingual speech in real time and also recognizes and reflects the user's emotions in the translation results. This system can translate user voice input into other languages ​​in real time and output it as speech data that reflects emotions. The program processing of this system is described below in natural language.

[0213] 1. Acquisition of voice input

[0214] The user uses the microphone on their PC or smartphone to perform voice input. For example, User A says, "Hello, nice to meet you."

[0215] 2. Sending audio data to the server

[0216] The terminal temporarily stores this audio data and sends it to the server. At this time, the audio data is converted to a specific format (for example, a WAV file).

[0217] 3. Perform speech recognition

[0218] The server inputs the received audio data into a speech recognition engine, which then converts the audio into text data. An external speech recognition API (such as a common cloud-based service) is used as the speech recognition engine. In this example, the text data "Hello, nice to meet you" is generated.

[0219] 4. Performing emotion recognition

[0220] The server inputs the received voice data into the emotion engine to identify the user's emotion. The emotion engine identifies the emotion from the voice data and outputs that emotion (e.g., joy, sadness, excitement, etc.). Here, the result of emotion recognition is identified as joy.

[0221] 5. Detection of Language

[0222] The server uses a language detection algorithm to automatically detect the language of the text data. This algorithm also utilizes an external language detection API. For example, it is detected that this text data is in Japanese.

[0223] 6. Execution of Translation

[0224] The server inputs the detected Japanese text into a translation engine to translate it into the target language. As the translation engine, an external translation API is utilized. Here, the text "こんにちは、はじめまして" is translated into Chinese as "你好,很高兴认识你".

[0225] 7. Reflection of Emotion in the Translation Result

[0226] The server reflects the identified emotion (in this case, "joy") in the translated text. For this purpose, emotion information is added to the text output by the translation engine. Specifically, the intonation, strength, speed, etc. of the voice are adjusted based on the emotion information.

[0227] 8. Speech Synthesis of the Translated Text

[0228] The server inputs the translated text data containing emotion information into a speech synthesis engine to convert it into voice data. As the speech synthesis engine, an external speech synthesis API is utilized. Here, the translated text "你好,很高兴认识你" is converted into Chinese voice data reflecting the emotion of joy.

[0229] 9. Transmission of Voice Data to the Terminal

[0230] The server converts the generated voice data into a certain format (e.g., MP3 file) and prepares to send it to the user's terminal. This voice data is sent to the terminal.

[0231] 10. Output of Translated Voice

[0232] The terminal receives the voice data sent from the server. The terminal plays this voice data through a playback device (speaker or headphones) and provides the translated voice to the user. In this way, the voice of User A, "Konnichiwa, hajimemashite," is transmitted to User B as the Chinese voice "Nihao, hen gaoxing renshi ni" that reflects the emotion.

[0233] As a specific example, when User A (a Japanese speaker) communicates with Chinese speaker User B in a real meeting, by using this system, the speech of User A is quickly translated into Chinese and transmitted to User B in a form that reflects the emotion. In this way, smooth and emotional communication among multiple languages is realized.

[0234] As described above, the system of the present invention enables smooth and emotional communication among multiple languages by translating the user's voice into multiple languages in real time, further recognizing the emotion and outputting it as voice again. This system features flexibility and high-precision processing by using external APIs and meets the needs of large-scale global communication.

[0235] The processing flow will be described below.

[0236] Step 1:

[0237] The user inputs voice towards the microphone of a PC or smartphone. For example, say "Konnichiwa, hajimemashite."

[0238] Step 2:

[0239] The device temporarily stores the audio signal acquired from the microphone as digital audio data.

[0240] Step 3:

[0241] The device converts the recorded audio data into a specific format (for example, a WAV file).

[0242] Step 4:

[0243] The device sends voice data to the server over the network.

[0244] Step 5:

[0245] The server receives the audio data.

[0246] Step 6:

[0247] The server inputs the received audio data into a speech recognition engine, which then converts the audio data into text data. An external speech recognition API (such as a common cloud-based service) is used as the speech recognition engine.

[0248] Step 7:

[0249] The server retrieves the text data returned by the speech recognition engine (for example, "Hello, nice to meet you").

[0250] Step 8:

[0251] The server inputs the received audio data into the emotion engine to identify the user's emotion. The emotion engine identifies the emotion (e.g., joy, sadness, excitement) from the audio data and outputs that emotion. In this case, the emotion recognition result is identified as "joy."

[0252] Step 9:

[0253] The server inputs the text data into a language detection algorithm to detect the language of the text data. The language detection algorithm also utilizes an external language detection API.

[0254] Step 10:

[0255] The server detects that the language of the text data is Japanese.

[0256] Step 11:

[0257] The server inputs the Japanese text data into a translation engine to translate it into a target language (e.g., Chinese). As the translation engine, an external translation API is utilized.

[0258] Step 12:

[0259] The server obtains the translated text data (e.g., "Hello, nice to meet you") returned from the translation engine.

[0260] Step 13:

[0261] Based on the identified emotion (in this case, "joy"), the server adds emotion information to the translated text.

[0262] Step 14:

[0263] The server inputs the translated text data containing emotion information into a text-to-speech engine to convert it into audio data. As the text-to-speech engine, an external text-to-speech API is utilized. Here, the translated text "Hello, nice to meet you" is converted into Chinese audio data reflecting the emotion of joy.

[0264] Step 15:

[0265] The server converts the generated audio data into a certain format (e.g., MP3 file).

[0266] Step 16:

[0267] The server transmits voice data to the terminal.

[0268] Step 17:

[0269] The terminal receives the voice data sent from the server.

[0270] Step 18:

[0271] The terminal plays the received voice data through a playback device (such as a speaker or headphones) and provides the translated voice to the user.

[0272] Through this specific processing step, the voice of user A, "Kon'nichiwa, Hajimemashite" is translated in real time with high precision into the Chinese voice "Nihao, Hen gaoxing renshi ni", and is transmitted to user B in a form that reflects the emotion.

[0273] (Example 2)

[0274] Next, Example 2 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart device 14 is referred to as the "terminal".

[0275] In modern society, the importance of technologies for real-time communication among multiple languages is increasing. However, conventional voice translation systems simply translate languages and cannot accurately reflect the emotions of users. Also, it has been difficult to perform real-time translation of voice and recognition of emotions simultaneously. For this reason, in communication in multiple languages, there has been a problem that smooth conversations including emotions cannot be realized.

[0276] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following respective means.

[0277] In this invention, the server includes means for acquiring voice data from a user, means for transmitting the acquired voice data to the server, means for the server to convert the voice data into text data, means for the server to detect the language of the text data, means for the server to translate the text data into a target language, means for the server to reflect emotion information in the translated text data, means for the server to convert the translated text data with emotion information reflected into voice data, means for transmitting the translated voice data to the user's terminal, and means for the terminal to play back the translated voice data. As a result, real-time voice translation reflecting emotions among multiple languages becomes possible.

[0278] The "user" refers to a person who inputs voice data using the system.

[0279] The "voice data" refers to a digital voice signal acquired through a device such as a microphone.

[0280] The "terminal" is a device used by the user to input voice data or play back received voice data, including a PC, smartphone, etc.

[0281] The "server" refers to a remote computer system that processes voice data transmitted from a terminal and performs translation and emotion recognition.

[0282] The "text data" refers to character data converted from voice data by a voice recognition engine.

[0283] "Language detection" refers to the process of identifying the language of the acquired text data.

[0284] The "target language" refers to the target language after translation.

[0285] The "emotion information" refers to information about the user's emotions identified from voice data, including emotions such as joy, sadness, excitement, etc.

[0286] "Translated text data" refers to character data obtained by translating text data into the target language.

[0287] "Voice synthesis" refers to the process of generating voice data from text data.

[0288] "Voice recognition engine" refers to software or an API that converts voice data into text data.

[0289] "Voice synthesis engine" refers to software or an API that converts text data into voice data.

[0290] "Emotion recognition engine" refers to software or an API that identifies the user's emotion from voice data.

[0291] The present invention relates to a system for enabling smooth and emotional communication among multiple languages in real meetings and web meetings. This system has a function that allows a user to input voice, translate it into other languages in real time, and output it while reflecting the user's emotion.

[0292] The main components of the system are the user's terminal, server, and external API, and they function as follows.

[0293] 1. Acquisition of voice input

[0294] The user uses the microphone of a terminal such as a PC or smartphone to input voice. The terminal acquires this voice data as a digital signal and temporarily stores it. For example, user A says, "Hello, nice to meet you."

[0295] 2. Transmission of voice data to the server

[0296] The terminal converts the acquired voice data into a file in WAV format and sends it to the server. The server receives the data as an HTTP request and further stores the voice data internally for processing.

[0297] 3. Execution of Speech Recognition

[0298] The server sends the received voice data to a cloud-based speech recognition API (e.g., a common speech recognition engine) to convert the voice into text data. At this time, the speech recognition engine analyzes the voice signal and generates the corresponding text "Hello, nice to meet you for the first time".

[0299] 4. Execution of Emotion Recognition

[0300] The server further sends the voice data to an emotion recognition engine (e.g., a common emotion recognition engine) to identify the user's emotion. The emotion recognition engine analyzes the tone and pitch of the voice, etc., and identifies the emotion (happiness, sadness, excitement, etc.). Here, the emotion of "happiness" is identified.

[0301] 5. Detection of Language

[0302] The server sends the text data obtained from speech recognition to a common language detection API to identify the language being used. This API detects that the text data is in Japanese.

[0303] 6. Execution of Translation

[0304] The server sends the detected Japanese text to a cloud-based translation API (e.g., a common translation engine) to translate it into the target language (e.g., Chinese). For this reason, the text "Hello, nice to meet you for the first time" is translated into "你好,很高兴认识你".

[0305] 7. Reflection of Emotion

[0306] The server reflects the identified emotion (in this case, "joy") in the translated text. Specifically, it adds emotion information to reflect the intonation, strength, and speed of the voice. This enables the generation of voice that reflects the user's emotion.

[0307] 8. Execution of Speech Synthesis

[0308] The server sends the translated text data containing emotion information to a speech synthesis engine (e.g., a common speech synthesis engine) and converts it into voice data that reflects the emotion information. The speech synthesis engine generates the translated text "Hello, nice to meet you" in Chinese pronunciation as voice data that reflects the emotion of "joy".

[0309] 9. Transmission of Voice Data to the Terminal

[0310] The server encodes the generated voice data into the MP3 format and sends it to the user's terminal. The server encrypts the data using HTTPS and transfers it securely.

[0311] 10. Output of Translated Voice

[0312] The terminal receives the voice data sent from the server. The terminal outputs this voice data through a playback device (speaker or headphones) and provides the translated voice that reflects the emotion to User B. Thus, User A's voice "Konnichiwa, hajimemashite" is conveyed to User B as the Chinese voice "你好,很高兴认识你" with the emotion of joy.

[0313] Specific Example

[0314] For example, assume that User A (a Japanese speaker) is talking to User B, a Chinese speaker, in a real meeting. By using this system, User A's speech is quickly translated into Chinese and conveyed to User B in a form that reflects the emotion. This enables smooth and emotional communication among multiple languages.

[0315] Example of a prompt

[0316] "User A, who speaks Japanese, is talking to User B, who speaks Chinese. When User A says 'Hello, nice to meet you,' translate that into Chinese and output it as audio, reflecting the emotion (joy) of User A."

[0317] As described above, the system of the present invention enables smooth communication among users by providing real-time and emotion-reflecting translation in multilingual communication.

[0318] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0319] Step 1:

[0320] The user speaks into the microphone of their PC or smartphone. The device's microphone then captures the user's voice as a digital signal. The input is the user's voice, and the output is digital audio data.

[0321] Step 2:

[0322] The device temporarily stores the acquired audio data in its internal memory and converts the data into a WAV file. The input is digital audio data, and the output is a WAV audio file. The device then sends this audio data file to the server as an HTTP request.

[0323] Step 3:

[0324] The server sends the received audio data to a cloud-based speech recognition API. The speech recognition API analyzes the audio signal and converts it into corresponding text data. The input is an audio file in WAV format, and the output is the text data "Hello, nice to meet you".

[0325] Step 4:

[0326] The server sends the voice data to the emotion recognition engine to identify the user's emotion. The emotion recognition engine analyzes the tone and pitch of the voice, etc., and recognizes the emotion (such as joy, sadness, excitement, etc.). The input is voice data, and the output is the emotion information "joy".

[0327] Step 5:

[0328] The server sends the text data obtained by voice recognition to the language detection API to identify the language being used. The API analyzes the language characteristics of the text and identifies the corresponding language. The input is text data, and the output is the language information "Japanese".

[0329] Step 6:

[0330] The server sends the detected Japanese text data to the translation API to translate it into the target language. The translation engine translates "こんにちは、はじめまして" into "你好,很高兴认识你". The input is Japanese text data, and the output is Chinese translated text data.

[0331] Step 7:

[0332] The server reflects the identified emotion "joy" in the translated text. Specifically, it adds emotion information for adjusting the intonation, strength, and speed of the voice. The input is Chinese translated text data and emotion information, and the output is translated text data reflecting the emotion.

[0333] Step 8:

[0334] The server sends the translated text data containing emotion information to the text-to-speech engine to convert it into voice data. The text-to-speech engine generates the translated text "你好,很高兴认识你" in Chinese pronunciation and as voice data reflecting the emotion of "joy". The input is translated text data reflecting the emotion, and the output is voice data.

[0335] Step 9:

[0336] The server encodes the generated audio data into the MP3 format and prepares to transmit it to the user's terminal. This audio data is transmitted to the terminal as an HTTP response. The input is the generated audio data, and the output is the audio data in the MP3 format.

[0337] Step 10:

[0338] The terminal receives the audio data sent from the server. The terminal plays this audio data through a playback device (such as a speaker or headphones) and provides the translated audio to the user. The input is the audio data received from the server, and the output is the played audio. As a result, the voice of user A, "Hello, nice to meet you for the first time," is transmitted to user B as the Chinese voice "你好,很高兴认识你" that reflects the emotion.

[0339] (Application Example 2)

[0340] Next, Application Example 2 will be described. In the following description, the data processing device 12 is referred to as the "server," and the smart device 14 is referred to as the "terminal."

[0341] With the increase in multinational workers in the factory, it is necessary to convey work instructions quickly and accurately. However, there is a language barrier, which may cause misunderstandings and confusion. Also, it is important to accurately convey the urgency and emotion of the instructor, but this is also a difficult issue. Therefore, in addition to real-time translation between multiple languages, a method for conveying work instructions considering emotions is required.

[0342] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following respective means.

[0343] In this invention, the server includes means for acquiring voice data from a user, means for transmitting the acquired voice data to the server, and means for the server to input the received voice data into an emotion engine, identify emotions, and reflect the emotional information in the translated voice data. This makes it possible to accurately translate the voice instructions of a work supervisor into multiple languages ​​and to convey them to a factory robot while reflecting the supervisor's emotions.

[0344] A "user" refers to a person who uses the system to input voice data.

[0345] "Voice data" refers to data that records the user's voice in digital format.

[0346] A "server" refers to a computer system that processes and converts audio data.

[0347] "Text data" refers to the character information that has been converted from audio data by a speech recognition engine.

[0348] "Means for detecting language" refers to algorithms or programs that identify the language of text data.

[0349] The "target language" refers to the language that is the subject of the translation.

[0350] An "emotion engine" refers to software or an algorithm that identifies emotions from audio data and outputs emotional information.

[0351] "Emotional information" refers to data that digitally represents emotions (e.g., joy, sadness, urgency, etc.) identified from the user's voice.

[0352] "Multilingual" refers to multiple different languages.

[0353] "Factory robots" refer to automated equipment or devices that perform tasks within a factory.

[0354] "Work instructions" refer to specific instructions that should be given to workers and equipment within a factory.

[0355] A "translation API" refers to an external software interface that automatically translates text data into another language.

[0356] A "speech recognition API" refers to an external software interface that converts speech data into text data.

[0357] A "speech synthesis engine" refers to software or an algorithm that converts text data into speech data.

[0358] A "prompt" refers to a text-based instruction given to a generative AI model.

[0359] This invention relates to a system for transmitting multilingual and emotionally reflective voice instructions to factory robots within a factory. This system translates the voice of the person giving the instructions into multiple languages ​​in real time, recognizes the person's emotions, and reflects them in the translation result.

[0360] Hardware and software

[0361] This system uses the following hardware and software:

[0362] 1. Hardware

[0363] Smart glasses: A device for inputting voice.

[0364] Server: A central device used for data processing and calculations.

[0365] Robot control terminal: A device that plays voice data and transmits instructions to the robot.

[0366] 2. Software

[0367] Speech-to-Text APIs: To convert speech data into text data, common cloud-based speech recognition APIs are used. An example is the Google Cloud Speech-to-Text API.

[0368] Emotion Recognition API: To identify emotions from audio data, an external emotion recognition API is used. For example, IBM Watson® Tone Analyzer is one such API.

[0369] Language detection API: To detect the language of text data, use an external language detection API. For example, use the Microsoft® Azure® Language Recognition API.

[0370] Translation API: Use an external translation API to translate text data into the target language. For example, the Google Cloud Translation API supports this.

[0371] Speech synthesis APIs: Speech synthesis APIs are used to convert translated text data into speech data that reflects emotion. Amazon Polly is an example of this.

[0372] Data processing and calculation

[0373] 1. Acquisition and transmission of audio data

[0374] The user uses the microphone on the smart glasses to input voice data. The acquired voice data is temporarily stored on the smart glasses' terminal and then sent to the server. At this time, the voice data is saved in WAV format.

[0375] 2. Speech Recognition Processing

[0376] The server uses the Google Cloud Speech-to-Text API to convert the received audio data into text data. The converted text data is then processed further.

[0377] 3. Emotion recognition processing

[0378] The server uses IBM Watson Tone Analyzer to identify the user's emotions from the voice data. The identified emotion information is then added to the text data.

[0379] 4. Language detection and translation

[0380] The Microsoft Azure Language Recognition API is used to detect the language, and the Google Cloud Translation API is used to translate it into the target language.

[0381] 5. Reflecting emotions in the translation results

[0382] The translated text data incorporates sentiment information identified by IBM Watson Tone Analyzer. Based on this sentiment information, the intonation and speed of the speech are adjusted.

[0383] 6. Speech synthesis and terminal transmission

[0384] Amazon Polly is used to convert the audio data into emotionally reflective audio, which is then sent to the robot control terminal. Finally, the terminal plays this audio data.

[0385] Specific example

[0386] For example, if a factory supervisor gives the instruction to "hurry up and start assembling the next product batch," this system will perform the following actions:

[0387] 1. The audio input is acquired in WAV format and sent to the server.

[0388] 2. Use the Google Cloud Speech-to-Text API to generate the text data "Hurry up and start assembling the next product batch."

[0389] 3. Use IBM Watson Tone Analyzer to identify "urgency" and emotion.

[0390] 4. Detect the language of the text data as Japanese using the Microsoft Azure Language Recognition API.

[0391] 5. Use the Google Cloud Translation API to translate the text data into English as "Hurry up and start assembling the next product batch".

[0392] 6. The "urgent" emotion identified by IBM Watson Tone Analyzer is reflected in the voice, and then converted into voice data using Amazon Polly.

[0393] 7. Finally, the translated audio data is sent to the robot control terminal and played back.

[0394] Example of a prompt

[0395] The user said via voice input, "Hurry up and start assembling the next product batch." Translate this voice data into English in real time and convey the instruction to the robot, reflecting an urgent emotion. The translated voice should be "Hurry up and start assembling the next product batch," and the voice should be synthesized to reflect an urgent emotion.

[0396] This system will enable factory supervisors to efficiently communicate multilingual and emotionally resonant voice instructions to factory robots.

[0397] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0398] Step 1:

[0399] The user uses the microphone on their smart glasses to input voice commands. For example, the user might say, "Hurry up and start assembling the next product batch." This voice data is the input. The voice data is temporarily stored in WAV format on the smart glasses' terminal. This terminal then sends the data to the server.

[0400] Step 2:

[0401] The server receives audio data from the smart glasses and inputs it into the Google Cloud Speech-to-Text API, converting the audio data into text data. The specific operations performed here are audio signal processing and string parsing. The converted text data is then output.

[0402] Step 3:

[0403] The server inputs text data into the IBM Watson Tone Analyzer for sentiment recognition. The input is text data, and the output is sentiment information (e.g., "urgent"). The specific operation is text-based sentiment analysis.

[0404] Step 4:

[0405] The server inputs text data into the Microsoft Azure Language Recognition API to detect the language. The input is text data, and the output is language information (e.g., Japanese). The specific operation is language profiling.

[0406] Step 5:

[0407] The server inputs text data into the Google Cloud Translation API and translates it into the target language (e.g., English). The output is the translated text data (e.g., "Hurry up and start assembling the next product batch"). The specific operation is a text translation process.

[0408] Step 6:

[0409] The server inputs translated text data and sentiment information into Amazon Polly, which then synthesizes it into speech data. The input is translated text data and sentiment information, and the output is speech data that reflects the sentiment. The specific operation is speech synthesis.

[0410] Step 7:

[0411] The server sends the generated voice data to the robot control terminal. The input is voice data that reflects emotions, and the server performs the specific action of transferring the data to the terminal.

[0412] Step 8:

[0413] The terminal receives audio data sent from the server and plays this audio data. The output is a translated voice instruction that reflects emotion. The specific operation involves decoding the audio data and playing it back through the speaker.

[0414] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0415] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0416] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0417] [Second Embodiment]

[0418] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0419] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0420] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0421] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0422] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0423] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0424] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0425] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0426] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0427] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0428] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0429] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0430] This invention relates to a system that translates multilingual speech in real time using a PC or smartphone. This system can translate user voice input into another language in real time and output it back as speech. The program processing of this system is described below in natural language.

[0431] 1. Acquisition of voice input

[0432] The user uses the microphone on their PC or smartphone to perform voice input. For example, User A says, "Hello, nice to meet you."

[0433] 2. Server Transmission of Audio Data

[0434] The terminal temporarily stores this audio data and transmits it to the server. At this time, the audio data is converted into a certain format (for example, WAV file).

[0435] 3. Execution of Speech Recognition

[0436] The server inputs the received audio data to the speech recognition engine and converts the audio into text data. As the speech recognition engine, it is possible to use an external speech recognition API (for example, a general cloud-based service). Here, text data such as "Hello, nice to meet you" is generated.

[0437] 4. Detection of Language

[0438] The server uses a language detection algorithm to automatically detect the language of this text data. This algorithm can also use an external language detection API. For example, it is detected that this text data is in Japanese.

[0439] 5. Execution of Translation

[0440] The server inputs the detected Japanese text to a translation engine to translate it into the target language. As the translation engine, an external translation API can be used. Here, the text "Hello, nice to meet you" is translated into Chinese as "你好,很高兴认识你".

[0441] 6. Speech Synthesis of Translated Text

[0442] The server inputs the translated Chinese text to a speech synthesis engine to convert it into audio data. As the speech synthesis engine, an external speech synthesis API can be used. Here, the translated text "你好,很高兴认识你" is converted into Chinese audio data.

[0443] 7. Terminal Transmission of Voice Data

[0444] The server converts the generated voice data into a certain format (e.g., MP3 file) and prepares to send it to the user's terminal. This voice data is sent to the terminal.

[0445] 8. Output of Translated Voice

[0446] The terminal receives the voice data sent from the server. The terminal plays this voice data through a playback device (speaker or headphones) and provides the translated voice to the user. In this way, the voice of user A, "Konnichiwa, hajimemashite," is transmitted to user B as the Chinese voice "Nihao, hen gaoxing renshi ni."

[0447] As a specific example, when user A (a Japanese speaker) communicates with user B (a Chinese speaker) in a real meeting, by using this system, the speech of user A is quickly translated into Chinese and transmitted to user B. In this way, smooth communication between multiple languages is realized.

[0448] As described above, the system of the present invention enables smooth communication between multiple languages by translating the user's voice into multiple languages in real time and outputting it as voice again. This system features flexibility and high-precision processing by using external APIs and meets the needs of large-scale global communication.

[0449] The processing flow will be described below.

[0450] Step 1:

[0451] The user inputs voice towards the microphone of a PC or smartphone. For example, say "Konnichiwa, hajimemashite."

[0452] Step 2:

[0453] The device temporarily stores the audio signal acquired from the microphone as digital audio data.

[0454] Step 3:

[0455] The device converts the recorded audio data into a specific format (for example, a WAV file).

[0456] Step 4:

[0457] The device sends voice data to the server over the network.

[0458] Step 5:

[0459] The server receives the audio data.

[0460] Step 6:

[0461] The server inputs the received audio data into a speech recognition engine, which then converts the audio data into text data. An external speech recognition API (such as a common cloud-based service) is used as the speech recognition engine.

[0462] Step 7:

[0463] The server retrieves the text data returned by the speech recognition engine (for example, "Hello, nice to meet you").

[0464] Step 8:

[0465] The server inputs text data into a language detection algorithm to detect the language of the text data. The language detection algorithm also utilizes an external language detection API.

[0466] Step 9:

[0467] The server detects that the language of the text data is Japanese.

[0468] Step 10:

[0469] The server inputs Japanese text data into a translation engine in order to translate it into a target language (e.g., Chinese). As the translation engine, an external translation API is used.

[0470] Step 11:

[0471] The server obtains the translated text data (e.g., "Hello, nice to meet you") returned from the translation engine.

[0472] Step 12:

[0473] The server inputs the translated text data into a speech synthesis engine and converts it into audio data. As the speech synthesis engine, an external speech synthesis API is used.

[0474] Step 13:

[0475] The server obtains the audio data (e.g., Chinese audio data) returned from the speech synthesis engine.

[0476] Step 14:

[0477] The server converts the generated audio data into a certain format (e.g., MP3 file).

[0478] Step 15:

[0479] The server sends the audio data to the terminal.

[0480] Step 16:

[0481] The terminal receives the audio data sent from the server.

[0482] Step 17:

[0483] The device plays the received audio data through a playback device (speaker or headphones) and provides the user with translated audio.

[0484] Through the steps described above, the user's Japanese voice input is translated into Chinese in real time with high accuracy and transmitted as audio to other users.

[0485] (Example 1)

[0486] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0487] In today's global society, facilitating smooth communication between people who speak different languages ​​is crucial. However, real-time translation and playback of speech across multiple languages ​​is technically complex, and building a user-friendly system is a particularly challenging task. To overcome this challenge, efficient and highly accurate speech recognition, translation, and speech synthesis are necessary, along with a flexible and reliable system to achieve these capabilities.

[0488] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0489] In this invention, the server includes means for acquiring audio data from a user, means for transmitting the acquired audio data to the server, means for the server to convert the audio data into text data, means for the server to detect the language of the text data, means for the server to translate the text data into a target language, means for the server to convert the translated text data into audio data, means for transmitting the translated audio data to the user's terminal, means for the terminal to play the translated audio data, means for the server to convert the format of the audio data using an external audio processing API, and means for the terminal to output the received audio data to a predetermined playback device. This enables real-time audio translation and playback between users using different languages.

[0490] "Audio data" refers to data that records and stores the voice spoken by a user in digital format.

[0491] A "server" is a computer system that provides various services over a network, and it plays a role in processing and converting audio data.

[0492] "Text data" refers to data obtained by converting audio data into written text.

[0493] "Language detection" is the process of identifying the language of text data.

[0494] The "target language" is the language that will be output as a result of the translation.

[0495] "Speech recognition" is a technology that analyzes speech data and converts it into corresponding text data.

[0496] "Translation" is the process of converting text data written in one language into another language.

[0497] "Speech synthesis" is a technology that generates speech data based on text data.

[0498] An "external speech processing API" is an interface that provides external services and functions for performing speech recognition and speech synthesis on speech data.

[0499] A "device" refers to a device used by a user, such as a PC or smartphone.

[0500] A "playback device" refers to a device such as a speaker or headphones used to output audio data as sound.

[0501] This invention is a system that enables users to translate multilingual voices in real time using a PC or smartphone. The following is a natural language explanation of the processing of the program of this system.

[0502] To implement this system, the following hardware and software are utilized:

[0503] Hardware: PC, smartphone, microphone, speaker, headphones

[0504] Software: Google Cloud Speech-to-Text API, Google Cloud Translation API, Google Cloud Text-to-Speech API

[0505] First, the user uses the microphone of a PC or smartphone to perform voice input. For example, user A utters the words "Hello, nice to meet you." The terminal temporarily stores this voice data, converts it into the WAV format, and sends it to the server.

[0506] The server inputs the received voice data into the Google Cloud Speech-to-Text API and converts it into text data "Hello, nice to meet you." Next, the server automatically detects the language of this text data using the Google Cloud Translation API and determines that it is Japanese.

[0507] Next, the server translates the Japanese text into Chinese using the Google Cloud Translation API. For example, the text "Hello, nice to meet you" is converted into the Chinese "你好,很高兴认识你". Then, the server inputs this translated text into the Google Cloud Text-to-Speech API and generates Chinese voice data of "你好,很高兴认识你".

[0508] The server converts the generated voice data into the MP3 format and transmits it to the terminal. The terminal outputs the received voice data to a playback device (such as a speaker or headphones) to convey it to User B. In this way, User A's voice is transmitted to User B as Chinese voice.

[0509] As a specific example, when User A communicates with Chinese-speaking User B in a real meeting, by using this system, User A's speech can be quickly translated into Chinese and transmitted to User B. In this way, smooth communication among multiple languages is realized.

[0510] Examples of prompt sentences input to the generation AI model:

[0511] When implementing a program for performing speech recognition, language detection, translation, and speech synthesis, please follow the following steps:

[0512] 1. The user spoke in Japanese, "Konnichiwa, hajimemashite". The terminal temporarily stores this voice data, converts it into the WAV file format, and transmits it to the server.

[0513] 2. The server receives the WAV file and uses the Google Cloud Speech-to-Text API to convert it into the text "Konnichiwa, hajimemashite".

[0514] 3. The server automatically detects the language of this text data using the Google Cloud Translation API and determines that it is Japanese.

[0515] 4. The server translates the text data into Chinese using the Google Cloud Translation API.

[0516] 5. The translated text "Hello, nice to meet you" is converted into voice data using the Google Cloud Text-to-Speech API.

[0517] 6. Convert the generated audio data to MP3 format and send it to your device.

[0518] 7. The terminal outputs the received audio data through the playback device and provides it to user B.

[0519] In summary, the system of the present invention enables smooth multilingual communication by translating the user's voice into multiple languages ​​in real time and outputting it back as voice. This system is characterized by its flexibility and high-precision processing through the use of external APIs, and meets the needs of large-scale global communication.

[0520] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0521] Step 1:

[0522] The user uses the microphone on their PC or smartphone to perform voice input. The subject is the user. The specific actions in this step are as follows:

[0523] Input: The user says, "Hello, nice to meet you."

[0524] Output: Input to the terminal as audio data.

[0525] Operation: The user picks up their smartphone and speaks into the microphone, or speaks into the microphone on their PC. The voice is captured by the microphone and temporarily stored as digital audio data.

[0526] Step 2:

[0527] The terminal temporarily stores the audio data, converts it to WAV format, and sends it to the server. The subject is the terminal. The specific actions in this step are as follows:

[0528] Input: Temporarily stored audio data.

[0529] Output: WAV format audio data sent to the server.

[0530] Operation: The device converts temporarily stored audio data into a WAV file format and sends it to the server.

[0531] Step 3:

[0532] The server inputs the received audio data into the Google Cloud Speech-to-Text API, converting the audio data into text data. The subject is the server. The specific actions in this step are as follows:

[0533] Input: WAV format audio data received from the device.

[0534] Output: Text data "Hello, nice to meet you."

[0535] Operation: The server sends WAV formatted audio data to the Google Cloud Speech-to-Text API and retrieves the returned text data.

[0536] Step 4:

[0537] The server automatically detects the language of the text data. The subject is the server. The specific actions in this step are as follows:

[0538] Input: Text data "Hello, nice to meet you".

[0539] Output: Determination that the language is Japanese.

[0540] Operation: The server inputs text data into the Google Cloud Translation API and receives the language detection results.

[0541] Step 5:

[0542] The server translates Japanese text using the Google Cloud Translation API. The subject is the server. The specific operations in this step are as follows.

[0543] Input: Japanese text data "こんにちは、はじめまして".

[0544] Output: Chinese text data "你好,很高兴认识你".

[0545] Operation: The server sends the Japanese text to the Google Cloud Translation API and receives the translated Chinese text.

[0546] Step 6:

[0547] The server converts the translated Chinese text into audio data using the Google Cloud Text-to-Speech API. The subject is the server. The specific operations in this step are as follows.

[0548] Input: Chinese text data "你好,很高兴认识你".

[0549] Output: Chinese audio data.

[0550] Operation: The server sends the Chinese text to the Google Cloud Text-to-Speech API and obtains the returned audio data.

[0551] Step 7:

[0552] The server converts the generated audio data into the MP3 format and sends it to the user's terminal. The subject is the server. The specific operations in this step are as follows.

[0553] Input: The translated audio data.

[0554] Output: The audio data in MP3 format sent to the user terminal.

[0555] Operation: The server converts the audio data to MP3 format and prepares it for transmission to the terminal. The converted audio data is then sent to the user's terminal.

[0556] Step 8:

[0557] The terminal receives audio data sent from the server and plays it back through the playback device. The subject is the terminal. The specific actions in this step are as follows:

[0558] Input: MP3 audio data received from the server.

[0559] Output: The translated audio "Hello, I'm happy to have met you" is played.

[0560] Operation: The device receives audio data in MP3 format and outputs it through its built-in playback device (speaker or headphones). Communication is established when User B listens to the audio.

[0561] (Application Example 1)

[0562] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0563] In autonomous vehicles, it is difficult to facilitate rapid and accurate communication between passengers who speak multiple languages, as well as between passengers and the system. The present invention aims to provide a system that translates speech in real time and enables smooth communication between passengers who speak different languages.

[0564] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0565] In this invention, the server includes means for acquiring voice data from a user, means for transmitting the acquired voice data to the server, means for the server to convert the voice data into text data, means for the server to detect the language of the text data, means for the server to translate the text data into a target language, means for the server to convert the translated text data back into voice data, means for transmitting the translated voice data to the user's terminal, means for the terminal to play the translated voice data, and means integrated into the vehicle system to translate passenger voices into other languages ​​in real time and output them as voice. This enables smooth multilingual communication within an autonomous vehicle.

[0566] A "user" is an individual or group that uses the system.

[0567] "Audio data" refers to information recorded in digital format from the voice spoken by a user.

[0568] A "server" is a computer system that provides functions such as processing, converting, and transmitting audio data.

[0569] "Text data" refers to data in a format where audio data has been converted into written text.

[0570] "Language detection" is the process of determining which language text data is written in.

[0571] The "target language" is the target language to which the translated text will be converted.

[0572] Translation is the process of converting text written in one language into another language.

[0573] "Speech synthesis" is a technology that converts text data into speech data.

[0574] A "device" refers to a computer, smartphone, tablet, or other device used by a user.

[0575] "Audio output" is the process of making audio data playable to the user through a playback device such as a speaker.

[0576] A "vehicle system" refers to a system that provides various functions, including the operation of autonomous vehicles and infotainment systems.

[0577] "Real-time" refers to a state where processing and output occur almost instantly, with virtually no delay.

[0578] This invention is a system that translates user voice data into other languages ​​in real time, facilitating smooth multilingual communication within autonomous vehicles.

[0579] The server acquires the user's voice data and converts it into text data. Specifically, it collects voice data spoken by the user through a smartphone or in-car microphone. The collected voice data is sent to the server and converted into text data by a speech recognition engine on the server. A common speech recognition API (e.g., Google Speech-to-Text API) can be used as this speech recognition engine.

[0580] Next, the server detects the language of the converted text data and translates it into the target language. This translation process can utilize an external translation API (e.g., Google Translate API). The translated text data is then input into a speech synthesis engine (e.g., Google Text-to-Speech API) and converted into speech data in the target language.

[0581] The translated audio data generated on the server is sent to the user's device. The device then plays the received translated audio data through speakers or headphones. This system enables users to communicate in different languages ​​in real time.

[0582] As a concrete example, consider a scenario where passenger A, a Japanese speaker, is in an autonomous vehicle and asks passenger B, an English speaker, "Where is this car going?" Passenger A's Japanese voice is captured by a smartphone or the car's microphone and sent to a server. The server converts the voice data into text data, which is then translated into English using a translation engine. This text is then converted back into English voice data and output to passenger B as "Where is this car going?". Similarly, passenger B's English response is also translated into Japanese and conveyed to passenger A.

[0583] An example of a prompt given so far is, "Please translate 'Hello, nice to meet you' into English and output it as speech."

[0584] This system enables smooth multilingual communication within autonomous vehicles.

[0585] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0586] Step 1:

[0587] The user uses their smartphone or the in-car microphone to input voice data. The voice data is acquired digitally and temporarily stored on the device. This is how the voice data is collected.

[0588] Step 2:

[0589] The device sends the acquired audio data to the server. The audio data is converted to an appropriate format (e.g., WAV file format) and uploaded to the server. This transfers the audio data to the server.

[0590] Step 3:

[0591] The server inputs the received audio data into a speech recognition engine and converts the audio data into text data. Specifically, it uses an external speech recognition service such as the Google Speech-to-Text API to analyze the audio waveform and generate the corresponding text data. The input is audio data, and the output is text data.

[0592] Step 4:

[0593] The server uses an external language detection API to determine the language of the generated text data. For example, it can determine whether the text data is in Japanese or English. The input is the text data, and the output is language information. Based on the language information, the processing of the text data proceeds to the next step.

[0594] Step 5:

[0595] The server uses an external translation engine, such as the Google Translate API, to translate the detected text data into the target language. This is the process of converting text data into another language. The input is text data and language information, and the output is translated text data.

[0596] Step 6:

[0597] The server inputs the translated text data into a speech synthesis engine and converts it into speech data. External speech synthesis services, such as the Google Text-to-Speech API, are used to convert the text data into speech output. The input is translated text data, and the output is speech data.

[0598] Step 7:

[0599] The server converts the generated audio data into an appropriate format (e.g., MP3 file format) and sends it to the user's device. This transfers the translated audio data to the device.

[0600] Step 8:

[0601] The terminal plays the received translated audio data through a playback device (speaker or headphones) and provides it to the user. The input is the audio data received from the server, and the output is the played audio. Specifically, this includes a process of decoding and playing the audio data.

[0602] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0603] This invention provides a system for real-time and web-based meetings using PCs or smartphones that translates multilingual speech in real time and also recognizes and reflects the user's emotions in the translation results. This system can translate user voice input into other languages ​​in real time and output it as speech data that reflects emotions. The program processing of this system is described below in natural language.

[0604] 1. Acquisition of voice input

[0605] The user uses the microphone on their PC or smartphone to perform voice input. For example, User A says, "Hello, nice to meet you."

[0606] 2. Sending audio data to the server

[0607] The terminal temporarily stores this audio data and sends it to the server. At this time, the audio data is converted to a specific format (for example, a WAV file).

[0608] 3. Execution of Speech Recognition

[0609] The server inputs the received voice data to the speech recognition engine and converts the voice into text data. As the speech recognition engine, an external speech recognition API (for example, a general cloud-based service) is used. Here, text data such as "Hello, nice to meet you" is generated.

[0610] 4. Execution of Emotion Recognition

[0611] The server inputs the received voice data to the emotion engine to identify the user's emotion. The emotion engine identifies the emotion from the voice data and outputs the emotion (for example, joy, sadness, excitement, etc.). Here, the result of emotion recognition is identified as joy.

[0612] 5. Detection of Language

[0613] The server uses a language detection algorithm to automatically detect the language of the text data. This algorithm also uses an external language detection API. For example, it is detected that this text data is in Japanese.

[0614] 6. Execution of Translation

[0615] The server inputs the detected Japanese text to a translation engine to translate it into the target language. As the translation engine, an external translation API is used. Here, the text "Hello, nice to meet you" is translated into Chinese as "你好,很高兴认识你".

[0616] 7. Reflection of Emotion in the Translation Result

[0617] The server reflects the identified emotion (in this case, "joy") in the translated text. For this reason, emotion information is added to the text output by the translation engine. Specifically, the intonation, strength, speed, etc. of the voice are adjusted based on the emotion information.

[0618] 8. Speech synthesis of translated text

[0619] The server inputs the translated text data containing emotion information into a speech synthesis engine and converts it into voice data. As the speech synthesis engine, an external speech synthesis API is used. Here, the translated text "你好,很高兴认识你" is converted into Chinese voice data reflecting the emotion of joy.

[0620] 9. Transmission of voice data to the terminal

[0621] The server converts the generated voice data into a certain format (for example, an MP3 file) and prepares to send it to the user's terminal. This voice data is sent to the terminal.

[0622] 10. Output of translated voice

[0623] The terminal receives the voice data sent from the server. The terminal plays this voice data through a playback device (such as a speaker or headphones) and provides the translated voice to the user. In this way, the voice of user A, "こんにちは、はじめまして", is transmitted to user B as the Chinese voice "你好,很高兴认识你" reflecting the emotion.

[0624] As a specific example, when user A (a Japanese speaker) communicates with user B (a Chinese speaker) in a real meeting, by using this system, the speech of user A is quickly translated into Chinese and transmitted to user B in a form reflecting the emotion. In this way, smooth and emotional communication among multiple languages is realized.

[0625] In summary, the system of the present invention enables smooth and emotionally charged multilingual communication by translating user speech into multiple languages ​​in real time, recognizing emotions, and outputting them back as speech. This system features flexibility and high-precision processing through the use of external APIs, meeting the needs of large-scale global communication.

[0626] The following describes the processing flow.

[0627] Step 1:

[0628] The user speaks into the microphone of their PC or smartphone. For example, they might say, "Hello, nice to meet you."

[0629] Step 2:

[0630] The device temporarily stores the audio signal acquired from the microphone as digital audio data.

[0631] Step 3:

[0632] The device converts the recorded audio data into a specific format (for example, a WAV file).

[0633] Step 4:

[0634] The device sends voice data to the server over the network.

[0635] Step 5:

[0636] The server receives the audio data.

[0637] Step 6:

[0638] The server inputs the received audio data into a speech recognition engine, which then converts the audio data into text data. An external speech recognition API (such as a common cloud-based service) is used as the speech recognition engine.

[0639] Step 7:

[0640] The server obtains the text data (e.g., "Hello, nice to meet you") returned from the speech recognition engine.

[0641] Step 8:

[0642] The server inputs the received voice data into the emotion engine to identify the user's emotion. The emotion engine identifies the emotion (e.g., joy, sadness, excitement, etc.) from the voice data and outputs the emotion. Here, the result of emotion recognition is identified as "joy".

[0643] Step 9:

[0644] The server inputs the text data into the language detection algorithm to detect the language of the text data. The language detection algorithm also uses an external language detection API.

[0645] Step 10:

[0646] The server detects that the language of the text data is Japanese.

[0647] Step 11:

[0648] The server inputs the Japanese text data into a translation engine to translate it into a target language (e.g., Chinese). As the translation engine, an external translation API is used.

[0649] Step 12:

[0650] The server obtains the translated text data (e.g., "Hello, nice to meet you") returned from the translation engine.

[0651] Step 13:

[0652] Based on the emotion identified by the server (in this case, "joy"), add emotion information to the translated text.

[0653] Step 14:

[0654] The server inputs the translation text data containing emotion information into the speech synthesis engine and converts it into audio data. As the speech synthesis engine, an external speech synthesis API is used. Here, the translated text "Hello, nice to meet you" is converted into Chinese audio data reflecting the emotion of joy.

[0655] Step 15:

[0656] The server converts the generated audio data into a certain format (for example, an MP3 file).

[0657] Step 16:

[0658] The server sends the audio data to the terminal.

[0659] Step 17:

[0660] The terminal receives the audio data sent from the server.

[0661] Step 18:

[0662] The terminal plays the received audio data through a playback device (such as a speaker or headphones) and provides the translated audio to the user.

[0663] Through this specific processing step, the voice "Konichiwa, Hajimemashite" of user A is translated into the Chinese voice "Hello, nice to meet you" with high precision in real time and transmitted to user B in a form reflecting the emotion.

[0664] (Example 2)

[0665] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0666] In modern society, the importance of technologies for real-time communication across multiple languages ​​is increasing. However, conventional voice translation systems simply translate languages ​​and fail to accurately reflect the user's emotions. Furthermore, it has been difficult to simultaneously perform real-time voice translation and emotion recognition. As a result, there has been a problem in achieving smooth conversations that include emotions in multilingual communication.

[0667] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0668] In this invention, the server includes means for acquiring voice data from a user, means for transmitting the acquired voice data to the server, means for the server to convert the voice data into text data, means for the server to detect the language of the text data, means for the server to translate the text data into a target language, means for the server to reflect emotional information in the translated text data, means for the server to convert the translated text data with reflected emotional information into voice data, means for transmitting the translated voice data to the user's terminal, and means for the terminal to play the translated voice data. This enables real-time voice translation that reflects emotions across multiple languages.

[0669] A "user" refers to a person who uses the system to input voice data.

[0670] "Audio data" refers to digital audio signals acquired through devices such as microphones.

[0671] A "terminal" is a device used by a user to input or play back received audio data, and includes PCs, smartphones, and other similar devices.

[0672] A "server" refers to a remote computer system that processes audio data transmitted from a terminal and performs tasks such as translation and emotion recognition.

[0673] "Text data" refers to character data converted from speech data by a speech recognition engine.

[0674] "Language detection" refers to the process of identifying the language of acquired text data.

[0675] "Target language" refers to the target language after translation.

[0676] "Emotional information" refers to information about the user's emotions identified from voice data, including emotions such as joy, sadness, and excitement.

[0677] "Translated text data" refers to character data that has been translated into the target language.

[0678] "Speech synthesis" refers to the process of generating speech data from text data.

[0679] A "speech recognition engine" refers to software or an API that converts speech data into text data.

[0680] A "speech synthesis engine" refers to software or an API that converts text data into speech data.

[0681] An "emotion recognition engine" refers to software or an API that identifies a user's emotions from voice data.

[0682] This invention relates to a system for enabling smooth and emotionally charged communication between multiple languages ​​in real-world and web meetings. The system has the function of taking voice input from a user, translating it into another language in real time, and outputting it while reflecting the user's emotions.

[0683] The main components of the system are the user's terminal, the server, and the external API, and it functions as follows:

[0684] 1. Acquisition of voice input

[0685] The user inputs voice using the microphone on a device such as a PC or smartphone. The device acquires this voice data as a digital signal and temporarily stores it. For example, user A says, "Hello, nice to meet you."

[0686] 2. Sending audio data to the server

[0687] The terminal converts the acquired audio data into a WAV file and sends it to the server. The server receives the data as an HTTP request and stores the audio data internally for further processing.

[0688] 3. Perform speech recognition

[0689] The server sends the received audio data to a cloud-based speech recognition API (for example, a common speech recognition engine) to convert the audio into text data. In this process, the speech recognition engine analyzes the audio signal and generates the corresponding text, "Hello, nice to meet you."

[0690] 4. Performing emotion recognition

[0691] The server then sends the audio data to an emotion recognition engine (for example, a general emotion recognition engine) to identify the user's emotion. The emotion recognition engine analyzes the tone and pitch of the voice to identify the emotion (joy, sadness, excitement, etc.). In this case, the emotion "joy" is identified.

[0692] 5. Language detection

[0693] The server sends the text data obtained by voice recognition to a general language detection API to identify the language being used. This API detects that the text data is in Japanese.

[0694] 6. Execution of Translation

[0695] The server sends the detected Japanese text to a cloud-based translation API (e.g., a general translation engine) to translate it into the target language (e.g., Chinese). For this reason, the text "こんにちは、はじめまして" is translated as "你好,很高兴认识你".

[0696] 7. Reflection of Emotion

[0697] The server reflects the identified emotion (in this case, "joy") in the translated text. Specifically, emotion information for reflecting the intonation, strength, and speed of the voice is added. This enables the generation of voice that reflects the user's emotion. <​​​​​​​​​​​​​​​​​​​​​​The terminal receives the voice data sent from the server. The terminal outputs this voice data through a playback device (such as a speaker or headphones) to provide the translated voice reflecting the emotion to User B. As a result, the voice of User A, "Hello, nice to meet you for the first time," is conveyed to User B as a Chinese voice with a happy emotion, "你好,很高兴认识你".

[0704] Specific example

[0705] For example, assume that User A (a Japanese speaker) is talking to User B (a Chinese speaker) in a real meeting. By using this system, the speech of User A is quickly translated into Chinese and transmitted to User B in a form reflecting the emotion. As a result, smooth and emotional communication among multiple languages is realized.

[0706] Example of prompt sentence

[0707] "User A who speaks Japanese is talking to User B who speaks Chinese. When User A says 'Hello, nice to meet you for the first time,' please translate the voice into Chinese and output it as a voice reflecting the emotion (happiness). "

[0708] As described above, the system of the present invention can smoothly perform user communication by providing real-time and emotion-reflecting translation in communication among multiple languages.

[0709] The flow of the specific process in the second embodiment will be described with reference to FIG. 13.

[0710] Step 1:

[0711] The user performs voice input toward the microphone of a PC or a smartphone. As a result, the microphone of the terminal captures the voice of the user as a digital signal. The input is the voice of the user, and the output is voice data in digital format.

[0712] Step 2:

[0713] The terminal temporarily stores the acquired voice data in the internal memory and converts the data into a file in WAV format. The input is digital voice data, and the output is a voice file in WAV format. The terminal sends this voice data file to the server as an HTTP request.

[0714] Step 3:

[0715] The server sends the received voice data to a cloud-based voice recognition API. The voice recognition API analyzes the voice signal and converts it into corresponding text data. The input is a voice file in WAV format, and the output is text data "Hello, nice to meet you".

[0716] Step 4:

[0717] The server sends the voice data to an emotion recognition engine to identify the user's emotion. The emotion recognition engine analyzes the tone and pitch of the voice, etc., and recognizes the emotion (happiness, sadness, excitement, etc.). The input is voice data, and the output is emotion information "happiness".

[0718] Step 5:

[0719] The server sends the text data obtained by voice recognition to a language detection API to identify the language being used. The API analyzes the language characteristics of the text and identifies the corresponding language. The input is text data, and the output is language information "Japanese".

[0720] Step 6:

[0721] The server sends the detected Japanese text data to a translation API to translate it into the target language. The translation engine translates "Hello, nice to meet you" into "你好,很高兴认识你". The input is Japanese text data, and the output is Chinese translated text data.

[0722] Step 7:

[0723] The server reflects the identified emotion "joy" in the translated text. Specifically, emotion information for adjusting the intonation, strength, and speed of the voice is added. The input is Chinese translation text data and emotion information, and the output is translated text data that reflects emotion.

[0724] Step 8:

[0725] The server sends the translated text data containing emotion information to the text-to-speech engine and converts it into voice data. The text-to-speech engine generates the translated text "Hello, nice to meet you" in Chinese pronunciation and as voice data that reflects the emotion of "joy". The input is translated text data that reflects emotion, and the output is voice data.

[0726] Step 9:

[0727] The server encodes the generated voice data into the MP3 format and prepares to send it to the user's terminal. This voice data is sent to the terminal as an HTTP response. The input is the generated voice data, and the output is voice data in the MP3 format.

[0728] Step 10:

[0729] The terminal receives the voice data sent from the server. The terminal plays this voice data through a playback device (speaker or headphones) and provides the translated voice to the user. The input is the voice data received from the server, and the output is the played voice. In this way, the voice "Kon'nichiwa, Hajimemashite" of User A is transmitted to User B as the Chinese voice "Hello, nice to meet you" that reflects emotion.

[0730] (Application Example 2)

[0731] Next, Application Example 2 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0732] As the number of multinational workers in factories increases, it becomes necessary to communicate work instructions quickly and accurately. However, language barriers exist, which can lead to misunderstandings and confusion. Accurately conveying the urgency and emotions of the person giving the instructions is also crucial, but this too presents a challenge. Therefore, in addition to real-time translation between multiple languages, there is a need for methods to communicate work instructions while taking emotions into consideration.

[0733] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0734] In this invention, the server includes means for acquiring voice data from a user, means for transmitting the acquired voice data to the server, and means for the server to input the received voice data into an emotion engine, identify emotions, and reflect the emotional information in the translated voice data. This makes it possible to accurately translate the voice instructions of a work supervisor into multiple languages ​​and to convey them to a factory robot while reflecting the supervisor's emotions.

[0735] A "user" refers to a person who uses the system to input voice data.

[0736] "Voice data" refers to data that records the user's voice in digital format.

[0737] A "server" refers to a computer system that processes and converts audio data.

[0738] "Text data" refers to the character information that has been converted from audio data by a speech recognition engine.

[0739] "Means for detecting language" refers to algorithms or programs that identify the language of text data.

[0740] The "target language" refers to the language that is the subject of the translation.

[0741] An "emotion engine" refers to software or an algorithm that identifies emotions from audio data and outputs emotional information.

[0742] "Emotional information" refers to data that digitally represents emotions (e.g., joy, sadness, urgency, etc.) identified from the user's voice.

[0743] "Multilingual" refers to multiple different languages.

[0744] "Factory robots" refer to automated equipment or devices that perform tasks within a factory.

[0745] "Work instructions" refer to specific instructions that should be given to workers and equipment within a factory.

[0746] A "translation API" refers to an external software interface that automatically translates text data into another language.

[0747] A "speech recognition API" refers to an external software interface that converts speech data into text data.

[0748] A "speech synthesis engine" refers to software or an algorithm that converts text data into speech data.

[0749] A "prompt" refers to a text-based instruction given to a generative AI model.

[0750] This invention relates to a system for transmitting multilingual and emotionally reflective voice instructions to factory robots within a factory. This system translates the voice of the person giving the instructions into multiple languages ​​in real time, recognizes the person's emotions, and reflects them in the translation result.

[0751] Hardware and software

[0752] This system uses the following hardware and software:

[0753] 1. Hardware

[0754] Smart glasses: A device for inputting voice.

[0755] Server: A central device used for data processing and calculations.

[0756] Robot control terminal: A device that plays voice data and transmits instructions to the robot.

[0757] 2. Software

[0758] Speech-to-Text APIs: To convert speech data into text data, common cloud-based speech recognition APIs are used. An example is the Google Cloud Speech-to-Text API.

[0759] Emotion Recognition API: To identify emotions from audio data, an external emotion recognition API is used. IBM Watson Tone Analyzer is an example of this.

[0760] Language detection API: To detect the language of text data, use an external language detection API. For example, use the Microsoft Azure Language Recognition API.

[0761] Translation API: Use an external translation API to translate text data into the target language. For example, the Google Cloud Translation API supports this.

[0762] Speech synthesis APIs: Speech synthesis APIs are used to convert translated text data into speech data that reflects emotion. Amazon Polly is an example of this.

[0763] Data processing and calculation

[0764] 1. Acquisition and transmission of audio data

[0765] The user uses the microphone on the smart glasses to input voice data. The acquired voice data is temporarily stored on the smart glasses' terminal and then sent to the server. At this time, the voice data is saved in WAV format.

[0766] 2. Speech Recognition Processing

[0767] The server uses the Google Cloud Speech-to-Text API to convert the received audio data into text data. The converted text data is then processed further.

[0768] 3. Emotion recognition processing

[0769] The server uses IBM Watson Tone Analyzer to identify the user's emotions from the voice data. The identified emotion information is then added to the text data.

[0770] 4. Language detection and translation

[0771] The Microsoft Azure Language Recognition API is used to detect the language, and the Google Cloud Translation API is used to translate it into the target language.

[0772] 5. Reflecting emotions in the translation results

[0773] The translated text data incorporates sentiment information identified by IBM Watson Tone Analyzer. Based on this sentiment information, the intonation and speed of the speech are adjusted.

[0774] 6. Speech synthesis and terminal transmission

[0775] Amazon Polly is used to convert the audio data into emotionally reflective audio, which is then sent to the robot control terminal. Finally, the terminal plays this audio data.

[0776] Specific example

[0777] For example, if a factory supervisor gives the instruction to "hurry up and start assembling the next product batch," this system will perform the following actions:

[0778] 1. The audio input is acquired in WAV format and sent to the server.

[0779] 2. Use the Google Cloud Speech-to-Text API to generate the text data "Hurry up and start assembling the next product batch."

[0780] 3. Use IBM Watson Tone Analyzer to identify "urgency" and emotion.

[0781] 4. Detect the language of the text data as Japanese using the Microsoft Azure Language Recognition API.

[0782] 5. Use the Google Cloud Translation API to translate the text data into English as "Hurry up and start assembling the next product batch".

[0783] 6. The "urgent" emotion identified by IBM Watson Tone Analyzer is reflected in the voice, and then converted into voice data using Amazon Polly.

[0784] 7. Finally, the translated audio data is sent to the robot control terminal and played back.

[0785] Example of a prompt

[0786] The user said via voice input, "Hurry up and start assembling the next product batch." Translate this voice data into English in real time and convey the instruction to the robot, reflecting an urgent emotion. The translated voice should be "Hurry up and start assembling the next product batch," and the voice should be synthesized to reflect an urgent emotion.

[0787] This system will enable factory supervisors to efficiently communicate multilingual and emotionally resonant voice instructions to factory robots.

[0788] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0789] Step 1:

[0790] The user uses the microphone on their smart glasses to input voice commands. For example, the user might say, "Hurry up and start assembling the next product batch." This voice data is the input. The voice data is temporarily stored in WAV format on the smart glasses' terminal. This terminal then sends the data to the server.

[0791] Step 2:

[0792] The server receives audio data from the smart glasses and inputs it into the Google Cloud Speech-to-Text API, converting the audio data into text data. The specific operations performed here are audio signal processing and string parsing. The converted text data is then output.

[0793] Step 3:

[0794] The server inputs text data into the IBM Watson Tone Analyzer for sentiment recognition. The input is text data, and the output is sentiment information (e.g., "urgent"). The specific operation is text-based sentiment analysis.

[0795] Step 4:

[0796] The server inputs text data into the Microsoft Azure Language Recognition API to detect the language. The input is text data, and the output is language information (e.g., Japanese). The specific operation is language profiling.

[0797] Step 5:

[0798] The server inputs text data into the Google Cloud Translation API and translates it into the target language (e.g., English). The output is the translated text data (e.g., "Hurry up and start assembling the next product batch"). The specific operation is a text translation process.

[0799] Step 6:

[0800] The server inputs translated text data and sentiment information into Amazon Polly, which then synthesizes it into speech data. The input is translated text data and sentiment information, and the output is speech data that reflects the sentiment. The specific operation is speech synthesis.

[0801] Step 7:

[0802] The server sends the generated voice data to the robot control terminal. The input is voice data that reflects emotions, and the server performs the specific action of transferring the data to the terminal.

[0803] Step 8:

[0804] The terminal receives audio data sent from the server and plays this audio data. The output is a translated voice instruction that reflects emotion. The specific operation involves decoding the audio data and playing it back through the speaker.

[0805] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0806] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0807] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0808] [Third Embodiment]

[0809] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0810] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0811] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0812] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0813] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0814] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0815] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0816] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0817] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0818] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0819] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0820] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0821] This invention relates to a system that translates multilingual speech in real time using a PC or smartphone. This system can translate user voice input into another language in real time and output it back as speech. The program processing of this system is described below in natural language.

[0822] 1. Acquisition of voice input

[0823] The user uses the microphone on their PC or smartphone to perform voice input. For example, User A says, "Hello, nice to meet you."

[0824] 2. Server Transmission of Audio Data

[0825] The terminal temporarily stores this audio data and transmits it to the server. At this time, the audio data is converted into a certain format (for example, WAV file).

[0826] 3. Execution of Speech Recognition

[0827] The server inputs the received audio data to the speech recognition engine and converts the audio into text data. As the speech recognition engine, it is possible to use an external speech recognition API (for example, a general cloud-based service). Here, text data such as "Hello, nice to meet you" is generated.

[0828] 4. Detection of Language

[0829] The server uses a language detection algorithm to automatically detect the language of this text data. This algorithm can also use an external language detection API. For example, it is detected that this text data is in Japanese.

[0830] 5. Execution of Translation

[0831] The server inputs the detected Japanese text to a translation engine to translate it into the target language. As the translation engine, an external translation API can be used. Here, the text "Hello, nice to meet you" is translated into Chinese as "你好,很高兴认识你".

[0832] 6. Speech Synthesis of Translated Text

[0833] The server inputs the translated Chinese text to a speech synthesis engine to convert it into audio data. As the speech synthesis engine, an external speech synthesis API can be used. Here, the translated text "你好,很高兴认识你" is converted into Chinese audio data.

[0834] 7. Terminal Transmission of Voice Data

[0835] The server converts the generated voice data into a certain format (for example, MP3 file) and prepares to send it to the user's terminal. This voice data is sent to the terminal.

[0836] 8. Output of Translated Voice

[0837] The terminal receives the voice data sent from the server. The terminal plays this voice data through a playback device (speaker or headphones) and provides the translated voice to the user. In this way, the voice of user A, "Kon'nichiwa, hajimemashite," is transmitted to user B as the Chinese voice "Nihao, hen gaoxing renshi ni."

[0838] As a specific example, when user A (a Japanese speaker) communicates with Chinese-speaking user B in a real meeting, by using this system, the speech of user A is quickly translated into Chinese and transmitted to user B. In this way, smooth communication between multiple languages is realized.

[0839] As described above, the system of the present invention enables smooth communication between multiple languages by translating the user's voice into multiple languages in real time and outputting it as voice again. This system features flexibility and high-precision processing by using external APIs, and meets the needs of large-scale global communication.

[0840] The following describes the processing flow.

[0841] Step 1:

[0842] The user inputs voice towards the microphone of the PC or smartphone. For example, say "Kon'nichiwa, hajimemashite."

[0843] Step 2:

[0844] The device temporarily stores the audio signal acquired from the microphone as digital audio data.

[0845] Step 3:

[0846] The device converts the recorded audio data into a specific format (for example, a WAV file).

[0847] Step 4:

[0848] The device sends voice data to the server over the network.

[0849] Step 5:

[0850] The server receives the audio data.

[0851] Step 6:

[0852] The server inputs the received audio data into a speech recognition engine, which then converts the audio data into text data. An external speech recognition API (such as a common cloud-based service) is used as the speech recognition engine.

[0853] Step 7:

[0854] The server retrieves the text data returned by the speech recognition engine (for example, "Hello, nice to meet you").

[0855] Step 8:

[0856] The server inputs text data into a language detection algorithm to detect the language of the text data. The language detection algorithm also utilizes an external language detection API.

[0857] Step 9:

[0858] The server detects that the language of the text data is Japanese.

[0859] Step 10:

[0860] The server inputs Japanese text data into a translation engine for translation into a target language (e.g., Chinese). As the translation engine, an external translation API is used.

[0861] Step 11:

[0862] The server obtains the translated text data (e.g., "Hello, nice to meet you") returned from the translation engine.

[0863] Step 12:

[0864] The server inputs the translated text data into a speech synthesis engine and converts it into audio data. As the speech synthesis engine, an external speech synthesis API is used.

[0865] Step 13:

[0866] The server obtains the audio data (e.g., Chinese audio data) returned from the speech synthesis engine.

[0867] Step 14:

[0868] The server converts the generated audio data into a certain format (e.g., MP3 file).

[0869] Step 15:

[0870] The server transmits the audio data to the terminal.

[0871] Step 16:

[0872] The terminal receives the audio data sent from the server.

[0873] Step 17:

[0874] The device plays the received audio data through a playback device (speaker or headphones) and provides the user with translated audio.

[0875] Through the steps described above, the user's Japanese voice input is translated into Chinese in real time with high accuracy and transmitted as audio to other users.

[0876] (Example 1)

[0877] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0878] In today's global society, facilitating smooth communication between people who speak different languages ​​is crucial. However, real-time translation and playback of speech across multiple languages ​​is technically complex, and building a user-friendly system is a particularly challenging task. To overcome this challenge, efficient and highly accurate speech recognition, translation, and speech synthesis are necessary, along with a flexible and reliable system to achieve these capabilities.

[0879] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0880] In this invention, the server includes means for acquiring audio data from a user, means for transmitting the acquired audio data to the server, means for the server to convert the audio data into text data, means for the server to detect the language of the text data, means for the server to translate the text data into a target language, means for the server to convert the translated text data into audio data, means for transmitting the translated audio data to the user's terminal, means for the terminal to play the translated audio data, means for the server to convert the format of the audio data using an external audio processing API, and means for the terminal to output the received audio data to a predetermined playback device. This enables real-time audio translation and playback between users using different languages.

[0881] "Audio data" refers to data that records and stores the voice spoken by a user in digital format.

[0882] A "server" is a computer system that provides various services over a network, and it plays a role in processing and converting audio data.

[0883] "Text data" refers to data obtained by converting audio data into written text.

[0884] "Language detection" is the process of identifying the language of text data.

[0885] The "target language" is the language that will be output as a result of the translation.

[0886] "Speech recognition" is a technology that analyzes speech data and converts it into corresponding text data.

[0887] "Translation" is the process of converting text data written in one language into another language.

[0888] "Speech synthesis" is a technology that generates speech data based on text data.

[0889] An "external speech processing API" is an interface that provides external services and functions for performing speech recognition and speech synthesis on speech data.

[0890] A "device" refers to a device used by a user, such as a PC or smartphone.

[0891] A "playback device" refers to a device such as a speaker or headphones used to output audio data as sound.

[0892] This invention is a system that enables users to translate multilingual voices in real-time using a PC or smartphone. The following is a natural language explanation of the processing of the program of this system.

[0893] To implement this system, the following hardware and software are utilized:

[0894] Hardware: PC, smartphone, microphone, speaker, headphones

[0895] Software: Google Cloud Speech-to-Text API, Google Cloud Translation API, Google Cloud Text-to-Speech API

[0896] First, the user uses the microphone of a PC or smartphone to perform voice input. For example, user A utters the words "Hello, nice to meet you." The terminal temporarily stores this voice data, converts it to the WAV format, and sends it to the server.

[0897] The server inputs the received voice data into the Google Cloud Speech-to-Text API and converts it into text data "Hello, nice to meet you." Next, the server automatically detects the language of this text data using the Google Cloud Translation API and determines that it is Japanese.

[0898] Next, the server translates the Japanese text into Chinese using the Google Cloud Translation API. For example, the text "Hello, nice to meet you" is converted into the Chinese "你好,很高兴认识你". Then, the server inputs this translated text into the Google Cloud Text-to-Speech API to generate Chinese voice data of "你好,很高兴认识你".

[0899] The server converts the generated voice data into the MP3 format and transmits it to the terminal. The terminal outputs the received voice data to a playback device (such as a speaker or headphones) to convey it to User B. In this way, the voice of User A is conveyed to User B as Chinese voice.

[0900] As a specific example, when User A communicates with Chinese-speaking User B in a real meeting, by using this system, the speech of User A is quickly translated into Chinese and transmitted to User B. Thereby, smooth communication among multiple languages is realized.

[0901] Examples of prompt sentences input to the generation AI model:

[0902] When implementing a program for performing speech recognition, language detection, translation, and speech synthesis, please follow the following steps:

[0903] 1. The user spoke in Japanese as "Konnichiwa, hajimemashite". The terminal temporarily stores this voice data, converts it into the WAV file format, and transmits it to the server.

[0904] 2. The server receives the WAV file and converts it into the text "Konnichiwa, hajimemashite" using the Google Cloud Speech-to-Text API.

[0905] 3. The server automatically detects the language of this text data using the Google Cloud Translation API and determines that it is Japanese.

[0906] 4. The server translates the text data into Chinese using the Google Cloud Translation API.

[0907] 5. The translated text "Hello, nice to meet you" is converted into voice data using the Google Cloud Text-to-Speech API.

[0908] 6. Convert the generated audio data to MP3 format and send it to your device.

[0909] 7. The terminal outputs the received audio data through the playback device and provides it to user B.

[0910] In summary, the system of the present invention enables smooth multilingual communication by translating the user's voice into multiple languages ​​in real time and outputting it back as voice. This system is characterized by its flexibility and high-precision processing through the use of external APIs, and meets the needs of large-scale global communication.

[0911] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0912] Step 1:

[0913] The user uses the microphone on their PC or smartphone to perform voice input. The subject is the user. The specific actions in this step are as follows:

[0914] Input: The user says, "Hello, nice to meet you."

[0915] Output: Input to the terminal as audio data.

[0916] Operation: The user picks up their smartphone and speaks into the microphone, or speaks into the microphone on their PC. The voice is captured by the microphone and temporarily stored as digital audio data.

[0917] Step 2:

[0918] The terminal temporarily stores the audio data, converts it to WAV format, and sends it to the server. The subject is the terminal. The specific actions in this step are as follows:

[0919] Input: Temporarily stored audio data.

[0920] Output: WAV format audio data sent to the server.

[0921] Operation: The device converts temporarily stored audio data into a WAV file format and sends it to the server.

[0922] Step 3:

[0923] The server inputs the received audio data into the Google Cloud Speech-to-Text API, converting the audio data into text data. The subject is the server. The specific actions in this step are as follows:

[0924] Input: WAV format audio data received from the device.

[0925] Output: Text data "Hello, nice to meet you."

[0926] Operation: The server sends WAV formatted audio data to the Google Cloud Speech-to-Text API and retrieves the returned text data.

[0927] Step 4:

[0928] The server automatically detects the language of the text data. The subject is the server. The specific actions in this step are as follows:

[0929] Input: Text data "Hello, nice to meet you".

[0930] Output: Determination that the language is Japanese.

[0931] Operation: The server inputs text data into the Google Cloud Translation API and receives the language detection results.

[0932] Step 5:

[0933] The server translates Japanese text using the Google Cloud Translation API. The subject is the server. The specific operations in this step are as follows.

[0934] Input: Japanese text data "こんにちは、はじめまして".

[0935] Output: Chinese text data "你好,很高兴认识你".

[0936] Operation: The server sends the Japanese text to the Google Cloud Translation API and receives the translated Chinese text.

[0937] Step 6:

[0938] The server converts the translated Chinese text into audio data using the Google Cloud Text-to-Speech API. The subject is the server. The specific operations in this step are as follows.

[0939] Input: Chinese text data "你好,很高兴认识你".

[0940] Output: Chinese audio data.

[0941] Operation: The server sends the Chinese text to the Google Cloud Text-to-Speech API and obtains the returned audio data.

[0942] Step 7:

[0943] The server converts the generated audio data into the MP3 format and sends it to the user's terminal. The subject is the server. The specific operations in this step are as follows.

[0944] Input: The translated audio data.

[0945] Output: The MP3-formatted audio data sent to the user's terminal.

[0946] Operation: The server converts the audio data to MP3 format and prepares it for transmission to the terminal. The converted audio data is then sent to the user's terminal.

[0947] Step 8:

[0948] The terminal receives audio data sent from the server and plays it back through the playback device. The subject is the terminal. The specific actions in this step are as follows:

[0949] Input: MP3 audio data received from the server.

[0950] Output: The translated audio "Hello, I'm happy to have met you" is played.

[0951] Operation: The device receives audio data in MP3 format and outputs it through its built-in playback device (speaker or headphones). Communication is established when User B listens to the audio.

[0952] (Application Example 1)

[0953] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0954] In autonomous vehicles, it is difficult to facilitate rapid and accurate communication between passengers who speak multiple languages, as well as between passengers and the system. The present invention aims to provide a system that translates speech in real time and enables smooth communication between passengers who speak different languages.

[0955] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0956] In this invention, the server includes means for acquiring voice data from a user, means for transmitting the acquired voice data to the server, means for the server to convert the voice data into text data, means for the server to detect the language of the text data, means for the server to translate the text data into a target language, means for the server to convert the translated text data back into voice data, means for transmitting the translated voice data to the user's terminal, means for the terminal to play the translated voice data, and means integrated into the vehicle system to translate passenger voices into other languages ​​in real time and output them as voice. This enables smooth multilingual communication within an autonomous vehicle.

[0957] A "user" is an individual or group that uses the system.

[0958] "Audio data" refers to information recorded in digital format from the voice spoken by a user.

[0959] A "server" is a computer system that provides functions such as processing, converting, and transmitting audio data.

[0960] "Text data" refers to data in a format where audio data has been converted into written text.

[0961] "Language detection" is the process of determining which language text data is written in.

[0962] The "target language" is the target language to which the translated text will be converted.

[0963] Translation is the process of converting text written in one language into another language.

[0964] "Speech synthesis" is a technology that converts text data into speech data.

[0965] A "device" refers to a computer, smartphone, tablet, or other device used by a user.

[0966] "Audio output" is the process of making audio data playable to the user through a playback device such as a speaker.

[0967] A "vehicle system" refers to a system that provides various functions, including the operation of autonomous vehicles and infotainment systems.

[0968] "Real-time" refers to a state where processing and output occur almost instantly, with virtually no delay.

[0969] This invention is a system that translates user voice data into other languages ​​in real time, facilitating smooth multilingual communication within autonomous vehicles.

[0970] The server acquires the user's voice data and converts it into text data. Specifically, it collects voice data spoken by the user through a smartphone or in-car microphone. The collected voice data is sent to the server and converted into text data by a speech recognition engine on the server. A common speech recognition API (e.g., Google Speech-to-Text API) can be used as this speech recognition engine.

[0971] Next, the server detects the language of the converted text data and translates it into the target language. This translation process can utilize an external translation API (e.g., Google Translate API). The translated text data is then input into a speech synthesis engine (e.g., Google Text-to-Speech API) and converted into speech data in the target language.

[0972] The translated audio data generated on the server is sent to the user's device. The device then plays the received translated audio data through speakers or headphones. This system enables users to communicate in different languages ​​in real time.

[0973] As a concrete example, consider a scenario where passenger A, a Japanese speaker, is in an autonomous vehicle and asks passenger B, an English speaker, "Where is this car going?" Passenger A's Japanese voice is captured by a smartphone or the car's microphone and sent to a server. The server converts the voice data into text data, which is then translated into English using a translation engine. This text is then converted back into English voice data and output to passenger B as "Where is this car going?". Similarly, passenger B's English response is also translated into Japanese and conveyed to passenger A.

[0974] An example of a prompt given so far is, "Please translate 'Hello, nice to meet you' into English and output it as speech."

[0975] This system enables smooth multilingual communication within autonomous vehicles.

[0976] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0977] Step 1:

[0978] The user uses their smartphone or the in-car microphone to input voice data. The voice data is acquired digitally and temporarily stored on the device. This is how the voice data is collected.

[0979] Step 2:

[0980] The device sends the acquired audio data to the server. The audio data is converted to an appropriate format (e.g., WAV file format) and uploaded to the server. This transfers the audio data to the server.

[0981] Step 3:

[0982] The server inputs the received audio data into a speech recognition engine and converts the audio data into text data. Specifically, it uses an external speech recognition service such as the Google Speech-to-Text API to analyze the audio waveform and generate the corresponding text data. The input is audio data, and the output is text data.

[0983] Step 4:

[0984] The server uses an external language detection API to determine the language of the generated text data. For example, it can determine whether the text data is in Japanese or English. The input is the text data, and the output is language information. Based on the language information, the processing of the text data proceeds to the next step.

[0985] Step 5:

[0986] The server uses an external translation engine, such as the Google Translate API, to translate the detected text data into the target language. This is the process of converting text data into another language. The input is text data and language information, and the output is translated text data.

[0987] Step 6:

[0988] The server inputs the translated text data into a speech synthesis engine and converts it into speech data. External speech synthesis services, such as the Google Text-to-Speech API, are used to convert the text data into speech output. The input is translated text data, and the output is speech data.

[0989] Step 7:

[0990] The server converts the generated audio data into an appropriate format (e.g., MP3 file format) and sends it to the user's device. This transfers the translated audio data to the device.

[0991] Step 8:

[0992] The terminal plays the received translated audio data through a playback device (speaker or headphones) and provides it to the user. The input is the audio data received from the server, and the output is the played audio. Specifically, this includes a process of decoding and playing the audio data.

[0993] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0994] This invention provides a system for real-time and web-based meetings using PCs or smartphones that translates multilingual speech in real time and also recognizes and reflects the user's emotions in the translation results. This system can translate user voice input into other languages ​​in real time and output it as speech data that reflects emotions. The program processing of this system is described below in natural language.

[0995] 1. Acquisition of voice input

[0996] The user uses the microphone on their PC or smartphone to perform voice input. For example, User A says, "Hello, nice to meet you."

[0997] 2. Sending audio data to the server

[0998] The terminal temporarily stores this audio data and sends it to the server. At this time, the audio data is converted to a specific format (for example, a WAV file).

[0999] 3. Execution of Speech Recognition

[1000] The server inputs the received voice data to the speech recognition engine and converts the voice into text data. As the speech recognition engine, an external speech recognition API (for example, a general cloud-based service) is used. Here, text data such as "Hello, nice to meet you" is generated.

[1001] 4. Execution of Emotion Recognition

[1002] The server inputs the received voice data to the emotion engine to identify the user's emotion. The emotion engine identifies the emotion from the voice data and outputs the emotion (for example, joy, sadness, excitement, etc.). Here, the result of emotion recognition is identified as joy.

[1003] 5. Detection of Language

[1004] The server uses a language detection algorithm to automatically detect the language of the text data. This algorithm also uses an external language detection API. For example, it is detected that this text data is in Japanese.

[1005] 6. Execution of Translation

[1006] The server inputs the detected Japanese text to a translation engine to translate it into the target language. As the translation engine, an external translation API is used. Here, the text "Hello, nice to meet you" is translated into Chinese as "你好,很高兴认识你".

[1007] 7. Reflection of Emotion in the Translation Result

[1008] The server reflects the identified emotion (in this case, "joy") in the translated text. For this purpose, emotion information is added to the text output by the translation engine. Specifically, the intonation, strength, speed, etc. of the voice are adjusted based on the emotion information.

[1009] 8. Voice synthesis of translated text

[1010] The server inputs the translated text data containing emotion information into the voice synthesis engine and converts it into voice data. As the voice synthesis engine, an external voice synthesis API is utilized. Here, the translated text "Hello, nice to meet you" is converted into Chinese voice data reflecting a happy emotion.

[1011] 9. Transmission of voice data to the terminal

[1012] The server converts the generated voice data into a certain format (e.g., MP3 file) and prepares to transmit it to the user's terminal. This voice data is transmitted to the terminal.

[1013] 10. Output of translated voice

[1014] The terminal receives the voice data sent from the server. The terminal plays this voice data through a playback device (speaker or headphones) and provides the translated voice to the user. In this way, the voice of user A, "Kon'nichiwa, Hajimemashite," is transmitted to user B as the Chinese voice "Hello, nice to meet you" reflecting the emotion.

[1015] As a specific example, when user A (a Japanese speaker) communicates with user B (a Chinese speaker) in a real meeting, by using this system, the speech of user A is quickly translated into Chinese and transmitted to user B in a form reflecting the emotion. In this way, smooth and emotion - accompanied communication among multiple languages is realized.

[1016] In summary, the system of the present invention enables smooth and emotionally charged multilingual communication by translating user speech into multiple languages ​​in real time, recognizing emotions, and outputting them back as speech. This system features flexibility and high-precision processing through the use of external APIs, meeting the needs of large-scale global communication.

[1017] The following describes the processing flow.

[1018] Step 1:

[1019] The user speaks into the microphone of their PC or smartphone. For example, they might say, "Hello, nice to meet you."

[1020] Step 2:

[1021] The device temporarily stores the audio signal acquired from the microphone as digital audio data.

[1022] Step 3:

[1023] The device converts the recorded audio data into a specific format (for example, a WAV file).

[1024] Step 4:

[1025] The device sends voice data to the server over the network.

[1026] Step 5:

[1027] The server receives the audio data.

[1028] Step 6:

[1029] The server inputs the received audio data into a speech recognition engine, which then converts the audio data into text data. An external speech recognition API (such as a common cloud-based service) is used as the speech recognition engine.

[1030] Step 7:

[1031] The server obtains the text data (e.g., "Hello, nice to meet you") returned from the speech recognition engine.

[1032] Step 8:

[1033] The server inputs the received voice data into the emotion engine to identify the user's emotion. The emotion engine identifies the emotion (e.g., joy, sadness, excitement, etc.) from the voice data and outputs that emotion. Here, the result of emotion recognition is identified as "joy".

[1034] Step 9:

[1035] The server inputs the text data into the language detection algorithm to detect the language of the text data. The language detection algorithm also uses an external language detection API.

[1036] Step 10:

[1037] The server detects that the language of the text data is Japanese.

[1038] Step 11:

[1039] The server inputs the Japanese text data into a translation engine to translate it into a target language (e.g., Chinese). As the translation engine, an external translation API is used.

[1040] Step 12:

[1041] The server obtains the translated text data (e.g., "你好,很高兴认识你") returned from the translation engine.

[1042] Step 13:

[1043] Based on the emotion identified by the server (in this case, "joy"), add emotion information to the translated text.

[1044] Step 14:

[1045] The server inputs the translated text data containing emotion information into the text-to-speech engine and converts it into audio data. As the text-to-speech engine, an external text-to-speech API is utilized. Here, the translated text "Hello, nice to meet you" is converted into Chinese audio data reflecting the emotion of joy.

[1046] Step 15:

[1047] The server converts the generated audio data into a certain format (for example, an MP3 file).

[1048] Step 16:

[1049] The server sends the audio data to the terminal.

[1050] Step 17:

[1051] The terminal receives the audio data sent from the server.

[1052] Step 18:

[1053] The terminal plays the received audio data through a playback device (such as a speaker or headphones) and provides the translated audio to the user.

[1054] Through this specific processing step, the voice "Hello, nice to meet you" of User A is translated into the Chinese voice "你好,很高兴认识你" in real time with high accuracy and conveyed to User B in a form that reflects the emotion.

[1055] (Example 2)

[1056] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1057] In modern society, the importance of technologies for real-time communication across multiple languages ​​is increasing. However, conventional voice translation systems simply translate languages ​​and fail to accurately reflect the user's emotions. Furthermore, it has been difficult to simultaneously perform real-time voice translation and emotion recognition. As a result, there has been a problem in achieving smooth conversations that include emotions in multilingual communication.

[1058] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1059] In this invention, the server includes means for acquiring voice data from a user, means for transmitting the acquired voice data to the server, means for the server to convert the voice data into text data, means for the server to detect the language of the text data, means for the server to translate the text data into a target language, means for the server to reflect emotional information in the translated text data, means for the server to convert the translated text data with reflected emotional information into voice data, means for transmitting the translated voice data to the user's terminal, and means for the terminal to play the translated voice data. This enables real-time voice translation that reflects emotions across multiple languages.

[1060] A "user" refers to a person who uses the system to input voice data.

[1061] "Audio data" refers to digital audio signals acquired through devices such as microphones.

[1062] A "terminal" is a device used by a user to input or play back received audio data, and includes PCs, smartphones, and other similar devices.

[1063] A "server" refers to a remote computer system that processes audio data transmitted from a terminal and performs tasks such as translation and emotion recognition.

[1064] "Text data" refers to character data converted from speech data by a speech recognition engine.

[1065] "Language detection" refers to the process of identifying the language of acquired text data.

[1066] "Target language" refers to the target language after translation.

[1067] "Emotional information" refers to information about the user's emotions identified from voice data, including emotions such as joy, sadness, and excitement.

[1068] "Translated text data" refers to character data that has been translated into the target language.

[1069] "Speech synthesis" refers to the process of generating speech data from text data.

[1070] A "speech recognition engine" refers to software or an API that converts speech data into text data.

[1071] A "speech synthesis engine" refers to software or an API that converts text data into speech data.

[1072] An "emotion recognition engine" refers to software or an API that identifies a user's emotions from voice data.

[1073] This invention relates to a system for enabling smooth and emotionally charged communication between multiple languages ​​in real-world and web meetings. The system has the function of taking voice input from a user, translating it into another language in real time, and outputting it while reflecting the user's emotions.

[1074] The main components of the system are the user's terminal, the server, and the external API, and it functions as follows:

[1075] 1. Acquisition of voice input

[1076] The user inputs voice using the microphone on a device such as a PC or smartphone. The device acquires this voice data as a digital signal and temporarily stores it. For example, user A says, "Hello, nice to meet you."

[1077] 2. Sending audio data to the server

[1078] The terminal converts the acquired audio data into a WAV file and sends it to the server. The server receives the data as an HTTP request and stores the audio data internally for further processing.

[1079] 3. Perform speech recognition

[1080] The server sends the received audio data to a cloud-based speech recognition API (for example, a common speech recognition engine) to convert the audio into text data. In this process, the speech recognition engine analyzes the audio signal and generates the corresponding text, "Hello, nice to meet you."

[1081] 4. Performing emotion recognition

[1082] The server then sends the audio data to an emotion recognition engine (for example, a general emotion recognition engine) to identify the user's emotion. The emotion recognition engine analyzes the tone and pitch of the voice to identify the emotion (joy, sadness, excitement, etc.). In this case, the emotion "joy" is identified.

[1083] 5. Language detection

[1084] The server sends the text data obtained from speech recognition to a general language detection API to identify the language being used. This API detects that the text data is in Japanese.

[1085] 6. Execution of Translation

[1086] The server sends the detected Japanese text to a cloud-based translation API (e.g., a general translation engine) to translate it into the target language (e.g., Chinese). As a result, the text "こんにちは、はじめまして" is translated to "你好,很高兴认识你".

[1087] 7. Reflection of Emotion

[1088] The server reflects the identified emotion (in this case, "happiness") in the translated text. Specifically, it adds emotion information to reflect the intonation, strength, and speed of the voice. This enables the generation of voice that reflects the user's emotion.

[1089] 8. Execution of Speech Synthesis

[1090] The server sends the translated text data containing emotion information to a speech synthesis engine (e.g., a general speech synthesis engine) to convert it into voice data that reflects the emotion information. The speech synthesis engine generates the translated text "你好,很高兴认识你" in Chinese pronunciation as voice data that reflects the emotion of "happiness".

[1091] 9. Transmission of Voice Data to the Terminal

[1092] The server encodes this generated voice data into the MP3 format and sends it to the user's terminal. The server encrypts the data using HTTPS and transfers it securely.

[1093] 10. Output of Translated Voice

[1094] The terminal receives the voice data sent from the server. The terminal outputs this voice data through a playback device (such as a speaker or headphones) to provide the translated voice reflecting the emotion to User B. As a result, the voice of User A, "Hello, nice to meet you for the first time," is conveyed to User B as the Chinese voice "你好,很高兴认识你" with a happy emotion.

[1095] Specific example

[1096] For example, assume that User A (a Japanese speaker) is talking to User B, a Chinese speaker, in a real meeting. By using this system, the speech of User A is quickly translated into Chinese and conveyed to User B in a form that reflects the emotion. As a result, smooth and emotional communication among multiple languages is realized.

[1097] Example of prompt sentence

[1098] "User A who speaks Japanese is talking to User B who speaks Chinese. When User A says 'Hello, nice to meet you for the first time,' please translate that voice into Chinese and output it as a voice reflecting the emotion (happiness)."

[1099] As described above, the system of the present invention can smoothly communicate with users by providing real-time and emotion-reflecting translation in communication among multiple languages.

[1100] The flow of the specific process in Example 2 will be described with reference to FIG. 13.

[1101] Step 1:

[1102] The user performs voice input toward the microphone of a PC or a smartphone. As a result, the microphone of the terminal captures the user's voice as a digital signal. The input is the user's voice, and the output is voice data in digital format.

[1103] Step 2:

[1104] The terminal temporarily stores the acquired voice data in the internal memory and converts the data into a file in WAV format. The input is digital voice data, and the output is a voice file in WAV format. The terminal sends this voice data file to the server as an HTTP request.

[1105] Step 3:

[1106] The server sends the received voice data to a cloud-based voice recognition API. The voice recognition API analyzes the voice signal and converts it into corresponding text data. The input is a voice file in WAV format, and the output is text data "Hello, nice to meet you".

[1107] Step 4:

[1108] The server sends the voice data to an emotion recognition engine to identify the user's emotion. The emotion recognition engine analyzes the tone and pitch of the voice, etc., and recognizes the emotion (happiness, sadness, excitement, etc.). The input is voice data, and the output is emotion information "happiness".

[1109] Step 5:

[1110] The server sends the text data obtained by voice recognition to a language detection API to identify the language being used. The API analyzes the language characteristics of the text and identifies the corresponding language. The input is text data, and the output is language information "Japanese".

[1111] Step 6:

[1112] The server sends the detected Japanese text data to a translation API to translate it into the target language. The translation engine translates "Hello, nice to meet you" into "你好,很高兴认识你". The input is Japanese text data, and the output is Chinese translated text data.

[1113] Step 7:

[1114] The server reflects the identified emotion "joy" in the translated text. Specifically, emotion information for adjusting the intonation, strength, and speed of the voice is added. The input is Chinese translation text data and emotion information, and the output is translation text data reflecting the emotion.

[1115] Step 8:

[1116] The server sends the translation text data containing emotion information to the speech synthesis engine and converts it into voice data. The speech synthesis engine generates the translated text "Hello, nice to meet you" in Chinese pronunciation and as voice data reflecting the emotion of "joy". The input is translation text data reflecting the emotion, and the output is voice data.

[1117] Step 9:

[1118] The server encodes the generated voice data into the MP3 format and prepares to send it to the user's terminal. This voice data is sent to the terminal as an HTTP response. The input is the generated voice data, and the output is voice data in the MP3 format.

[1119] Step 10:

[1120] The terminal receives the voice data sent from the server. The terminal plays this voice data through a playback device (such as a speaker or headphones) and provides the translated voice to the user. The input is the voice data received from the server, and the output is the played voice. Thus, the voice of user A, "Konnichiwa, hajimemashite", is transmitted to user B as the Chinese voice "Hello, nice to meet you" reflecting the emotion.

[1121] (Application Example 2)

[1122] Next, Application Example 2 will be described. In the following description, the data processing device 12 is referred to as the "server", and the headset-type terminal 314 is referred to as the "terminal".

[1123] As the number of multinational workers in factories increases, it becomes necessary to communicate work instructions quickly and accurately. However, language barriers exist, which can lead to misunderstandings and confusion. Accurately conveying the urgency and emotions of the person giving the instructions is also crucial, but this too presents a challenge. Therefore, in addition to real-time translation between multiple languages, there is a need for methods to communicate work instructions while taking emotions into consideration.

[1124] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1125] In this invention, the server includes means for acquiring voice data from a user, means for transmitting the acquired voice data to the server, and means for the server to input the received voice data into an emotion engine, identify emotions, and reflect the emotional information in the translated voice data. This makes it possible to accurately translate the voice instructions of a work supervisor into multiple languages ​​and to convey them to a factory robot while reflecting the supervisor's emotions.

[1126] A "user" refers to a person who uses the system to input voice data.

[1127] "Voice data" refers to data that records the user's voice in digital format.

[1128] A "server" refers to a computer system that processes and converts audio data.

[1129] "Text data" refers to the character information that has been converted from audio data by a speech recognition engine.

[1130] "Means for detecting language" refers to algorithms or programs that identify the language of text data.

[1131] The "target language" refers to the language that is the subject of the translation.

[1132] An "emotion engine" refers to software or an algorithm that identifies emotions from audio data and outputs emotional information.

[1133] "Emotional information" refers to data that digitally represents emotions (e.g., joy, sadness, urgency, etc.) identified from the user's voice.

[1134] "Multilingual" refers to multiple different languages.

[1135] "Factory robots" refer to automated equipment or devices that perform tasks within a factory.

[1136] "Work instructions" refer to specific instructions that should be given to workers and equipment within a factory.

[1137] A "translation API" refers to an external software interface that automatically translates text data into another language.

[1138] A "speech recognition API" refers to an external software interface that converts speech data into text data.

[1139] A "speech synthesis engine" refers to software or an algorithm that converts text data into speech data.

[1140] A "prompt" refers to a text-based instruction given to a generative AI model.

[1141] This invention relates to a system for transmitting multilingual and emotionally reflective voice instructions to factory robots within a factory. This system translates the voice of the person giving the instructions into multiple languages ​​in real time, recognizes the person's emotions, and reflects them in the translation result.

[1142] Hardware and software

[1143] This system uses the following hardware and software:

[1144] 1. Hardware

[1145] Smart glasses: A device for inputting voice.

[1146] Server: A central device used for data processing and calculations.

[1147] Robot control terminal: A device that plays voice data and transmits instructions to the robot.

[1148] 2. Software

[1149] Speech-to-Text APIs: To convert speech data into text data, common cloud-based speech recognition APIs are used. An example is the Google Cloud Speech-to-Text API.

[1150] Emotion Recognition API: To identify emotions from audio data, an external emotion recognition API is used. IBM Watson Tone Analyzer is an example of this.

[1151] Language detection API: To detect the language of text data, use an external language detection API. For example, use the Microsoft Azure Language Recognition API.

[1152] Translation API: Use an external translation API to translate text data into the target language. For example, the Google Cloud Translation API supports this.

[1153] Speech synthesis APIs: Speech synthesis APIs are used to convert translated text data into speech data that reflects emotion. Amazon Polly is an example of this.

[1154] Data processing and calculation

[1155] 1. Acquisition and transmission of audio data

[1156] The user uses the microphone on the smart glasses to input voice data. The acquired voice data is temporarily stored on the smart glasses' terminal and then sent to the server. At this time, the voice data is saved in WAV format.

[1157] 2. Speech Recognition Processing

[1158] The server uses the Google Cloud Speech-to-Text API to convert the received audio data into text data. The converted text data is then processed further.

[1159] 3. Emotion recognition processing

[1160] The server uses IBM Watson Tone Analyzer to identify the user's emotions from the voice data. The identified emotion information is then added to the text data.

[1161] 4. Language detection and translation

[1162] The Microsoft Azure Language Recognition API is used to detect the language, and the Google Cloud Translation API is used to translate it into the target language.

[1163] 5. Reflecting emotions in the translation results

[1164] The translated text data incorporates sentiment information identified by IBM Watson Tone Analyzer. Based on this sentiment information, the intonation and speed of the speech are adjusted.

[1165] 6. Speech synthesis and terminal transmission

[1166] Amazon Polly is used to convert the audio data into emotionally reflective audio, which is then sent to the robot control terminal. Finally, the terminal plays this audio data.

[1167] Specific example

[1168] For example, if a factory supervisor gives the instruction to "hurry up and start assembling the next product batch," this system will perform the following actions:

[1169] 1. The audio input is acquired in WAV format and sent to the server.

[1170] 2. Use the Google Cloud Speech-to-Text API to generate the text data "Hurry up and start assembling the next product batch."

[1171] 3. Use IBM Watson Tone Analyzer to identify "urgency" and emotion.

[1172] 4. Detect the language of the text data as Japanese using the Microsoft Azure Language Recognition API.

[1173] 5. Use the Google Cloud Translation API to translate the text data into English as "Hurry up and start assembling the next product batch".

[1174] 6. The "urgent" emotion identified by IBM Watson Tone Analyzer is reflected in the voice, and then converted into voice data using Amazon Polly.

[1175] 7. Finally, the translated audio data is sent to the robot control terminal and played back.

[1176] Example of a prompt

[1177] The user said via voice input, "Hurry up and start assembling the next product batch." Translate this voice data into English in real time and convey the instruction to the robot, reflecting an urgent emotion. The translated voice should be "Hurry up and start assembling the next product batch," and the voice should be synthesized to reflect an urgent emotion.

[1178] This system will enable factory supervisors to efficiently communicate multilingual and emotionally resonant voice instructions to factory robots.

[1179] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1180] Step 1:

[1181] The user uses the microphone on their smart glasses to input voice commands. For example, the user might say, "Hurry up and start assembling the next product batch." This voice data is the input. The voice data is temporarily stored in WAV format on the smart glasses' terminal. This terminal then sends the data to the server.

[1182] Step 2:

[1183] The server receives audio data from the smart glasses and inputs it into the Google Cloud Speech-to-Text API, converting the audio data into text data. The specific operations performed here are audio signal processing and string parsing. The converted text data is then output.

[1184] Step 3:

[1185] The server inputs text data into the IBM Watson Tone Analyzer for sentiment recognition. The input is text data, and the output is sentiment information (e.g., "urgent"). The specific operation is text-based sentiment analysis.

[1186] Step 4:

[1187] The server inputs text data into the Microsoft Azure Language Recognition API to detect the language. The input is text data, and the output is language information (e.g., Japanese). The specific operation is language profiling.

[1188] Step 5:

[1189] The server inputs text data into the Google Cloud Translation API and translates it into the target language (e.g., English). The output is the translated text data (e.g., "Hurry up and start assembling the next product batch"). The specific operation is a text translation process.

[1190] Step 6:

[1191] The server inputs translated text data and sentiment information into Amazon Polly, which then synthesizes it into speech data. The input is translated text data and sentiment information, and the output is speech data that reflects the sentiment. The specific operation is speech synthesis.

[1192] Step 7:

[1193] The server sends the generated voice data to the robot control terminal. The input is voice data that reflects emotions, and the server performs the specific action of transferring the data to the terminal.

[1194] Step 8:

[1195] The terminal receives audio data sent from the server and plays this audio data. The output is a translated voice instruction that reflects emotion. The specific operation involves decoding the audio data and playing it back through the speaker.

[1196] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1197] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1198] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1199] [Fourth Embodiment]

[1200] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1201] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1202] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1203] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1204] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1205] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1206] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1207] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1208] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1209] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1210] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1211] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1212] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1213] This invention relates to a system that translates multilingual speech in real time using a PC or smartphone. This system can translate user voice input into another language in real time and output it back as speech. The program processing of this system is described below in natural language.

[1214] 1. Acquisition of voice input

[1215] The user performs voice input using the microphone of a PC or smartphone. As an example, user A says "Hello, nice to meet you for the first time."

[1216] 2. Server transmission of voice data

[1217] The terminal temporarily stores this voice data and transmits it to the server. At this time, the voice data is converted into a certain format (for example, a WAV file).

[1218] 3. Execution of voice recognition

[1219] The server inputs the received voice data to a voice recognition engine and converts the voice into text data. As the voice recognition engine, it is possible to use an external voice recognition API (for example, a general cloud-based service). Here, text data "Hello, nice to meet you for the first time." is generated.

[1220] 4. Detection of language

[1221] The server uses a language detection algorithm to automatically detect the language of this text data. This algorithm can also use an external language detection API. For example, it is detected that this text data is in Japanese.

[1222] 5. Execution of translation

[1223] The server inputs the detected Japanese text into a translation engine to translate it into the target language. As the translation engine, an external translation API can be used. Here, the text "Hello, nice to meet you for the first time." is translated into Chinese as "你好,很高兴认识你".

[1224] 6. Speech synthesis of translated text

[1225] The server inputs the translated Chinese text into a speech synthesis engine in order to convert it into speech data. As the speech synthesis engine, an external speech synthesis API can be used. Here, the translated text "Hello, nice to meet you" is converted into Chinese speech data.

[1226] 7. Transmission of Speech Data to the Terminal

[1227] The server converts the generated speech data into a certain format (for example, an MP3 file) and prepares to transmit it to the user's terminal. This speech data is transmitted to the terminal.

[1228] 8. Output of Translated Speech

[1229] The terminal receives the speech data sent from the server. The terminal plays this speech data through a playback device (such as a speaker or headphones) and provides the translated speech to the user. In this way, the speech of user A, "Kon'nichiwa, Hajimemashite", is transmitted to user B as the Chinese speech "Hello, nice to meet you".

[1230] As a specific example, when user A (a Japanese speaker) communicates with Chinese speaker user B in a real meeting, by using this system, the speech of user A is quickly translated into Chinese and transmitted to user B. In this way, smooth communication between multiple languages is realized.

[1231] As described above, the system of the present invention enables smooth communication between multiple languages by translating the user's speech into multiple languages in real time and outputting it as speech again. This system features flexibility and high-precision processing by using external APIs, and meets the needs of large-scale global communication.

[1232] The processing flow will be described below.

[1233] Step 1:

[1234] The user speaks into the microphone of their PC or smartphone. For example, they might say, "Hello, nice to meet you."

[1235] Step 2:

[1236] The device temporarily stores the audio signal acquired from the microphone as digital audio data.

[1237] Step 3:

[1238] The device converts the recorded audio data into a specific format (for example, a WAV file).

[1239] Step 4:

[1240] The device sends voice data to the server over the network.

[1241] Step 5:

[1242] The server receives the audio data.

[1243] Step 6:

[1244] The server inputs the received audio data into a speech recognition engine, which then converts the audio data into text data. An external speech recognition API (such as a common cloud-based service) is used as the speech recognition engine.

[1245] Step 7:

[1246] The server retrieves the text data returned by the speech recognition engine (for example, "Hello, nice to meet you").

[1247] Step 8:

[1248] The server inputs the text data into a language detection algorithm to detect the language of the text data. The language detection algorithm also utilizes an external language detection API.

[1249] Step 9:

[1250] The server detects that the language of the text data is Japanese.

[1251] Step 10:

[1252] The server inputs the Japanese text data into a translation engine to translate it into a target language (e.g., Chinese). As the translation engine, an external translation API is utilized.

[1253] Step 11:

[1254] The server obtains the translated text data (e.g., "Hello, nice to meet you") returned from the translation engine.

[1255] Step 12:

[1256] The server inputs the translated text data into a speech synthesis engine to convert it into audio data. As the speech synthesis engine, an external speech synthesis API is utilized.

[1257] Step 13:

[1258] The server obtains the audio data (e.g., Chinese audio data) returned from the speech synthesis engine.

[1259] Step 14:

[1260] The server converts the generated audio data into a certain format (e.g., MP3 file).

[1261] Step 15:

[1262] The server transmits the audio data to the terminal.

[1263] Step 16:

[1264] The terminal receives audio data sent from the server.

[1265] Step 17:

[1266] The device plays the received audio data through a playback device (speaker or headphones) and provides the user with translated audio.

[1267] Through the steps described above, the user's Japanese voice input is translated into Chinese in real time with high accuracy and transmitted as audio to other users.

[1268] (Example 1)

[1269] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1270] In today's global society, facilitating smooth communication between people who speak different languages ​​is crucial. However, real-time translation and playback of speech across multiple languages ​​is technically complex, and building a user-friendly system is a particularly challenging task. To overcome this challenge, efficient and highly accurate speech recognition, translation, and speech synthesis are necessary, along with a flexible and reliable system to achieve these capabilities.

[1271] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1272] In this invention, the server includes means for acquiring audio data from a user, means for transmitting the acquired audio data to the server, means for the server to convert the audio data into text data, means for the server to detect the language of the text data, means for the server to translate the text data into a target language, means for the server to convert the translated text data into audio data, means for transmitting the translated audio data to the user's terminal, means for the terminal to play the translated audio data, means for the server to convert the format of the audio data using an external audio processing API, and means for the terminal to output the received audio data to a predetermined playback device. This enables real-time audio translation and playback between users using different languages.

[1273] "Audio data" refers to data that records and stores the voice spoken by a user in digital format.

[1274] A "server" is a computer system that provides various services over a network, and it plays a role in processing and converting audio data.

[1275] "Text data" refers to data obtained by converting audio data into written text.

[1276] "Language detection" is the process of identifying the language of text data.

[1277] The "target language" is the language that will be output as a result of the translation.

[1278] "Speech recognition" is a technology that analyzes speech data and converts it into corresponding text data.

[1279] "Translation" is the process of converting text data written in one language into another language.

[1280] "Speech synthesis" is a technology that generates speech data based on text data.

[1281] An "external speech processing API" is an interface that provides external services and functions for performing speech recognition and speech synthesis on speech data.

[1282] A "device" refers to a device used by a user, such as a PC or smartphone.

[1283] A "playback device" refers to a device such as a speaker or headphones used to output audio data as sound.

[1284] This invention is a system that allows users to translate multilingual speech in real time using a PC or smartphone. The program processing of this system is described below in natural language.

[1285] To implement this system, the following hardware and software will be used:

[1286] Hardware: PC, smartphone, microphone, speakers, headphones

[1287] Software: Google Cloud Speech-to-Text API, Google Cloud Translation API, Google Cloud Text-to-Speech API

[1288] First, the user uses the microphone on their PC or smartphone to input voice. For example, user A might say, "Hello, nice to meet you." The device temporarily stores this voice data, converts it to WAV format, and sends it to the server.

[1289] The server inputs the received audio data into the Google Cloud Speech-to-Text API and converts it into text data that says "Hello, nice to meet you." Next, the server automatically detects the language of this text data using the Google Cloud Translation API and determines that it is Japanese.

[1290] Next, the server translates the Japanese text into Chinese using the Google Cloud Translation API. For example, the text "こんにちは、はじめまして" is converted into "你好,很高兴认识你" in Chinese. Then, the server inputs this translated text into the Google Cloud Text-to-Speech API to generate the Chinese voice data of "你好,很高兴认识你".

[1291] The server converts the generated voice data into the MP3 format and sends it to the terminal. The terminal outputs the received voice data to a playback device (such as a speaker or headphones) to convey it to User B. In this way, the voice of User A is transmitted to User B as Chinese voice.

[1292] As a specific example, when User A communicates with Chinese-speaking User B in a real meeting, by using this system, the speech of User A is quickly translated into Chinese and transmitted to User B. This enables smooth communication among multiple languages.

[1293] Example of a prompt sentence for inputting into the generation AI model:

[1294] When implementing a program for performing speech recognition, language detection, translation, and speech synthesis, please follow the following steps:

[1295] 1. The user spoke in Japanese as "こんにちは、はじめまして". The terminal temporarily stores this voice data, converts it into the WAV file format, and sends it to the server.

[1296] 2. The server receives the WAV file and converts it into the text "こんにちは、はじめまして" using the Google Cloud Speech-to-Text API.

[1297] 3. The server automatically detects the language of this text data using the Google Cloud Translation API and determines that it is Japanese.

[1298] 4. The server translates the text data into Chinese using the Google Cloud Translation API.

[1299] 5. The translated text "Hello, nice to meet you" is converted into audio data using the Google Cloud Text-to-Speech API.

[1300] 6. The generated audio data is converted into the MP3 format and sent to the terminal.

[1301] 7. The terminal outputs the received audio data through a playback device and provides it to User B.

[1302] As described above, the system of the present invention enables smooth communication between multiple languages by translating the user's voice into multiple languages in real time and outputting it as voice again. This system features flexibility and high-precision processing by using external APIs, and meets the needs of large-scale global communication.

[1303] The flow of the specific process in Example 1 will be described using FIG. 11.

[1304] Step 1:

[1305] The user performs voice input using the microphone of a PC or smartphone. The subject is the user. The specific operations in this step are as follows.

[1306] Input: The user says "Hello, nice to meet you".

[1307] Output: It is input to the terminal as audio data.

[1308] Operation: The user picks up their smartphone and speaks into the microphone, or speaks into the microphone on their PC. The voice is captured by the microphone and temporarily stored as digital audio data.

[1309] Step 2:

[1310] The terminal temporarily stores the audio data, converts it to WAV format, and sends it to the server. The subject is the terminal. The specific actions in this step are as follows:

[1311] Input: Temporarily stored audio data.

[1312] Output: WAV format audio data sent to the server.

[1313] Operation: The device converts temporarily stored audio data into a WAV file format and sends it to the server.

[1314] Step 3:

[1315] The server inputs the received audio data into the Google Cloud Speech-to-Text API, converting the audio data into text data. The subject is the server. The specific actions in this step are as follows:

[1316] Input: WAV format audio data received from the device.

[1317] Output: Text data "Hello, nice to meet you."

[1318] Operation: The server sends WAV formatted audio data to the Google Cloud Speech-to-Text API and retrieves the returned text data.

[1319] Step 4:

[1320] The server automatically detects the language of the text data. The subject is the server. The specific actions in this step are as follows:

[1321] Input: Text data "Hello, nice to meet you for the first time".

[1322] Output: Judgment that it is Japanese.

[1323] Operation: The server inputs the text data into the Google Cloud Translation API and receives the language detection result.

[1324] Step 5:

[1325] The server translates the Japanese text with the Google Cloud Translation API. The subject is the server. The specific operations in this step are as follows.

[1326] Input: Japanese text data "Hello, nice to meet you for the first time".

[1327] Output: Chinese text data "你好,很高兴认识你".

[1328] Operation: The server sends the Japanese text to the Google Cloud Translation API and receives the translated Chinese text.

[1329] Step 6:

[1330] The server converts the translated Chinese text into audio data with the Google Cloud Text-to-Speech API. The subject is the server. The specific operations in this step are as follows.

[1331] Input: Chinese text data "你好,很高兴认识你".

[1332] Output: Chinese audio data.

[1333] Operation: The server sends Chinese text to the Google Cloud Text-to-Speech API and retrieves the returned audio data.

[1334] Step 7:

[1335] The server converts the generated audio data into MP3 format and sends it to the user's terminal. The subject is the server. The specific actions in this step are as follows:

[1336] Input: Translated audio data.

[1337] Output: MP3 audio data sent to the user's terminal.

[1338] Operation: The server converts the audio data to MP3 format and prepares it for transmission to the terminal. The converted audio data is then sent to the user's terminal.

[1339] Step 8:

[1340] The terminal receives audio data sent from the server and plays it back through the playback device. The subject is the terminal. The specific actions in this step are as follows:

[1341] Input: MP3 audio data received from the server.

[1342] Output: The translated audio "Hello, I'm happy to have met you" is played.

[1343] Operation: The device receives audio data in MP3 format and outputs it through its built-in playback device (speaker or headphones). Communication is established when User B listens to the audio.

[1344] (Application Example 1)

[1345] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1346] In autonomous vehicles, it is difficult to facilitate rapid and accurate communication between passengers who speak multiple languages, as well as between passengers and the system. The present invention aims to provide a system that translates speech in real time and enables smooth communication between passengers who speak different languages.

[1347] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1348] In this invention, the server includes means for acquiring voice data from a user, means for transmitting the acquired voice data to the server, means for the server to convert the voice data into text data, means for the server to detect the language of the text data, means for the server to translate the text data into a target language, means for the server to convert the translated text data back into voice data, means for transmitting the translated voice data to the user's terminal, means for the terminal to play the translated voice data, and means integrated into the vehicle system to translate passenger voices into other languages ​​in real time and output them as voice. This enables smooth multilingual communication within an autonomous vehicle.

[1349] A "user" is an individual or group that uses the system.

[1350] "Audio data" refers to information recorded in digital format from the voice spoken by a user.

[1351] A "server" is a computer system that provides functions such as processing, converting, and transmitting audio data.

[1352] "Text data" refers to data in a format where audio data has been converted into written text.

[1353] "Language detection" is the process of determining which language text data is written in.

[1354] The "target language" is the target language to which the translated text will be converted.

[1355] Translation is the process of converting text written in one language into another language.

[1356] "Speech synthesis" is a technology that converts text data into speech data.

[1357] A "device" refers to a computer, smartphone, tablet, or other device used by a user.

[1358] "Audio output" is the process of making audio data playable to the user through a playback device such as a speaker.

[1359] A "vehicle system" refers to a system that provides various functions, including the operation of autonomous vehicles and infotainment systems.

[1360] "Real-time" refers to a state where processing and output occur almost instantly, with virtually no delay.

[1361] This invention is a system that translates user voice data into other languages ​​in real time, facilitating smooth multilingual communication within autonomous vehicles.

[1362] The server acquires the user's voice data and converts it into text data. Specifically, it collects voice data spoken by the user through a smartphone or in-car microphone. The collected voice data is sent to the server and converted into text data by a speech recognition engine on the server. A common speech recognition API (e.g., Google Speech-to-Text API) can be used as this speech recognition engine.

[1363] Next, the server detects the language of the converted text data and translates it into the target language. This translation process can utilize an external translation API (e.g., Google Translate API). The translated text data is then input into a speech synthesis engine (e.g., Google Text-to-Speech API) and converted into speech data in the target language.

[1364] The translated audio data generated on the server is sent to the user's device. The device then plays the received translated audio data through speakers or headphones. This system enables users to communicate in different languages ​​in real time.

[1365] As a concrete example, consider a scenario where passenger A, a Japanese speaker, is in an autonomous vehicle and asks passenger B, an English speaker, "Where is this car going?" Passenger A's Japanese voice is captured by a smartphone or the car's microphone and sent to a server. The server converts the voice data into text data, which is then translated into English using a translation engine. This text is then converted back into English voice data and output to passenger B as "Where is this car going?". Similarly, passenger B's English response is also translated into Japanese and conveyed to passenger A.

[1366] An example of a prompt given so far is, "Please translate 'Hello, nice to meet you' into English and output it as speech."

[1367] This system enables smooth multilingual communication within autonomous vehicles.

[1368] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1369] Step 1:

[1370] The user uses their smartphone or the in-car microphone to input voice data. The voice data is acquired digitally and temporarily stored on the device. This is how the voice data is collected.

[1371] Step 2:

[1372] The device sends the acquired audio data to the server. The audio data is converted to an appropriate format (e.g., WAV file format) and uploaded to the server. This transfers the audio data to the server.

[1373] Step 3:

[1374] The server inputs the received audio data into a speech recognition engine and converts the audio data into text data. Specifically, it uses an external speech recognition service such as the Google Speech-to-Text API to analyze the audio waveform and generate the corresponding text data. The input is audio data, and the output is text data.

[1375] Step 4:

[1376] The server uses an external language detection API to determine the language of the generated text data. For example, it can determine whether the text data is in Japanese or English. The input is the text data, and the output is language information. Based on the language information, the processing of the text data proceeds to the next step.

[1377] Step 5:

[1378] The server uses an external translation engine, such as the Google Translate API, to translate the detected text data into the target language. This is the process of converting text data into another language. The input is text data and language information, and the output is translated text data.

[1379] Step 6:

[1380] The server inputs the translated text data into a speech synthesis engine and converts it into speech data. External speech synthesis services, such as the Google Text-to-Speech API, are used to convert the text data into speech output. The input is translated text data, and the output is speech data.

[1381] Step 7:

[1382] The server converts the generated audio data into an appropriate format (e.g., MP3 file format) and sends it to the user's device. This transfers the translated audio data to the device.

[1383] Step 8:

[1384] The terminal plays the received translated audio data through a playback device (speaker or headphones) and provides it to the user. The input is the audio data received from the server, and the output is the played audio. Specifically, this includes a process of decoding and playing the audio data.

[1385] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1386] This invention provides a system for real-time and web-based meetings using PCs or smartphones that translates multilingual speech in real time and also recognizes and reflects the user's emotions in the translation results. This system can translate user voice input into other languages ​​in real time and output it as speech data that reflects emotions. The program processing of this system is described below in natural language.

[1387] 1. Acquisition of voice input

[1388] The user uses the microphone on their PC or smartphone to perform voice input. For example, User A says, "Hello, nice to meet you."

[1389] 2. Sending audio data to the server

[1390] The terminal temporarily stores this audio data and sends it to the server. At this time, the audio data is converted to a specific format (for example, a WAV file).

[1391] 3. Perform speech recognition

[1392] The server inputs the received audio data into a speech recognition engine, which then converts the audio into text data. An external speech recognition API (such as a common cloud-based service) is used as the speech recognition engine. In this example, the text data "Hello, nice to meet you" is generated.

[1393] 4. Performing emotion recognition

[1394] The server inputs the received audio data into the emotion engine to identify the user's emotion. The emotion engine identifies the emotion from the audio data and outputs that emotion (e.g., joy, sadness, excitement). In this case, the emotion recognition result is identified as joy.

[1395] 5. Language detection

[1396] The server uses a language detection algorithm to automatically detect the language of the text data. This algorithm also utilizes an external language detection API. For example, it might detect that this text data is in Japanese.

[1397] 6. Perform the translation.

[1398] The server inputs the detected Japanese text into a translation engine in order to translate it into the target language. As the translation engine, an external translation API is used. Here, the text "こんにちは、はじめまして" is translated into Chinese as "你好,很高兴认识你".

[1399] 7. Reflection of Emotions in the Translation Result

[1400] The server reflects the identified emotion (in this case, "joy") in the translated text. For this purpose, emotion information is added to the text output by the translation engine. Specifically, the intonation, strength, speed, etc. of the voice are adjusted based on the emotion information.

[1401] 8. Speech Synthesis of the Translated Text

[1402] The server inputs the translated text data containing emotion information into a speech synthesis engine and converts it into voice data. As the speech synthesis engine, an external speech synthesis API is used. Here, the translated text "你好,很高兴认识你" is converted into Chinese voice data reflecting the emotion of joy.

[1403] 9. Transmission of Voice Data to the Terminal

[1404] The server converts the generated voice data into a certain format (e.g., MP3 file) and prepares to transmit it to the user's terminal. This voice data is transmitted to the terminal.

[1405] 10. Output of the Translated Voice

[1406] The terminal receives the voice data sent from the server. The terminal plays this voice data through a playback device (such as a speaker or headphones) and provides the translated voice to the user. In this way, the voice of user A, "こんにちは、はじめまして", is transmitted to user B as the Chinese voice "你好,很高兴认识你" reflecting the emotion.

[1407] As a concrete example, when User A (a Japanese speaker) communicates with User B (a Chinese speaker) in a real-world meeting, this system allows User A's statements to be quickly translated into Chinese and conveyed to User B in a way that reflects their emotions. In this way, smooth and emotionally resonant communication between multiple languages ​​is achieved.

[1408] In summary, the system of the present invention enables smooth and emotionally charged multilingual communication by translating user speech into multiple languages ​​in real time, recognizing emotions, and outputting them back as speech. This system features flexibility and high-precision processing through the use of external APIs, meeting the needs of large-scale global communication.

[1409] The following describes the processing flow.

[1410] Step 1:

[1411] The user speaks into the microphone of their PC or smartphone. For example, they might say, "Hello, nice to meet you."

[1412] Step 2:

[1413] The device temporarily stores the audio signal acquired from the microphone as digital audio data.

[1414] Step 3:

[1415] The device converts the recorded audio data into a specific format (for example, a WAV file).

[1416] Step 4:

[1417] The device sends voice data to the server over the network.

[1418] Step 5:

[1419] The server receives the audio data.

[1420] Step 6:

[1421] The server inputs the received audio data into a speech recognition engine, which then converts the audio data into text data. An external speech recognition API (such as a common cloud-based service) is used as the speech recognition engine.

[1422] Step 7:

[1423] The server retrieves the text data returned by the speech recognition engine (for example, "Hello, nice to meet you").

[1424] Step 8:

[1425] The server inputs the received audio data into the emotion engine to identify the user's emotion. The emotion engine identifies the emotion (e.g., joy, sadness, excitement) from the audio data and outputs that emotion. In this case, the emotion recognition result is identified as "joy."

[1426] Step 9:

[1427] The server inputs text data into a language detection algorithm to detect the language of the text data. The language detection algorithm also utilizes an external language detection API.

[1428] Step 10:

[1429] The server detects that the language of the text data is Japanese.

[1430] Step 11:

[1431] The server inputs Japanese text data into a translation engine to translate it into a target language (e.g., Chinese). An external translation API is used as the translation engine.

[1432] Step 12:

[1433] The server obtains the translated text data (e.g., "Hello, nice to meet you") returned from the translation engine.

[1434] Step 13:

[1435] Based on the identified emotion of the server (in this case, "joy"), the server adds emotion information to the translated text.

[1436] Step 14:

[1437] The server inputs the translated text data containing emotion information into the speech synthesis engine and converts it into audio data. As the speech synthesis engine, an external speech synthesis API is utilized. Here, the translated text "Hello, nice to meet you" is converted into Chinese audio data reflecting the emotion of joy.

[1438] Step 15:

[1439] The server converts the generated audio data into a certain format (e.g., MP3 file).

[1440] Step 16:

[1441] The server transmits the audio data to the terminal.

[1442] Step 17:

[1443] The terminal receives the audio data sent from the server.

[1444] Step 18:

[1445] The terminal plays the received audio data through a playback device (speaker or headphones) and provides the translated audio to the user.

[1446] Through this specific processing step, the voice of User A, "Hello, nice to meet you for the first time," is translated in real time with high precision into the Chinese voice "Hello, nice to meet you," and is transmitted to User B in a form that reflects the emotion.

[1447] (Example 2)

[1448] Next, Example 2 will be described. In the following description, the data processing device 12 is referred to as a "server," and the robot 414 is referred to as a "terminal."

[1449] In modern society, the importance of technologies for real-time communication among multiple languages is increasing. However, conventional voice translation systems simply translate languages and cannot accurately reflect the emotions of users. Also, it was difficult to perform real-time translation of voices and recognition of emotions simultaneously. For this reason, in communication among multiple languages, there was a problem that smooth conversations including emotions could not be realized.

[1450] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1451] In this invention, the server includes means for acquiring voice data from a user, means for transmitting the acquired voice data to the server, means for the server to convert the voice data into text data, means for the server to detect the language of the text data, means for the server to translate the text data into a target language, means for the server to reflect emotion information in the translated text data, means for the server to convert the translated text data with emotion information reflected into voice data, means for transmitting the translated voice data to the user's terminal, and means for the terminal to play back the translated voice data. Thereby, real-time voice translation reflecting emotions among multiple languages becomes possible.

[1452] The "user" refers to a person who inputs voice data using the system.

[1453] "Audio data" refers to digital audio signals acquired through devices such as microphones.

[1454] A "terminal" is a device used by a user to input or play back received audio data, and includes PCs, smartphones, and other similar devices.

[1455] A "server" refers to a remote computer system that processes audio data transmitted from a terminal and performs tasks such as translation and emotion recognition.

[1456] "Text data" refers to character data converted from speech data by a speech recognition engine.

[1457] "Language detection" refers to the process of identifying the language of acquired text data.

[1458] "Target language" refers to the target language after translation.

[1459] "Emotional information" refers to information about the user's emotions identified from voice data, including emotions such as joy, sadness, and excitement.

[1460] "Translated text data" refers to character data that has been translated into the target language.

[1461] "Speech synthesis" refers to the process of generating speech data from text data.

[1462] A "speech recognition engine" refers to software or an API that converts speech data into text data.

[1463] A "speech synthesis engine" refers to software or an API that converts text data into speech data.

[1464] An "emotion recognition engine" refers to software or an API that identifies a user's emotions from voice data.

[1465] This invention relates to a system for enabling smooth and emotionally charged communication between multiple languages ​​in real-world and web meetings. The system has the function of taking voice input from a user, translating it into another language in real time, and outputting it while reflecting the user's emotions.

[1466] The main components of the system are the user's terminal, the server, and the external API, and it functions as follows:

[1467] 1. Acquisition of voice input

[1468] The user inputs voice using the microphone on a device such as a PC or smartphone. The device acquires this voice data as a digital signal and temporarily stores it. For example, user A says, "Hello, nice to meet you."

[1469] 2. Sending audio data to the server

[1470] The terminal converts the acquired audio data into a WAV file and sends it to the server. The server receives the data as an HTTP request and stores the audio data internally for further processing.

[1471] 3. Perform speech recognition

[1472] The server sends the received audio data to a cloud-based speech recognition API (for example, a common speech recognition engine) to convert the audio into text data. In this process, the speech recognition engine analyzes the audio signal and generates the corresponding text, "Hello, nice to meet you."

[1473] 4. Performing emotion recognition

[1474] The server further sends the voice data to an emotion recognition engine (e.g., a general emotion recognition engine) to identify the user's emotion. The emotion recognition engine analyzes the tone, pitch, etc. of the voice to identify emotions (such as joy, sadness, excitement, etc.). Here, the emotion of "joy" is identified.

[1475] 5. Detection of Language

[1476] The server sends the text data obtained by voice recognition to a general language detection API to identify the language being used. This API detects that the text data is in Japanese.

[1477] 6. Execution of Translation

[1478] The server sends the detected Japanese text to a cloud-based translation API (e.g., a general translation engine) to translate it into the target language (e.g., Chinese). For this reason, the text "こんにちは、はじめまして" is translated as "你好,很高兴认识你".

[1479] 7. Reflection of Emotion

[1480] The server reflects the identified emotion (in this case, "joy") in the translated text. Specifically, emotion information for reflecting the intonation, strength, and speed of the voice is added. This enables the generation of voice that reflects the user's emotion.

[1481] 8. Execution of Voice Synthesis

[1482] The server sends the translated text data containing emotion information to a voice synthesis engine (e.g., a general voice synthesis engine) to convert it into voice data that reflects the emotion information. The voice synthesis engine generates the translated text "你好,很高兴认识你" in Chinese pronunciation and as voice data that reflects the emotion of "joy".

[1483] 9. Transmission of Voice Data to the Terminal

[1484] The server encodes the generated voice data into the MP3 format and transmits it to the user's terminal. The server encrypts the data using HTTPS and transfers it securely.

[1485] 10. Output of translated voice

[1486] The terminal receives the voice data sent from the server. The terminal outputs this voice data through a playback device (speaker or headphones) and provides the translated voice reflecting the emotion to User B. In this way, the voice of User A, "Hello, nice to meet you," is transmitted to User B as the Chinese voice "你好,很高兴认识你" with a happy emotion.

[1487] Specific example

[1488] For example, assume that User A (a Japanese speaker) is talking to User B (a Chinese speaker) in a real meeting. By using this system, the speech of User A is quickly translated into Chinese and transmitted to User B in a form that reflects the emotion. In this way, smooth and emotional communication between multiple languages is realized.

[1489] Example of prompt sentence

[1490] "User A who speaks Japanese is talking to User B who speaks Chinese. When User A says 'Hello, nice to meet you,' please translate that voice into Chinese and output it as a voice reflecting the emotion (happiness)."

[1491] As described above, the system of the present invention can smoothly conduct user communication by providing real-time and emotion-reflecting translation in communication between multiple languages.

[1492] The flow of specific processing in Example 2 will be described with reference to FIG. 13.

[1493] Step 1:

[1494] The user speaks into the microphone of their PC or smartphone. The device's microphone then captures the user's voice as a digital signal. The input is the user's voice, and the output is digital audio data.

[1495] Step 2:

[1496] The device temporarily stores the acquired audio data in its internal memory and converts the data into a WAV file. The input is digital audio data, and the output is a WAV audio file. The device then sends this audio data file to the server as an HTTP request.

[1497] Step 3:

[1498] The server sends the received audio data to a cloud-based speech recognition API. The speech recognition API analyzes the audio signal and converts it into corresponding text data. The input is an audio file in WAV format, and the output is the text data "Hello, nice to meet you".

[1499] Step 4:

[1500] The server sends voice data to the emotion recognition engine to identify the user's emotions. The emotion recognition engine analyzes the tone and pitch of the voice to recognize emotions (joy, sadness, excitement, etc.). The input is voice data, and the output is emotion information, "joy."

[1501] Step 5:

[1502] The server sends the text data obtained through speech recognition to a language detection API to identify the language being used. The API analyzes the linguistic characteristics of the text and identifies the corresponding language. The input is text data, and the output is language information, "Japanese".

[1503] Step 6:

[1504] The server sends the detected Japanese text data to a translation API for translation into the target language. The translation engine translates "こんにちは、はじめまして" to "你好,很高兴认识你". The input is Japanese text data, and the output is Chinese translated text data.

[1505] Step 7:

[1506] The server reflects the identified emotion "happiness" in the translated text. Specifically, emotion information for adjusting the intonation, strength, and speed of the voice is added. The input is Chinese translated text data and emotion information, and the output is translated text data reflecting the emotion.

[1507] Step 8:

[1508] The server sends the translated text data containing emotion information to a text-to-speech engine for conversion into audio data. The text-to-speech engine generates the translated text "你好,很高兴认识你" in Chinese pronunciation and as audio data reflecting the emotion of "happiness". The input is translated text data reflecting the emotion, and the output is audio data.

[1509] Step 9:

[1510] The server encodes the generated audio data into the MP3 format and prepares to send it to the user's terminal. This audio data is sent to the terminal as an HTTP response. The input is the generated audio data, and the output is audio data in the MP3 format.

[1511] Step 10:

[1512] The terminal receives the voice data sent from the server. The terminal plays back this voice data through a playback device (such as a speaker or headphones) and provides the translated voice to the user. The input is the voice data received from the server, and the output is the voice to be played back. In this way, the voice of User A, "Hello, nice to meet you for the first time," is transmitted to User B as the Chinese voice "Ni hao, hen gao xing ren shi ni" that reflects the emotion.

[1513] (Application Example 2)

[1514] Next, Application Example 2 will be described. In the following description, the data processing device 12 is referred to as the "server," and the robot 414 is referred to as the "terminal."

[1515] With the increasing number of multinational workers in the factory, it is necessary to convey work instructions quickly and accurately. However, there is a language barrier, which may cause misunderstandings and confusion. Also, it is important to accurately convey the urgency and emotion of the instructor, but this is also a difficult issue. Therefore, in addition to real-time translation between multiple languages, a method for conveying work instructions considering emotions is required.

[1516] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1517] In this invention, the server includes means for acquiring voice data from a user, means for transmitting the acquired voice data to the server, and means for inputting the voice data received by the server into an emotion engine, identifying the emotion, and reflecting the emotion information in the translated voice data. Thereby, it becomes possible to accurately translate the voice instructions of the work instructor into multiple languages and further convey them to the factory robot while reflecting the emotion of the instructor.

[1518] The "user" refers to a person who inputs voice data using the system.

[1519] The "voice data" refers to data that records the voice emitted by the user in digital form.

[1520] A "server" refers to a computer system that processes and converts audio data.

[1521] "Text data" refers to the character information that has been converted from audio data by a speech recognition engine.

[1522] "Means for detecting language" refers to algorithms or programs that identify the language of text data.

[1523] The "target language" refers to the language that is the subject of the translation.

[1524] An "emotion engine" refers to software or an algorithm that identifies emotions from audio data and outputs emotional information.

[1525] "Emotional information" refers to data that digitally represents emotions (e.g., joy, sadness, urgency, etc.) identified from the user's voice.

[1526] "Multilingual" refers to multiple different languages.

[1527] "Factory robots" refer to automated equipment or devices that perform tasks within a factory.

[1528] "Work instructions" refer to specific instructions that should be given to workers and equipment within a factory.

[1529] A "translation API" refers to an external software interface that automatically translates text data into another language.

[1530] A "speech recognition API" refers to an external software interface that converts speech data into text data.

[1531] A "speech synthesis engine" refers to software or an algorithm that converts text data into speech data.

[1532] A "prompt" refers to a text-based instruction given to a generative AI model.

[1533] This invention relates to a system for transmitting multilingual and emotionally reflective voice instructions to factory robots within a factory. This system translates the voice of the person giving the instructions into multiple languages ​​in real time, recognizes the person's emotions, and reflects them in the translation result.

[1534] Hardware and software

[1535] This system uses the following hardware and software:

[1536] 1. Hardware

[1537] Smart glasses: A device for inputting voice.

[1538] Server: A central device used for data processing and calculations.

[1539] Robot control terminal: A device that plays voice data and transmits instructions to the robot.

[1540] 2. Software

[1541] Speech-to-Text APIs: To convert speech data into text data, common cloud-based speech recognition APIs are used. An example is the Google Cloud Speech-to-Text API.

[1542] Emotion Recognition API: To identify emotions from audio data, an external emotion recognition API is used. IBM Watson Tone Analyzer is an example of this.

[1543] Language detection API: To detect the language of text data, use an external language detection API. For example, use the Microsoft Azure Language Recognition API.

[1544] Translation API: Use an external translation API to translate text data into the target language. For example, the Google Cloud Translation API supports this.

[1545] Speech synthesis APIs: Speech synthesis APIs are used to convert translated text data into speech data that reflects emotion. Amazon Polly is an example of this.

[1546] Data processing and calculation

[1547] 1. Acquisition and transmission of audio data

[1548] The user uses the microphone on the smart glasses to input voice data. The acquired voice data is temporarily stored on the smart glasses' terminal and then sent to the server. At this time, the voice data is saved in WAV format.

[1549] 2. Speech Recognition Processing

[1550] The server uses the Google Cloud Speech-to-Text API to convert the received audio data into text data. The converted text data is then processed further.

[1551] 3. Emotion recognition processing

[1552] The server uses IBM Watson Tone Analyzer to identify the user's emotions from the voice data. The identified emotion information is then added to the text data.

[1553] 4. Language detection and translation

[1554] The Microsoft Azure Language Recognition API is used to detect the language, and the Google Cloud Translation API is used to translate it into the target language.

[1555] 5. Reflecting emotions in the translation results

[1556] The translated text data incorporates sentiment information identified by IBM Watson Tone Analyzer. Based on this sentiment information, the intonation and speed of the speech are adjusted.

[1557] 6. Speech synthesis and terminal transmission

[1558] Amazon Polly is used to convert the audio data into emotionally reflective audio, which is then sent to the robot control terminal. Finally, the terminal plays this audio data.

[1559] Specific example

[1560] For example, if a factory supervisor gives the instruction to "hurry up and start assembling the next product batch," this system will perform the following actions:

[1561] 1. The audio input is acquired in WAV format and sent to the server.

[1562] 2. Use the Google Cloud Speech-to-Text API to generate the text data "Hurry up and start assembling the next product batch."

[1563] 3. Use IBM Watson Tone Analyzer to identify "urgency" and emotion.

[1564] 4. Detect the language of the text data as Japanese using the Microsoft Azure Language Recognition API.

[1565] 5. Use the Google Cloud Translation API to translate the text data into English as "Hurry up and start assembling the next product batch".

[1566] 6. The "urgent" emotion identified by IBM Watson Tone Analyzer is reflected in the voice, and then converted into voice data using Amazon Polly.

[1567] 7. Finally, the translated audio data is sent to the robot control terminal and played back.

[1568] Example of a prompt

[1569] The user said via voice input, "Hurry up and start assembling the next product batch." Translate this voice data into English in real time and convey the instruction to the robot, reflecting an urgent emotion. The translated voice should be "Hurry up and start assembling the next product batch," and the voice should be synthesized to reflect an urgent emotion.

[1570] This system will enable factory supervisors to efficiently communicate multilingual and emotionally resonant voice instructions to factory robots.

[1571] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1572] Step 1:

[1573] The user uses the microphone on their smart glasses to input voice commands. For example, the user might say, "Hurry up and start assembling the next product batch." This voice data is the input. The voice data is temporarily stored in WAV format on the smart glasses' terminal. This terminal then sends the data to the server.

[1574] Step 2:

[1575] The server receives audio data from the smart glasses and inputs it into the Google Cloud Speech-to-Text API, converting the audio data into text data. The specific operations performed here are audio signal processing and string parsing. The converted text data is then output.

[1576] Step 3:

[1577] The server inputs text data into the IBM Watson Tone Analyzer for sentiment recognition. The input is text data, and the output is sentiment information (e.g., "urgent"). The specific operation is text-based sentiment analysis.

[1578] Step 4:

[1579] The server inputs text data into the Microsoft Azure Language Recognition API to detect the language. The input is text data, and the output is language information (e.g., Japanese). The specific operation is language profiling.

[1580] Step 5:

[1581] The server inputs text data into the Google Cloud Translation API and translates it into the target language (e.g., English). The output is the translated text data (e.g., "Hurry up and start assembling the next product batch"). The specific operation is a text translation process.

[1582] Step 6:

[1583] The server inputs translated text data and sentiment information into Amazon Polly, which then synthesizes it into speech data. The input is translated text data and sentiment information, and the output is speech data that reflects the sentiment. The specific operation is speech synthesis.

[1584] Step 7:

[1585] The server sends the generated voice data to the robot control terminal. The input is voice data that reflects emotions, and the server performs the specific action of transferring the data to the terminal.

[1586] Step 8:

[1587] The terminal receives audio data sent from the server and plays this audio data. The output is a translated voice instruction that reflects emotion. The specific operation involves decoding the audio data and playing it back through the speaker.

[1588] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1589] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1590] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1591] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1592] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1593] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1594] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1595] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1596] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1597] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1598] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1599] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1600] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1601] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1602] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1603] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1604] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1605] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1606] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1607] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1608] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[1609] The following is further disclosed regarding the embodiments described above.

[1610] (Claim 1)

[1611] A means of obtaining voice data from the user,

[1612] A means of sending the acquired audio data to the server,

[1613] A means by which the server converts audio data into text data,

[1614] A means for the server to detect the language of the text data,

[1615] A means for the server to translate text data into the target language,

[1616] A means by which the server converts translated text data into audio data,

[1617] A means of sending translated audio data to the user's device,

[1618] A system that includes means for a device to play translated audio data.

[1619] (Claim 2)

[1620] The system according to claim 1, wherein an external speech recognition API is used to perform speech recognition processing on the aforementioned audio data.

[1621] (Claim 3)

[1622] The system according to claim 1, which uses an external translation API to perform language detection and translation processing on the text data.

[1623] "Example 1"

[1624] (Claim 1)

[1625] A means of obtaining voice data from the user,

[1626] A means of sending the acquired audio data to the server,

[1627] A means by which the server converts audio data into text data,

[1628] A means for the server to detect the language of the text data,

[1629] A means for the server to translate text data into the target language,

[1630] A means by which the server converts translated text data into audio data,

[1631] A means of sending translated audio data to the user's device,

[1632] The device provides a means for playing translated audio data,

[1633] A means by which the server converts the format of audio data using an external audio processing API,

[1634] A system including means for outputting audio data received by a terminal to a predetermined playback device.

[1635] (Claim 2)

[1636] The system according to claim 1, wherein an external speech processing API is used to perform speech recognition processing and speech synthesis processing on the aforementioned speech data.

[1637] (Claim 3)

[1638] The system according to claim 1, which uses an external language processing API to perform language detection and translation processing on the text data.

[1639] "Application Example 1"

[1640] (Claim 1)

[1641] A means of obtaining voice data from the user,

[1642] A means of sending the acquired audio data to the server,

[1643] A means by which the server converts audio data into text data,

[1644] A means for the server to detect the language of the text data,

[1645] A means for the server to translate text data into the target language,

[1646] A means by which the server converts translated text data into audio data,

[1647] A means of sending translated audio data to the user's device,

[1648] The device provides a means for playing translated audio data,

[1649] A means of translating passenger voices into other languages ​​in real time and outputting them as audio, integrated into the vehicle system.

[1650] A system that includes this.

[1651] (Claim 2)

[1652] The system according to claim 1, wherein an external speech recognition API is used to perform speech recognition processing on the aforementioned audio data.

[1653] (Claim 3)

[1654] The system according to claim 1, which uses an external translation API to perform language detection and translation processing on the text data.

[1655] "Example 2 of combining an emotion engine"

[1656] (Claim 1)

[1657] A means of obtaining voice data from the user,

[1658] A means of sending the acquired audio data to the server,

[1659] A means by which the server converts audio data into text data,

[1660] A means for the server to detect the language of the text data,

[1661] A means for the server to translate text data into the target language,

[1662] A means for the server to reflect emotional information in translated text data,

[1663] A means by which a server converts translated text data that reflects emotional information into audio data,

[1664] A means of sending translated audio data to the user's device,

[1665] A system that includes means for a device to play translated audio data.

[1666] (Claim 2)

[1667] The system according to claim 1, wherein an external speech recognition API is used to perform speech recognition processing on the aforementioned audio data.

[1668] (Claim 3)

[1669] The system according to claim 1, which uses an external translation API to perform language detection and translation processing on the text data.

[1670] "Application example 2 when combining with an emotional engine"

[1671] (Claim 1)

[1672] A means of obtaining voice data from the user,

[1673] A means of sending the acquired audio data to the server,

[1674] A means by which the server converts audio data into text data,

[1675] A means for the server to detect the language of the text data,

[1676] A means for the server to translate text data into the target language,

[1677] A means by which the server converts translated text data into audio data,

[1678] A means of sending translated audio data to the user's device,

[1679] The device provides a means for playing translated audio data,

[1680] A means for inputting the audio data received by the server into an emotion engine, identifying emotions, and reflecting emotional information in the translated audio data,

[1681] A system that includes means for transmitting voice instructions from factory workers to factory robots as multilingual voice data that reflects emotions.

[1682] (Claim 2)

[1683] The system according to claim 1, wherein an external speech recognition API is used to perform speech recognition processing on the aforementioned audio data.

[1684] (Claim 3)

[1685] The system according to claim 1, which uses an external translation API to perform language detection and translation processing on the text data. [Explanation of symbols]

[1686] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of obtaining voice data from the user, A means of sending the acquired audio data to the server, A means for the server to convert audio data into text data, A means for the server to detect the language of the text data, A means for the server to translate text data into the target language, A means by which the server converts translated text data into audio data, A means of sending translated audio data to the user's device, A system that includes means for a device to play translated audio data.

2. The system according to claim 1, wherein an external speech recognition API is used to perform speech recognition processing on the aforementioned audio data.

3. The system according to claim 1, which uses an external translation API to perform language detection and translation processing on the text data.

Images