System

Abstract

To provide a system for quickly and efficiently removing a language barrier by recognizing a voice and an action of a user in real time, converting them into another language and uttering them.SOLUTION: The system comprises speech recognition means, motion recognition means, means for analyzing the speech and motion data and converting them into text data, means for translating the text data into another language, means for converting the translated data into speech data, and control means for integrating said means.SELECTED DRAWING: None

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Learning to speak a foreign language requires a lot of time and effort, and it is particularly difficult to master it in a short period of time. Learning a new language to cope with sudden overseas business trips or job transfers is not practical, and people who have lost their vocal cords due to accidents or illness have very limited means of communication. Under these circumstances, there is a need for a method to efficiently and quickly overcome language barriers. The present invention aims to provide a means to address this issue. [Means for solving the problem]

[0005] To solve the above-mentioned problems, the present invention provides a system including a voice recognition unit, a motion recognition unit, a unit for analyzing voice data and motion data, a unit for translating the analyzed data into another language, a unit for converting the translated data into speech, and a control unit for integrating the above units. The system learns the user's specific vocal cord and mouth movements and can generate and play back voice data in real time. The system also provides an error notification if a problem occurs in capturing the voice or motion data, and also provides a means for accommodating different dialects and accents within the same language. This makes it possible to quickly and efficiently overcome language barriers, even during sudden overseas business trips or other situations where communication is difficult.

[0006] A "voice recognition means" is a device or software that captures voice, analyzes the voice, and converts it into text data.

[0007] The "motion recognition means" is a device or software that uses a camera or sensor to capture the user's physical movements, particularly mouth movements, and analyzes them as data.

[0008] "Means for analyzing voice and motion data" are the techniques and algorithms used to process captured voice and motion data and convert it into identifiable text data.

[0009] A "translating means" is software or algorithms for converting the analyzed text data into another language.

[0010] "Means for converting into speech" refers to technology or devices that convert translated text data into speech data and generate natural speech.

[0011] The "control means for integration" is software or hardware that links various means and operates the entire system in a unified manner.

[0012] "Means for learning the movement of a user's vocal cords" refers to technology or algorithms that analyze the characteristics of a specific user's voice and use them as a model for learning.

[0013] A "means for learning mouth movements" is a technology or algorithm that analyzes the characteristics of a specific user's mouth movements and uses them as a model for learning.

[0014] "Means for generating and playing back in real time" refers to technology or devices for instantly processing data and playing back the generated audio from speakers, etc.

[0015] An "error notification means" is software or a system that notifies the user when a problem occurs in capturing or analyzing audio or motion data.

[0016] "Means to accommodate dialects and accents" refers to technologies and algorithms that analyze and adapt to differences in pronunciation due to regional and individual differences within the same language. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention is a system in which a user, a terminal, and a server work together to recognize voices and actions in real time, convert them into another language, and then speak the voices. Specific embodiments of the present invention are described below.

[0039] System Overview

[0040] This system basically uses a dedicated application installed on a smartphone or tablet device (hereinafter referred to as "device") and connects to a server located in the cloud. Using this application, users can communicate with speakers of other languages ​​without having to learn the foreign language.

[0041] Initial Setup and User Registration

[0042] 1. User: First, install the app on your smartphone and launch it.

[0043] 2. Device: When the app launches, an initial setup screen appears, prompting the user to enter necessary information (such as name and desired language).

[0044] 3. Terminal: Sends the information entered by the user to the server.

[0045] 4. Server: Stores the received user information in a database and generates an identifier corresponding to each individual user.

[0046] Voice and movement learning phase

[0047] 1. User: Follow the instructions of the app and speak a few short Japanese phrases.

[0048] 2. Device: A microphone captures audio and a camera records mouth movements.

[0049] 3. Terminal: Sends voice data and mouth movement data to the server in real time.

[0050] 4. Server: The AI ​​that generates speech and motion recognition analyzes the user's characteristics and learns a model for converting speech and motion data into text data.

[0051] Real-time speech recognition and translation

[0052] 1. User: After completing the setup, select the target language and speak in Japanese (e.g., "Hello, what's your name?").

[0053] 2. Device: The microphone captures the user's voice and the camera recognizes mouth movements.

[0054] 3. Terminal: Sends the captured voice and movement data to the server.

[0055] 4. Server: Analyzes the received data and converts it into Japanese text.

[0056] 5. Server: Translate the Japanese text into the target language (e.g., English).

[0057] 6. Server: Converts the translated text data into audio data and generates natural-sounding speech.

[0058] 7. Terminal: The generated foreign language speech is provided to the user and played back through the speaker.

[0059] Conversation assistance functions

[0060] 1. User: If you have lost your vocal cords, select mouth input mode.

[0061] 2. User: Imitate speech using mouth movements only (e.g., "please").

[0062] 3. Device: The camera captures mouth movements and sends the data to the server.

[0063] 4. Server: Analyzes the motion recognition data and converts it into Japanese text.

[0064] 5. Server: Translate the Japanese text into the target language.

[0065] 6. Server: Converts the translated text data into audio data and generates natural-sounding speech.

[0066] 7. Terminal: Plays back the generated foreign language speech and provides it to the user.

[0067] Specific examples

[0068] Example 1: Normal Japanese to English Conversation

[0069] 1. User: Open the app, select English mode and say "Hello, what's your name?"

[0070] 2. Device: Captures voice and movements and sends them to the server.

[0071] 3. Server: Translate this into "Hello, what is your name?" and generate it as voice data.

[0072] 4. Terminal: Plays the generated English audio and supports the conversation.

[0073] Example 2: Speech assistance for users who have lost their vocal cords

[0074] 1. User: Open the app and select mouth input mode.

[0075] 2. User: Imitate the word "please" using only mouth movements.

[0076] 3. Device: Captures the action and sends it to the server.

[0077] 4. Server: Translate this into "Please" and generate it as voice data.

[0078] 5. Terminal: Plays the generated English audio and supports the conversation.

[0079] The system of the present invention learns the user's voice characteristics and mouth movements to enable natural conversations that transcend language barriers. It also has a function that notifies the user of an error if a problem occurs with voice capture or translation, prompting the user to take appropriate action. This allows the system to efficiently overcome language barriers, even in situations where communication is difficult, such as sudden overseas business trips.

[0080] The processing flow will be explained below.

[0081] Step 1:

[0082] User: Install the dedicated app on your smartphone and launch it.

[0083] Step 2:

[0084] Terminal: Displays an initial setup screen and prompts the user to enter necessary information such as name and desired language.

[0085] Step 3:

[0086] User: Enter the required information on the initial setup screen and press the "Next" button after completing the entry.

[0087] Step 4:

[0088] Terminal: Sends the information entered by the user to the server.

[0089] Step 5:

[0090] Server: Stores the received user information in a database and generates a unique user identifier.

[0091] Step 6:

[0092] Terminal: Displays prompts to the user to begin the voice and movement learning phase.

[0093] Step 7:

[0094] User: Follow the app's instructions and speak a few short Japanese phrases.

[0095] Step 8:

[0096] Device: A microphone captures your voice and a camera records your mouth movements.

[0097] Step 9:

[0098] Terminal: Sends voice and movement data to the server in real time.

[0099] Step 10:

[0100] Server: Analyzes the received voice and movement data and learns the characteristics of the user's voice and movements.

[0101] Step 11:

[0102] Server: The generated analysis model is linked to the user's account and saved.

[0103] Step 12:

[0104] User: After the initial setup and learning phase is complete, select your target language on the app's main screen.

[0105] Step 13:

[0106] User: Speak in Japanese (e.g. "Hello, what's your name?").

[0107] Step 14:

[0108] Device: A microphone captures the user's voice and a camera recognizes mouth movements.

[0109] Step 15:

[0110] Terminal: Transmits captured audio and motion data to the server in real time.

[0111] Step 16:

[0112] Server: Analyzes the received voice data and converts it into Japanese text (e.g., "Hello, what's your name?").

[0113] Step 17:

[0114] Server: Translate Japanese text into the target language (e.g., English) (e.g., "Hello, what is your name?").

[0115] Step 18:

[0116] Server: Converts the translated text data into audio data and generates natural-sounding speech.

[0117] Step 19:

[0118] Terminal: Receives the generated foreign language audio and plays it back through the speaker.

[0119] Step 20:

[0120] User: Listen to the foreign language audio and continue communication.

[0121] Step 21:

[0122] Terminal: If an error is detected, an error message is displayed to the user.

[0123] Step 22:

[0124] User: Follow the error message and make any necessary corrections or try again.

[0125] Step 23:

[0126] User: If you have lost your vocal cords, select mouth input mode.

[0127] Step 24:

[0128] User: Mimic speech using mouth movements only (e.g., "please").

[0129] Step 25:

[0130] Device: The camera captures mouth movements and sends the data to the server.

[0131] Step 26:

[0132] Server: Analyzes the action recognition data and converts it into Japanese text (e.g., "Onegaishimasu").

[0133] Step 27:

[0134] Server: Translate Japanese text into the target language (e.g., English) (e.g., "Please").

[0135] Step 28:

[0136] Server: Converts the translated text data into audio data and generates natural-sounding speech.

[0137] Step 29:

[0138] Terminal: Receives the generated foreign language audio and plays it back through the speaker.

[0139] Step 30:

[0140] User: Listen to the foreign language audio and continue communication.

[0141] Example 1

[0142] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0143] In today's globalized society, communication between people who speak different languages ​​is becoming increasingly important. However, the existence of language barriers makes smooth communication difficult. Furthermore, people with certain physical limitations (e.g., those who have lost their vocal cords) cannot use standard speech recognition technology, making communication even more difficult. To solve these challenges, there is a need for a system that can recognize voices and movements in real time, convert them into speech in other languages, and then speak them.

[0144] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0145] In this invention, the server includes a voice recognition unit, a motion recognition unit, a unit for analyzing voice data and motion data, a unit for translating the analyzed data into another language, and a unit for converting the translated data into voice. This allows the user to perform initial setup, and a unit for saving and managing the setup information, a unit for capturing voice and motion and transmitting them to the server in real time, and a unit for the server to provide the user with the analysis results, enabling smooth communication between people who speak different languages. Furthermore, even people with certain physical limitations can have natural conversations using mouth movements.

[0146] "Speech recognition means" is a general term for hardware and software for capturing speech as digital data and analyzing its content.

[0147] "Movement recognition means" is a general term for hardware and software for capturing the user's body movements, particularly mouth movements, and analyzing those movements.

[0148] The "means for analyzing voice data and movement data" is a mechanism for analyzing the captured voice data and movement data using an algorithm and converting it into text data.

[0149] "Means for translating analyzed data into another language" is a general term for hardware and software for translating analyzed text data into another specified language.

[0150] "Means for converting translated data into speech" is a general term for hardware and software for converting translated text data into speech data and generating natural-sounding speech.

[0151] The "control means for integrating the above means" is a control system for integrally operating the voice recognition means, the motion recognition means, the analysis means, the translation means, and the voice conversion means.

[0152] "Means for users to perform initial setup and save and manage setup information" is a general term for the interface and database that allows users to perform initial setup of the system and save and manage that information.

[0153] "Means for capturing voice and movement and transmitting it to a server in real time" is a general term for software and network communication means for acquiring voice and movement data, compressing it, and transmitting it to a server.

[0154] "Means by which the server provides the analysis results to the user" refers to a mechanism by which the server returns the analysis and translation results to the user's terminal and displays or plays them aloud.

[0155] The present invention is a system in which a user, a terminal, and a server work together to recognize voices and actions in real time, convert them into another language, and then speak the voices. Specific embodiments of the present invention are described below.

[0156] System Overview

[0157] This system basically uses a dedicated application installed on a smartphone or tablet device (hereinafter referred to as "device") and connects to a server located in the cloud. Using this application, users can communicate with speakers of other languages ​​without having to learn the foreign language.

[0158] Initial Setup and User Registration

[0159] 1. User: Installs the app on a smartphone and launches it. The app can be downloaded from the GOOGLE PLAY Store or the App Store.

[0160] 2. Device: When the app launches, an initial setup screen appears, prompting the user to enter information such as their name, preferred language, and email address. Once the information is entered, the device encrypts and sends it to the server.

[0161] 3. Server: Stores the received user information in a database and generates a unique identifier that is sent back to the device for future operations and data management.

[0162] Voice and movement learning phase

[0163] 1. User: Follow the instructions of the app to speak a few short Japanese phrases, such as "hello" and "thank you."

[0164] 2. Device: A microphone captures audio and a camera records lip movements. Audio and video are synchronized to collect precise data.

[0165] 3. Terminal: Collected voice and movement data is compressed in real time and sent to the server.

[0166] 4. Server: Analyzes and learns voice and movement features using generative AI models, and optimizes the model to convert this data into text.

[0167] Real-time speech recognition and translation

[0168] 1. User: After completing the setup, select the target language and speak in Japanese. For example, the user selects "English mode" and speaks "Hello, what is your name?"

[0169] 2. Device: The microphone captures the voice and the camera recognizes the mouth movements. This data is sent to the server in real time.

[0170] 3. Server: Analyzes the received voice and motion data and converts it into Japanese text. Then, translates the Japanese text into the target language (e.g., English). For example, "Hello, what is your name?" is converted to "Hello, what is your name?"

[0171] 4. Server: Converts the translated text data into audio data and generates natural-sounding speech.

[0172] 5. Terminal: The generated foreign language speech is provided to the user and played back through the speaker.

[0173] Conversation assistance functions

[0174] 1. User: If you have lost your vocal cords, select "Mouth Recognition Mode" from the app settings.

[0175] 2. User: Imitate speech using only mouth movements, for example, recreating the mouth movements for "please."

[0176] 3. Device: The camera captures mouth movements and sends the data to the server.

[0177] 4. Server: Analyzes the motion recognition data and converts it into Japanese text. Next, translates "onegaishimasu" into "Please."

[0178] 5. Server: Converts the translated text data into audio data and generates natural-sounding speech.

[0179] 6. Terminal: Plays back the generated foreign language speech and provides it to the user.

[0180] Examples and prompts

[0181] Example 1: Normal Japanese to English Conversation

[0182] 1. User: Open the app, select English mode and say "Hello, what's your name?"

[0183] 2. Device: Captures voice and movements and sends them to the server.

[0184] 3. Server: Translate this into "Hello, what is your name?" and generate it as voice data.

[0185] 4. Terminal: Plays the generated English audio and supports the conversation.

[0186] Example 2: Speech assistance for users who have lost their vocal cords

[0187] 1. User: Open the app and select mouth input mode.

[0188] 2. User: Imitate the word "please" using only mouth movements.

[0189] 3. Device: Captures the action and sends it to the server.

[0190] 4. Server: Translate this into "Please" and generate it as voice data.

[0191] 5. Terminal: Plays the generated English audio and supports the conversation.

[0192] As an example of a prompt sentence, if you enter "Translate and speak the Japanese greeting into English," the Japanese "Konnichiwa" will be translated and spoken as the English "Hello."

[0193] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0194] Step 1:

[0195] User: Installs the app on their smartphone and launches it.

[0196] Input: Download the app file from Google(R) Play Store or App Store.

[0197] How it works: Press the app's install button, then tap the app icon to launch it after installation is complete.

[0198] Output: The initial setup screen is displayed on the terminal.

[0199] Step 2:

[0200] Device: An initial setup screen appears, prompting the user to enter information such as name, preferred language, and email address.

[0201] Input: The user enters their name, desired language, email address, etc.

[0202] Action: Enter the required information in the input form and press the "Submit" button.

[0203] Output: The input information is encrypted and sent from the device to the server.

[0204] Step 3:

[0205] Server: Stores the received user information in a database and generates and returns an identifier.

[0206] Input: User information data entered.

[0207] What it does: Connects to a database, stores the received data, and generates a unique identifier.

[0208] Output: The generated identifier is sent back to the terminal.

[0209] Step 4:

[0210] User: Follow the app's instructions and speak a few short Japanese phrases.

[0211] Input: A phrase to display on the app screen (e.g. "Hello", "Thank you").

[0212] Action: Reads the displayed text aloud.

[0213] Output: The user's voice and mouth movement data is generated.

[0214] Step 5:

[0215] Device: A microphone captures your voice and a camera records your mouth movements.

[0216] Input: The user's voice and mouth movements.

[0217] Movement: High-definition microphones and high-resolution cameras capture audio waveforms and movement data.

[0218] Output: Captured audio and movement data.

[0219] Step 6:

[0220] Terminal: The captured voice data and movement data are compressed in real time and sent to the server.

[0221] Input: Audio and motion data.

[0222] What it does: Compresses data and sends it to the server with low latency.

[0223] Output: Audio and movement data received by the server.

[0224] Step 7:

[0225] Server: Analyzes and learns voice and movement characteristics using a generative AI model.

[0226] Input: Received voice and movement data.

[0227] How it works: The generative AI model extracts and analyzes features, optimizing the model for conversion to text data.

[0228] Output: Optimized speech-to-motion translation model.

[0229] Step 8:

[0230] User: After completing the setup, select the target language and speak in Japanese.

[0231] Input: The target language selected on the app screen, the words you speak (e.g. "Hello, what's your name?").

[0232] Action: Select the target language and speak in Japanese.

[0233] Output: Spoken Japanese audio.

[0234] Step 9:

[0235] Device: A microphone captures the user's voice and a camera recognizes mouth movements.

[0236] Input: Spoken speech and mouth movements.

[0237] How it works: Captures audio waveforms in real time and records mouth movements with a camera.

[0238] Output: Captured audio and movement data.

[0239] Step 10:

[0240] Terminal: Sends captured audio and motion data to the server.

[0241] Input: Audio and motion data.

[0242] Operation: Real-time transmission of data.

[0243] Output: Audio and motion data received by the server.

[0244] Step 11:

[0245] Server: Analyzes the received voice data and movement data and converts it into Japanese text.

[0246] Input: Audio and motion data.

[0247] Actions: The generative AI model analyzes voice and actions and converts them into Japanese text data.

[0248] Output: Converted Japanese text data.

[0249] Step 12:

[0250] Server: Translates Japanese text into the target language.

[0251] Input: Japanese text.

[0252] Behavior: Translates to the specified target language (e.g., English).

[0253] Output: The translated text data (e.g. "Hello, what is your name?").

[0254] Step 13:

[0255] Server: Converts the translated text data into audio data and generates natural-sounding speech.

[0256] Input: Translated text data.

[0257] How it works: Generates natural-sounding speech using generative AI models.

[0258] Output: The generated audio data.

[0259] Step 14:

[0260] Terminal: Provides the generated foreign language speech to the user and plays it back through a speaker.

[0261] Input: The generated audio data.

[0262] Action: Plays audio through a playback device.

[0263] Output: Foreign language speech played through the speaker (e.g. "Hello, what is your name?").

[0264] Step 15:

[0265] User: If you have lost your vocal cords, select "Mouth Recognition Mode" from the app settings.

[0266] Input: Select "Mouth Recognition Mode" on the app's settings screen.

[0267] Behavior: Toggles selection mode.

[0268] Output: Selected mode.

[0269] Step 16:

[0270] User: Mimics speech using mouth movements only.

[0271] Input: Mouth movements (e.g. "please").

[0272] Action: Mimics mouth movements.

[0273] Output: Mouth movement data.

[0274] Step 17:

[0275] Device: The camera captures mouth movements and sends the data to the server.

[0276] Input: Mouth movement data.

[0277] Movement: A high-precision camera captures mouth movements and transmits the data.

[0278] Output: Lip movement data sent to the server.

[0279] Step 18:

[0280] Server: Analyzes the motion recognition data and converts it into Japanese text.

[0281] Input: Captured mouth movement data.

[0282] Movement: Movement recognition data is analyzed and converted into Japanese text.

[0283] Output: Japanese text (e.g. "Onegaishimasu").

[0284] Step 19:

[0285] Server: Translates Japanese text into the target language.

[0286] Input: Japanese text.

[0287] Behavior: Translates to the specified target language (e.g., English).

[0288] Output: The translated text data (e.g. "Please").

[0289] Step 20:

[0290] Server: Converts the translated text data into audio data and generates natural-sounding speech.

[0291] Input: Translated text data.

[0292] How it works: Generates natural-sounding speech using generative AI models.

[0293] Output: The generated audio data.

[0294] Step 21:

[0295] Terminal: Plays back the generated foreign language speech and provides it to the user.

[0296] Input: The generated audio data.

[0297] Action: Plays audio through a playback device.

[0298] Output: Foreign language voice (e.g. "Please") played through the speaker.

[0299] (Application example 1)

[0300] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0301] Conventional food delivery systems face a language barrier when customers who speak a foreign language place an order, making it difficult to accurately convey order information. This leads to frequent ordering errors and communication problems with customers, resulting in a decline in service quality. Furthermore, only a limited number of systems allow input using mouth movements in addition to voice input, making them inadequate for specific users. To solve these problems, a means was needed to translate and speak order details in real time across language barriers, enabling smooth communication.

[0302] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0303] In this invention, the server includes a voice recognition means, a motion recognition means, a means for analyzing the voice data and motion data, a means for providing the translated voice data as order information to the delivery person or the store, and a control means for integrating the above means. This enables accurate real-time transmission of order information between customers who speak a foreign language and the delivery person or the store, realizing smooth communication that transcends language barriers.

[0304] A "voice recognition means" is a device or technology that receives voice as input and converts it into digital data that can be analyzed.

[0305] "Movement recognition means" refers to a device or technology that captures a user's physical movements, particularly facial and mouth movements, using a sensor such as a camera, and converts them into digital data that can be analyzed.

[0306] "Means for analyzing voice data and movement data" refers to software or hardware that analyzes data based on input voice or movement and converts it into text data or other formats.

[0307] "Means for translating the analyzed data into another language" refers to software or algorithms for translating the analyzed text data into a target language.

[0308] "Means for converting translated data into audio" refers to a device or software that converts translated text data into audio data and outputs it as audio.

[0309] "Means for providing translated voice data as order information to delivery personnel and stores" refers to technology and devices that use translated voice data to transmit accurate order information to the delivery system.

[0310] "Control means" refers to software and hardware that integrates and manages the above means and controls the entire system to ensure smooth operation.

[0311] The present invention provides a system that recognizes speech and gestures in real time and translates them into other languages ​​for speech, which is particularly useful for food delivery services to assist customers who speak other languages ​​in placing orders.

[0312] System configuration

[0313] Hardware Configuration

[0314] The system includes the following hardware:

[0315] Device: Smartphone or tablet

[0316] Microphone: A device used to capture the user's voice.

[0317] Camera: A device for capturing the user's mouth movements.

[0318] Speaker: A device for playing the translated audio.

[0319] Software Configuration

[0320] The system uses the following software:

[0321] Speech Recognition Library: speech_recognition library

[0322] Action recognition library: OpenCV (cv2)

[0323] Translation API: Google Cloud Translation API

[0324] Speech synthesis API: gTTS (Google Text-to-Speech)

[0325] System Operation

[0326] Speech and gesture recognition

[0327] When a user speaks, the device's microphone captures the voice and the camera records the mouth movements. This information is converted into text data using a speech recognition library (speech_recognition) and a motion recognition library (OpenCV). The data captured by the speech recognition and motion recognition means is processed in real time.

[0328] Data analysis and translation

[0329] The server analyzes the voice and motion data in the cloud, converts the user's speech into text using the analysis method, and then translates the analyzed text into the target language using the Google Cloud Translation API.

[0330] Speech synthesis of translation data

[0331] The translated text data is converted into voice data using the gTTS (Google Text-to-Speech) API. The voice data generated using a voice synthesis tool is played back on the device's speaker. This allows users to smoothly place orders through food delivery services without having to learn a foreign language.

[0332] Specific examples

[0333] Order Scenarios

[0334] 1. User: Download the smartphone app and select the language you want to use (e.g., English).

[0335] 2. User: "Hello, I'd like one Margherita pizza please" in Japanese.

[0336] 3. Device: Captures voice and lip movements and sends the data to the server.

[0337] 4. Server: Translates the speech data to "Hello, one Margherita pizza please" and converts it to speech data.

[0338] 5. Terminal: Plays back the translated voice data and provides it to delivery personnel and stores.

[0339] Prompt Sentence Examples

[0340] "How can I develop a food delivery app that translates Japanese speech into English in real time and provides it in a way that delivery staff and restaurants can understand?"

[0341] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0342] Step 1:

[0343] Input: The user launches the app on their smartphone and selects the language they want to use (e.g., English).

[0344] Specific operation: The user selects the desired language on the app's settings screen and enters their information (name, address, desired language, etc.).

[0345] Output: Configuration information is sent from the device to the server.

[0346] Step 2:

[0347] Input: User speaks in Japanese (e.g., "Hello, I'd like one Margherita pizza please.").

[0348] What it does: The device's microphone captures the user's voice and the camera records their mouth movements.

[0349] Output: Audio and motion data are generated.

[0350] Step 3:

[0351] Input: Audio and motion data captured by the device.

[0352] Specific operation: Using the speech recognition library (speech_recognition), converts voice data into text data. It also analyzes mouth movement data recorded by a camera.

[0353] Output: Japanese text data is generated and sent to the server.

[0354] Step 4:

[0355] Input: Japanese text data received by the server.

[0356] What it does: It uses the Google Cloud Translation API in the cloud to translate text data into the desired language (e.g., English).

[0357] Output: The translated text data is generated.

[0358] Step 5:

[0359] Input: Translated text data.

[0360] What it does: Uses the gTTS (Google Text-to-Speech) API to convert the translated text data into audio data.

[0361] Output: The translated audio data is generated.

[0362] Step 6:

[0363] Input: Translated audio data.

[0364] Specific operation: The device's speaker plays the generated voice data, which is then provided to the delivery person or the store.

[0365] Output: The translated content is played back as audio, and accurate order information is communicated to the delivery person or store.

[0366] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0367] The present invention is a system that recognizes voice and actions in real time, translates them into other languages, and speaks in a way that reflects the user's emotions by linking the user, terminal, and server and combining them with an emotion engine. Specific embodiments of the present invention are described below.

[0368] System configuration

[0369] This system uses a dedicated application installed on a smartphone or tablet device (hereafter referred to as "device") and connects to a server located in the cloud. It also incorporates an emotion engine that analyzes the user's emotions from voice and movement data and reflects this in the translation results and speech generation. Using this app, users can communicate with speakers of other languages ​​without having to learn the language.

[0370] Initial Setup and User Registration

[0371] 1. User: First, install the app on your smartphone and launch it.

[0372] 2. Device: Displays the initial setup screen and prompts the user to enter necessary information such as their name, target language, and permission for sentiment analysis.

[0373] 3. Terminal: Sends the information entered by the user to the server.

[0374] 4. Server: Stores the received user information in a database and generates a unique user identifier.

[0375] Voice and movement learning phase

[0376] 1. User: Follow the instructions of the app and speak a few short Japanese phrases.

[0377] 2. Device: A microphone captures audio and a camera records mouth movements.

[0378] 3. Terminal: Sends voice data and mouth movement data to the server in real time.

[0379] 4. Server: The AI ​​that generates speech and motion recognition analyzes the user's characteristics and learns a model for converting speech and motion data into text data.

[0380] Emotion analysis using an emotion engine

[0381] 1. Server: Using the received voice data and movement data, the emotion engine analyzes the user's emotions.

[0382] 2. Server: Stores the analyzed emotion data along with the user's voice and movement characteristics.

[0383] Real-time speech recognition and translation

[0384] 1. User: After completing the setup, select the target language and speak in Japanese (e.g., "Hello, what's your name?").

[0385] 2. Device: The microphone captures audio and the camera recognizes mouth movements.

[0386] 3. Terminal: Transmits captured voice and movement data to the server in real time.

[0387] 4. Server: Analyzes the received voice data and converts it into Japanese text (e.g., "Hello, what's your name?").

[0388] 5. Server: Uses Japanese text and emotion data to translate into the target language (e.g., English) (e.g., "Hello, what is your name?").

[0389] 6. Server: Based on the translated text data and emotion data, it converts it into voice data and generates natural, emotionally-informed speech.

[0390] 7. Terminal: The generated foreign language speech is provided to the user and played back through the speaker.

[0391] Conversation assistance functions

[0392] 1. User: If you have lost your vocal cords, select mouth input mode.

[0393] 2. User: Imitate speech using mouth movements only (e.g., "please").

[0394] 3. Device: The camera captures mouth movements and sends the data to the server.

[0395] 4. Server: Analyzes the action recognition data and converts it into Japanese text (e.g., "Onegaishimasu").

[0396] 5. Server: Uses the Japanese text and emotion data to translate into the target language (e.g., English) (e.g., "Please").

[0397] 6. Server: Based on the translated text data and emotion data, it converts it into voice data and generates natural, emotionally-informed speech.

[0398] 7. Terminal: Receives the generated foreign language audio and plays it back through the speaker.

[0399] Specific examples

[0400] Example 1: Normal Japanese to English Conversation

[0401] 1. User: Open the app, select English mode and say "Hello, what's your name?"

[0402] 2. Device: Captures voice and movements and sends them to the server.

[0403] 3. Server: Translate this to "Hello, what is your name?" and incorporate sentiment data.

[0404] 4. Server: Converts the translation results into audio data and generates natural speech.

[0405] 5. Terminal: Plays the generated English audio and supports the conversation.

[0406] Example 2: Speech assistance for users who have lost their vocal cords

[0407] 1. User: Open the app and select mouth input mode.

[0408] 2. User: Imitate the word "please" using only mouth movements.

[0409] 3. Device: Captures the action and sends it to the server.

[0410] 4. Server: Translate this to "Please" and incorporate emotional data.

[0411] 5. Server: Converts the translation results into audio data and generates natural speech.

[0412] 6. Terminal: Plays the generated English audio and supports conversation.

[0413] In this way, the system of the present invention learns the user's voice characteristics and mouth movements and analyzes the user's emotions, enabling more natural and emotional communication. This not only effectively removes language barriers, but also provides a realistic conversation experience that reflects the user's emotions.

[0414] The processing flow will be explained below.

[0415] Step 1:

[0416] User: Install the dedicated app on your smartphone and launch it.

[0417] Step 2:

[0418] Device: Displays an initial setup screen and prompts the user to enter required information such as name, target language, and permission for sentiment analysis.

[0419] Step 3:

[0420] User: Enter the required information on the initial setup screen and press the "Next" button after completing the entry.

[0421] Step 4:

[0422] Terminal: Sends the information entered by the user to the server.

[0423] Step 5:

[0424] Server: Stores the received user information in a database and generates a unique user identifier.

[0425] Step 6:

[0426] Terminal: Displays prompts to the user to begin the voice and movement learning phase.

[0427] Step 7:

[0428] User: Follow the app's instructions and speak a few short Japanese phrases.

[0429] Step 8:

[0430] Device: A microphone captures your voice and a camera records your mouth movements.

[0431] Step 9:

[0432] Terminal: Sends voice and movement data to the server in real time.

[0433] Step 10:

[0434] Server: Analyzes the received voice and movement data and learns the characteristics of the user's voice and movements.

[0435] Step 11:

[0436] Server: The generated analysis model is linked to the user's account and saved.

[0437] Step 12:

[0438] Server: Analyzes the user's emotions using an emotion engine based on voice data and movement data.

[0439] Step 13:

[0440] Server: Stores the analyzed emotion data along with the user's voice and movement characteristics.

[0441] Step 14:

[0442] User: After completing the initial setup and learning phase, select the desired language on the app's main screen.

[0443] Step 15:

[0444] User: Speak in Japanese (e.g. "Hello, what's your name?").

[0445] Step 16:

[0446] Device: A microphone captures audio and a camera recognizes mouth movements.

[0447] Step 17:

[0448] Terminal: Transmits captured audio and motion data to the server in real time.

[0449] Step 18:

[0450] Server: Analyzes the received voice data and converts it into Japanese text (e.g., "Hello, what's your name?").

[0451] Step 19:

[0452] Server: Uses Japanese text and emotion data to translate into the target language (e.g., English) (e.g., "Hello, what is your name?").

[0453] Step 20:

[0454] Server: Based on the translated text data and emotion data, converts it into voice data and generates natural speech that reflects emotion.

[0455] Step 21:

[0456] Terminal: Receives the generated foreign language audio and plays it back through the speaker.

[0457] Step 22:

[0458] User: Listen to the foreign language audio and continue communicating with the other person.

[0459] Step 23:

[0460] Terminal: If an error is detected, an error message is displayed to the user.

[0461] Step 24:

[0462] User: Follow the error message and make any necessary corrections or try again.

[0463] Step 25:

[0464] User: If you have lost your vocal cords, select mouth input mode.

[0465] Step 26:

[0466] User: Mimic speech using mouth movements only (e.g., "please").

[0467] Step 27:

[0468] Device: The camera captures mouth movements and sends the data to the server.

[0469] Step 28:

[0470] Server: Analyzes the action recognition data and converts it into Japanese text (e.g., "Onegaishimasu").

[0471] Step 29:

[0472] Server: Uses Japanese text and emotion data to translate into the target language (e.g., English) (e.g., "Please").

[0473] Step 30:

[0474] Server: Based on the translated text data and emotion data, converts it into voice data and generates natural speech that reflects emotion.

[0475] Step 31:

[0476] Terminal: Receives the generated foreign language audio and plays it back through the speaker.

[0477] Step 32:

[0478] User: Listen to the foreign language audio and continue communicating with the other person.

[0479] Example 2

[0480] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0481] Conventional speech recognition and translation systems are unable to recognize speech and actions in real time and reflect emotions when translating into other languages, making it difficult to achieve natural, emotionally rich communication. Furthermore, they are unable to individually learn the speech and action characteristics of specific users, making it difficult to provide recognition accuracy tailored to each individual user. Furthermore, they lack the functionality to accommodate special situations, such as users who have lost their vocal cords.

[0482] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a voice recognition means, a motion recognition means, a means for analyzing voice data and motion data, a means for translating the analyzed data into another language, a means for converting the translated data into speech, a means for analyzing emotions and reflecting the emotions in translation and speech generation, and a control means for integrating the above means. This enables natural communication that reflects emotions in real time. Furthermore, the system can learn the voice and motion characteristics of specific users and provide recognition accuracy tailored to each individual user. Furthermore, by providing an input mode based on mouth movements, appropriate conversation assistance can be provided to users who have lost their vocal cords.

[0483] A "voice recognition means" is a device or mechanism that receives voice input and analyzes the voice signal to convert it into text data.

[0484] The "motion recognition means" is a device or mechanism that receives video input, analyzes the motions and gestures, and converts them into meaningful data.

[0485] The "means for analyzing voice data and movement data" refers to a device or mechanism that processes the acquired voice data and movement data, extracts its features, and performs further higher-level analysis.

[0486] The "means for translating parsed data into another language" is a device or mechanism for translating parsed text data into a target language.

[0487] The "means for converting translated data into speech" refers to a device or mechanism that converts the translated text data into speech signals and outputs them as speech.

[0488] The "means for analyzing emotions and reflecting them in translation and speech generation" is a device or mechanism that analyzes the emotional state of the user and reflects that emotional information in translation and speech generation.

[0489] The "control means for integrating the means" is a device or mechanism for efficiently coordinating and enabling a plurality of means to function.

[0490] A "means for learning the voice and movement characteristics of a specific user" is a device or mechanism that learns the voice and movement characteristics of a specific user using machine learning or the like, and generates a recognition model optimized for that user.

[0491] The present invention is a system in which a user, a terminal, and a server work together and combine an emotion engine to recognize voice and actions in real time, translate them into other languages, and then speak in a way that reflects the user's emotions.Specific embodiments of the present invention are described below.

[0492] System configuration

[0493] This system uses a dedicated application installed on a smartphone or tablet device (hereafter referred to as "device") and connects to a server located in the cloud. It also incorporates an emotion engine that analyzes the user's emotions from voice and movement data and reflects this in the translation results and speech generation. Using this app, users can communicate with speakers of other languages ​​without having to learn the language.

[0494] Initial Setup and User Registration

[0495] First, a user installs the app on their smartphone and launches it. The device displays an initial setup screen and prompts the user to enter necessary information such as their name, target language, and permission for sentiment analysis. The information entered by the user is sent by the device to a server, which then stores the received user information in a database and generates a unique user identifier.

[0496] Voice and movement learning phase

[0497] Next, the user follows the app's instructions to speak a few short Japanese phrases. The device captures the voice with a microphone and records the mouth movements with a camera. The captured voice and movement data are sent to the server in real time. The server uses a generative AI model to analyze the user's features for voice and movement recognition, and to train a model for converting the voice and movement data into text data.

[0498] Emotion analysis using an emotion engine

[0499] The server uses the received voice data and movement data to analyze the user's emotions with its emotion engine. The analyzed emotion data is then saved along with the user's voice and movement characteristics, allowing the user's emotional state to be properly reflected.

[0500] Real-time speech recognition and translation

[0501] After the user completes the setup, they select the target language and speak in Japanese (e.g., "Hello, what is your name?"). The device captures the voice with a microphone and recognizes mouth movements with a camera. The captured voice and movement data are sent to a server in real time, and the server analyzes the received voice data and converts it into Japanese text. The Japanese text and emotion data are then used to translate it into the target language (e.g., English) (e.g., "Hello, what is your name?"). The translated text data and emotion data are then converted into voice data, generating natural, emotionally-reflective speech. The generated foreign language voice is provided to the user by the device and played through the speaker.

[0502] Conversation assistance functions

[0503] If a user has lost their vocal cords, they can select the mouth movement input mode. The user mimics speech using only mouth movements (e.g., "Onegaishimasu"), and the device captures the mouth movements with a camera and sends the data to the server. The server analyzes the gesture recognition data and converts it into Japanese text (e.g., "Onegaishimasu"). Based on this, a translated target language (e.g., English) is generated (e.g., "Please") to create natural-sounding speech. The generated audio is played back from the device to support the conversation.

[0504] Specific examples

[0505] Example 1: Normal Japanese to English Conversation

[0506] When a user opens the app, selects English mode, and says, "Hello, what is your name?", the device captures the voice and movements and sends them to the server. The server translates this into "Hello, what is your name?" and incorporates emotion data. The translation result is converted into audio data, generating natural-sounding English speech, which the device then plays back to support the conversation.

[0507] Example 2: Speech assistance for users who have lost their vocal cords

[0508] When a user opens the app, selects the lip-activated input mode, and imitates the phrase "onegaishimasu" (please) using only their mouth movements, the device captures the movement and sends it to the server. The server translates this into "please" and incorporates emotional data. The translation result is converted into audio data, generating natural-sounding English speech, which the device then plays back to support the conversation.

[0509] In this way, the system of the present invention learns the user's voice characteristics and mouth movements and analyzes the user's emotions, enabling more natural and emotional communication. This not only effectively removes language barriers, but also provides a realistic conversation experience that reflects the user's emotions.

[0510] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0511] Step 1:

[0512] Initial Setup and User Registration

[0513] The user installs the dedicated app on their smartphone and launches the app.

[0514] The terminal displays an initial setup screen and prompts the user to enter information such as their name, target language, and permission for sentiment analysis.

[0515] The user enters information such as name, target language, and permission for sentiment analysis.

[0516] The terminal transmits the input information to the server (input: user information, output: data transmission to the server).

[0517] The server stores the received user information in a database and generates an identifier for each user (input: user information, output: identifier).

[0518] Step 2:

[0519] Voice and movement learning

[0520] The user speaks a short Japanese phrase as instructed by the app.

[0521] The device captures audio with a microphone and records mouth movements with a camera (input: audio, movement; output: captured data).

[0522] The terminal transmits the captured voice data and movement data to the server in real time (input: captured data, output: data transmission to the server).

[0523] The server analyzes the received data using a generative AI model (deep learning framework) and trains a model that converts voice data and movement data into text data (input: voice data, movement data; output: trained model).

[0524] Step 3:

[0525] Emotion analysis using an emotion engine

[0526] The server analyzes the received voice data and action data using an emotion engine to identify the user's emotion (input: voice data, action data, output: emotion data).

[0527] The server stores the analyzed emotion data in a database together with the user's voice characteristics and behavior characteristics (input: emotion data, output: saved data).

[0528] Step 4:

[0529] Real-time speech recognition and translation

[0530] The user selects the target language and speaks in Japanese (e.g., "Hello, what's your name?").

[0531] The device captures audio with a microphone and recognizes mouth movements with a camera (input: audio, movement; output: captured data).

[0532] The terminal transmits the captured voice data and motion data to the server in real time (input: captured data, output: data transmission to the server).

[0533] The server analyzes the received voice data and converts it into Japanese text (input: voice data, output: Japanese text).

[0534] The server uses the Japanese text and emotion data to translate into the target language (e.g., English: "Hello, what is your name?", input: Japanese text, emotion data, output: translated text).

[0535] The server converts the translated text data and emotional data into audio data, generating natural, emotionally-reflective speech (input: translated text, emotional data; output: foreign language audio).

[0536] The terminal plays the generated foreign language speech from a speaker (input: foreign language speech, output: speech playback).

[0537] Step 5:

[0538] Conversation assistance functions

[0539] If a user loses their vocal cords, they can select mouth input mode in the app.

[0540] The user mimics speech using only mouth movements (e.g., "please").

[0541] The device captures mouth movements with a camera and sends the data to the server (input: movement data, output: sending data to the server).

[0542] The server analyzes the action recognition data and converts it into Japanese text (input: action data, output: Japanese text).

[0543] The server uses the Japanese text and emotion data to translate into the target language (e.g., English: "Please", input: Japanese text, emotion data, output: translated text).

[0544] The server converts the translated text data and emotional data into audio data, generating natural, emotionally-reflective speech (input: translated text, emotional data; output: foreign language audio).

[0545] The terminal plays the generated foreign language speech from a speaker (input: foreign language speech, output: speech playback).

[0546] (Application example 2)

[0547] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0548] In modern society, smooth communication between multiple languages ​​remains a major challenge. Particularly in the food delivery field, it is difficult for foreign customers and the hearing impaired to easily place orders. Another problem is that the translated results do not reflect emotions, making it difficult to provide a natural conversation and user experience.

[0549] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0550] In this invention, the server includes means for analyzing voice data and motion data, means for analyzing emotion data and reflecting the emotion data in translation and speech generation, and means for analyzing, translating, and generating speech using a generative AI model. This makes it possible to capture voice and motion data in real time and provide translation results and speech that reflects emotion.

[0551] The "voice recognition means" is a means for capturing the voice spoken by the user and converting it into text data.

[0552] The "motion recognition means" is a means for capturing and analyzing the user's body movements, particularly the mouth movements.

[0553] The "means for analyzing voice data and motion data" refers to a means for processing the captured voice and motion data and extracting necessary information.

[0554] The "means for translating analyzed data into another language" refers to a means for automatically translating into another language based on the recognized speech and motion data.

[0555] The "means for converting translated data into speech" refers to a means for generating speech data from text data translated into another language.

[0556] The "means for analyzing emotion data and reflecting it in translation and speech generation" is a means for analyzing the user's emotion and reflecting it in translation and speech generation.

[0557] "Means that enable a standalone terminal and a server located on the cloud to work in cooperation" refers to means that enable data exchange and cooperative operation between multiple devices.

[0558] The "means for capturing voice and actions in real time and transmitting them to a server" refers to a means for recognizing the user's voice and actions in real time and immediately transmitting them to a server.

[0559] "Means for analyzing, translating, and generating speech data and movement data using a generative AI model" means means for analyzing speech and movement data and performing translation and speech generation processes using a machine learning model.

[0560] The "means for providing the user with the translation result and the speech that reflects the emotion" is a means for providing the user with the translated text and the speech that reflects the emotion.

[0561] This invention is a system that recognizes voice and actions in real time, translates them into other languages, and speaks in a way that reflects the user's emotions by linking the user, device, and server and combining them with an emotion engine. This system uses a dedicated application installed on a smartphone or tablet device and links with a server located in the cloud. Furthermore, it incorporates an emotion engine that analyzes the user's emotions from voice data and movement data and reflects them in the translation results and speech generation.

[0562] Initial Setup and User Registration

[0563] First, the user installs the application on their smartphone and follows the initial setup screen to enter necessary information such as their name, target language, permission for emotion analysis, etc. This information is sent from the device to the server, where it is stored in a database.

[0564] Voice and movement learning phase

[0565] To improve the accuracy of voice and movement data, users are prompted by the application to speak a few short Japanese phrases. The device then captures the voice with a microphone and records the mouth movements with a camera. This data is then sent in real time to a server, where the generative AI model is trained for voice and movement recognition.

[0566] Emotion analysis using an emotion engine

[0567] The server analyzes the user's emotions using the received voice and movement data. The analyzed emotion data is stored along with the user's voice and movement characteristics. This information is used in subsequent communications.

[0568] Real-time speech recognition and translation

[0569] When a user communicates in real time with a speaker of another language, the device's microphone captures the voice and the camera recognizes the mouth movements. The captured data is sent to a server, where the voice data is recognized and converted into Japanese text. The server then translates the Japanese text and emotion data into the target language and generates emotion-reflecting voice data using a generative AI model. The generated foreign language voice is then provided to the user from the device.

[0570] Conversation assistance functions

[0571] The system also provides a mouth-movement speech input mode for users who have lost their vocal cords. The user mimics speech using only their mouth movements, and the device captures and transmits the movements to the server. The server analyzes the movement data, translates it into the target language, and then generates speech that reflects the emotions.

[0572] Examples of concrete examples and prompts

[0573] Example 1:

[0574] Example: A user says, "I'd like a special hamburger and two cokes, please."

[0575] Example prompt: "I'd like a special hamburger and two colas, please." Translate this into English and speak it in a way that reflects the emotion.

[0576] Example 2:

[0577] Example: A user loses their vocal cords and uses mouth movements to mimic the saying "please."

[0578] Example prompt: Translate "please" into English and say it in a way that reflects your feelings.

[0579] The system is realized using cloud services such as Google Cloud Platform's AutoML, Translate API, and Text-to-Speech API. The generative AI model sends voice and movement data captured on the device to the server in real time and performs various data calculations to enable advanced translation and speech generation. This allows users to enjoy a natural and emotional conversation experience.

[0580] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0581] Step 1:

[0582] The user launches the application and enters the necessary information, such as their name, target language, and permission for sentiment analysis, on the initial setup screen. The entered information is sent from the device to the server, which stores it in a database and generates a user identifier. Once this initial setup process for user information is complete, the application is ready to proceed to the next step.

[0583] Step 2:

[0584] In the voice and movement learning phase, the user speaks a few short Japanese phrases according to the application's instructions. The device uses a microphone to capture voice data and a camera to record mouth movement data. The captured voice data and movement data are sent to the server in real time.

[0585] Step 3:

[0586] The server trains a generative AI model based on the received voice and motion data, learning the user's voice and motion characteristics. This process involves analyzing the data (voice recognition, motion recognition) and converting it into text. The resulting model is then used to perform voice and motion recognition optimized for each individual user.

[0587] Step 4:

[0588] The server analyzes the user's emotions from the voice data and movement data. The server stores the analyzed emotion data along with the user's voice characteristics and movement characteristics. For example, if the user says "please," the tone of voice and mouth movements are analyzed and stored as polite emotion data. This data will be used in a later step.

[0589] Step 5:

[0590] This is a real-time speech recognition and translation process. The user speaks Japanese into the device's microphone to translate the sentence into the target language. The device captures the speech and uses a camera to recognize mouth movements. This data is sent to the server in real time.

[0591] Step 6:

[0592] The server analyzes the received voice data and converts it into Japanese text. It then uses a generative AI model to translate the text based on the stored emotion data. For example, the Japanese text "Two special hamburgers and two Cokes, please" is translated into "Two special hamburgers and two Cokes, please."

[0593] Step 7:

[0594] Based on the translated text data and emotion data, the server uses a generative AI model to generate emotion-reflecting voice data, which is then sent to the device in real time.

[0595] Step 8:

[0596] The device receives the foreign language audio from the server and plays it back from the speaker, allowing the user to have a natural speaking experience and enabling multilingual communication.

[0597] Step 9:

[0598] This conversation assistance function for users who have lost their vocal cords imitates speech using only mouth movements. The user selects the mouth input mode and imitates speech using mouth movements. The device captures the movements and sends them to the server.

[0599] Step 10:

[0600] The server analyzes the motion data and converts it into Japanese text. It then uses a generative AI model to translate this text and emotion data into other languages ​​and generate audio data. The generated audio data is then sent to the device and played through the speaker.

[0601] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0602] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0603] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0604] [Second embodiment]

[0605] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0606] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0607] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0608] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0609] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0610] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0611] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0612] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0613] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0614] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0615] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0616] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0617] The present invention is a system in which a user, a terminal, and a server work together to recognize voices and actions in real time, convert them into another language, and then speak the voices. Specific embodiments of the present invention are described below.

[0618] System Overview

[0619] This system basically uses a dedicated application installed on a smartphone or tablet device (hereinafter referred to as "device") and connects to a server located in the cloud. Using this application, users can communicate with speakers of other languages ​​without having to learn the foreign language.

[0620] Initial Setup and User Registration

[0621] 1. User: First, install the app on your smartphone and launch it.

[0622] 2. Device: When the app launches, an initial setup screen appears, prompting the user to enter necessary information (such as name and desired language).

[0623] 3. Terminal: Sends the information entered by the user to the server.

[0624] 4. Server: Stores the received user information in a database and generates an identifier corresponding to each individual user.

[0625] Voice and movement learning phase

[0626] 1. User: Follow the instructions of the app and speak a few short Japanese phrases.

[0627] 2. Device: A microphone captures audio and a camera records mouth movements.

[0628] 3. Terminal: Sends voice data and mouth movement data to the server in real time.

[0629] 4. Server: The AI ​​that generates speech and motion recognition analyzes the user's characteristics and learns a model for converting speech and motion data into text data.

[0630] Real-time speech recognition and translation

[0631] 1. User: After completing the setup, select the target language and speak in Japanese (e.g., "Hello, what's your name?").

[0632] 2. Device: The microphone captures the user's voice and the camera recognizes mouth movements.

[0633] 3. Terminal: Sends the captured voice and movement data to the server.

[0634] 4. Server: Analyzes the received data and converts it into Japanese text.

[0635] 5. Server: Translate the Japanese text into the target language (e.g., English).

[0636] 6. Server: Converts the translated text data into audio data and generates natural-sounding speech.

[0637] 7. Terminal: The generated foreign language speech is provided to the user and played back through the speaker.

[0638] Conversation assistance functions

[0639] 1. User: If you have lost your vocal cords, select mouth input mode.

[0640] 2. User: Imitate speech using mouth movements only (e.g., "please").

[0641] 3. Device: The camera captures mouth movements and sends the data to the server.

[0642] 4. Server: Analyzes the motion recognition data and converts it into Japanese text.

[0643] 5. Server: Translate the Japanese text into the target language.

[0644] 6. Server: Converts the translated text data into audio data and generates natural-sounding speech.

[0645] 7. Terminal: Plays back the generated foreign language speech and provides it to the user.

[0646] Specific examples

[0647] Example 1: Normal Japanese to English Conversation

[0648] 1. User: Open the app, select English mode and say "Hello, what's your name?"

[0649] 2. Device: Captures voice and movements and sends them to the server.

[0650] 3. Server: Translate this into "Hello, what is your name?" and generate it as voice data.

[0651] 4. Terminal: Plays the generated English audio and supports the conversation.

[0652] Example 2: Speech assistance for users who have lost their vocal cords

[0653] 1. User: Open the app and select mouth input mode.

[0654] 2. User: Imitate the word "please" using only mouth movements.

[0655] 3. Device: Captures the action and sends it to the server.

[0656] 4. Server: Translate this into "Please" and generate it as voice data.

[0657] 5. Terminal: Plays the generated English audio and supports the conversation.

[0658] The system of the present invention learns the user's voice characteristics and mouth movements to enable natural conversations that transcend language barriers. It also has a function that notifies the user of an error if a problem occurs with voice capture or translation, prompting the user to take appropriate action. This allows the system to efficiently overcome language barriers, even in situations where communication is difficult, such as sudden overseas business trips.

[0659] The processing flow will be explained below.

[0660] Step 1:

[0661] User: Install the dedicated app on your smartphone and launch it.

[0662] Step 2:

[0663] Terminal: Displays an initial setup screen and prompts the user to enter necessary information such as name and desired language.

[0664] Step 3:

[0665] User: Enter the required information on the initial setup screen and press the "Next" button after completing the entry.

[0666] Step 4:

[0667] Terminal: Sends the information entered by the user to the server.

[0668] Step 5:

[0669] Server: Stores the received user information in a database and generates a unique user identifier.

[0670] Step 6:

[0671] Terminal: Displays prompts to the user to begin the voice and movement learning phase.

[0672] Step 7:

[0673] User: Follow the app's instructions and speak a few short Japanese phrases.

[0674] Step 8:

[0675] Device: A microphone captures your voice and a camera records your mouth movements.

[0676] Step 9:

[0677] Terminal: Sends voice and movement data to the server in real time.

[0678] Step 10:

[0679] Server: Analyzes the received voice and movement data and learns the characteristics of the user's voice and movements.

[0680] Step 11:

[0681] Server: The generated analysis model is linked to the user's account and saved.

[0682] Step 12:

[0683] User: After the initial setup and learning phase is complete, select your target language on the app's main screen.

[0684] Step 13:

[0685] User: Speak in Japanese (e.g. "Hello, what's your name?").

[0686] Step 14:

[0687] Device: A microphone captures the user's voice and a camera recognizes mouth movements.

[0688] Step 15:

[0689] Terminal: Transmits captured audio and motion data to the server in real time.

[0690] Step 16:

[0691] Server: Analyzes the received voice data and converts it into Japanese text (e.g., "Hello, what's your name?").

[0692] Step 17:

[0693] Server: Translate Japanese text into the target language (e.g., English) (e.g., "Hello, what is your name?").

[0694] Step 18:

[0695] Server: Converts the translated text data into audio data and generates natural-sounding speech.

[0696] Step 19:

[0697] Terminal: Receives the generated foreign language audio and plays it back through the speaker.

[0698] Step 20:

[0699] User: Listen to the foreign language audio and continue communication.

[0700] Step 21:

[0701] Terminal: If an error is detected, an error message is displayed to the user.

[0702] Step 22:

[0703] User: Follow the error message and make any necessary corrections or try again.

[0704] Step 23:

[0705] User: If you have lost your vocal cords, select mouth input mode.

[0706] Step 24:

[0707] User: Mimic speech using mouth movements only (e.g., "please").

[0708] Step 25:

[0709] Device: The camera captures mouth movements and sends the data to the server.

[0710] Step 26:

[0711] Server: Analyzes the action recognition data and converts it into Japanese text (e.g., "Onegaishimasu").

[0712] Step 27:

[0713] Server: Translate Japanese text into the target language (e.g., English) (e.g., "Please").

[0714] Step 28:

[0715] Server: Converts the translated text data into audio data and generates natural-sounding speech.

[0716] Step 29:

[0717] Terminal: Receives the generated foreign language audio and plays it back through the speaker.

[0718] Step 30:

[0719] User: Listen to the foreign language audio and continue communication.

[0720] Example 1

[0721] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0722] In today's globalized society, communication between people who speak different languages ​​is becoming increasingly important. However, the existence of language barriers makes smooth communication difficult. Furthermore, people with certain physical limitations (e.g., those who have lost their vocal cords) cannot use standard speech recognition technology, making communication even more difficult. To solve these challenges, there is a need for a system that can recognize voices and movements in real time, convert them into speech in other languages, and then speak them.

[0723] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0724] In this invention, the server includes a voice recognition unit, a motion recognition unit, a unit for analyzing voice data and motion data, a unit for translating the analyzed data into another language, and a unit for converting the translated data into voice. This allows the user to perform initial setup, and a unit for saving and managing the setup information, a unit for capturing voice and motion and transmitting them to the server in real time, and a unit for the server to provide the user with the analysis results, enabling smooth communication between people who speak different languages. Furthermore, even people with certain physical limitations can have natural conversations using mouth movements.

[0725] "Speech recognition means" is a general term for hardware and software for capturing speech as digital data and analyzing its content.

[0726] "Movement recognition means" is a general term for hardware and software for capturing the user's body movements, particularly mouth movements, and analyzing those movements.

[0727] The "means for analyzing voice data and movement data" is a mechanism for analyzing the captured voice data and movement data using an algorithm and converting it into text data.

[0728] "Means for translating analyzed data into another language" is a general term for hardware and software for translating analyzed text data into another specified language.

[0729] "Means for converting translated data into speech" is a general term for hardware and software for converting translated text data into speech data and generating natural-sounding speech.

[0730] The "control means for integrating the above means" is a control system for integrally operating the voice recognition means, the motion recognition means, the analysis means, the translation means, and the voice conversion means.

[0731] "Means for users to perform initial setup and save and manage setup information" is a general term for the interface and database that allows users to perform initial setup of the system and save and manage that information.

[0732] "Means for capturing voice and movement and transmitting it to a server in real time" is a general term for software and network communication means for acquiring voice and movement data, compressing it, and transmitting it to a server.

[0733] "Means by which the server provides the analysis results to the user" refers to a mechanism by which the server returns the analysis and translation results to the user's terminal and displays or plays them aloud.

[0734] The present invention is a system in which a user, a terminal, and a server work together to recognize voices and actions in real time, convert them into another language, and then speak the voices. Specific embodiments of the present invention are described below.

[0735] System Overview

[0736] This system basically uses a dedicated application installed on a smartphone or tablet device (hereinafter referred to as "device") and connects to a server located in the cloud. Using this application, users can communicate with speakers of other languages ​​without having to learn the foreign language.

[0737] Initial Setup and User Registration

[0738] 1. User: Installs the app on a smartphone and launches it. The app can be downloaded from the Google Play Store or the App Store.

[0739] 2. Device: When the app launches, an initial setup screen appears, prompting the user to enter information such as their name, preferred language, and email address. Once the information is entered, the device encrypts and sends it to the server.

[0740] 3. Server: Stores the received user information in a database and generates a unique identifier that is sent back to the device for future operations and data management.

[0741] Voice and movement learning phase

[0742] 1. User: Follow the instructions of the app to speak a few short Japanese phrases, such as "hello" and "thank you."

[0743] 2. Device: A microphone captures audio and a camera records lip movements. Audio and video are synchronized to collect precise data.

[0744] 3. Terminal: Collected voice and movement data is compressed in real time and sent to the server.

[0745] 4. Server: Analyzes and learns voice and movement features using generative AI models, and optimizes the model to convert this data into text.

[0746] Real-time speech recognition and translation

[0747] 1. User: After completing the setup, select the target language and speak in Japanese. For example, the user selects "English mode" and speaks "Hello, what is your name?"

[0748] 2. Device: The microphone captures the voice and the camera recognizes the mouth movements. This data is sent to the server in real time.

[0749] 3. Server: Analyzes the received voice and motion data and converts it into Japanese text. Then, translates the Japanese text into the target language (e.g., English). For example, "Hello, what is your name?" is converted to "Hello, what is your name?"

[0750] 4. Server: Converts the translated text data into audio data and generates natural-sounding speech.

[0751] 5. Terminal: The generated foreign language speech is provided to the user and played back through the speaker.

[0752] Conversation assistance functions

[0753] 1. User: If you have lost your vocal cords, select "Mouth Recognition Mode" from the app settings.

[0754] 2. User: Imitate speech using only mouth movements, for example, recreating the mouth movements for "please."

[0755] 3. Device: The camera captures mouth movements and sends the data to the server.

[0756] 4. Server: Analyzes the motion recognition data and converts it into Japanese text. Next, translates "onegaishimasu" into "Please."

[0757] 5. Server: Converts the translated text data into audio data and generates natural-sounding speech.

[0758] 6. Terminal: Plays back the generated foreign language speech and provides it to the user.

[0759] Examples and prompts

[0760] Example 1: Normal Japanese to English Conversation

[0761] 1. User: Open the app, select English mode and say "Hello, what's your name?"

[0762] 2. Device: Captures voice and movements and sends them to the server.

[0763] 3. Server: Translate this into "Hello, what is your name?" and generate it as voice data.

[0764] 4. Terminal: Plays the generated English audio and supports the conversation.

[0765] Example 2: Speech assistance for users who have lost their vocal cords

[0766] 1. User: Open the app and select mouth input mode.

[0767] 2. User: Imitate the word "please" using only mouth movements.

[0768] 3. Device: Captures the action and sends it to the server.

[0769] 4. Server: Translate this into "Please" and generate it as voice data.

[0770] 5. Terminal: Plays the generated English audio and supports the conversation.

[0771] As an example of a prompt sentence, if you enter "Translate and speak the Japanese greeting into English," the Japanese "Konnichiwa" will be translated and spoken as the English "Hello."

[0772] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0773] Step 1:

[0774] User: Installs the app on their smartphone and launches it.

[0775] Input: Download the app file from Google Play Store or App Store.

[0776] How it works: Press the app's install button, then tap the app icon to launch it after installation is complete.

[0777] Output: The initial setup screen is displayed on the terminal.

[0778] Step 2:

[0779] Device: An initial setup screen appears, prompting the user to enter information such as name, preferred language, and email address.

[0780] Input: The user enters their name, desired language, email address, etc.

[0781] Action: Enter the required information in the input form and press the "Submit" button.

[0782] Output: The input information is encrypted and sent from the device to the server.

[0783] Step 3:

[0784] Server: Stores the received user information in a database and generates and returns an identifier.

[0785] Input: User information data entered.

[0786] What it does: Connects to a database, stores the received data, and generates a unique identifier.

[0787] Output: The generated identifier is sent back to the terminal.

[0788] Step 4:

[0789] User: Follow the app's instructions and speak a few short Japanese phrases.

[0790] Input: A phrase to display on the app screen (e.g. "Hello", "Thank you").

[0791] Action: Reads the displayed text aloud.

[0792] Output: The user's voice and mouth movement data is generated.

[0793] Step 5:

[0794] Device: A microphone captures your voice and a camera records your mouth movements.

[0795] Input: The user's voice and mouth movements.

[0796] Movement: High-definition microphones and high-resolution cameras capture audio waveforms and movement data.

[0797] Output: Captured audio and movement data.

[0798] Step 6:

[0799] Terminal: The captured voice data and movement data are compressed in real time and sent to the server.

[0800] Input: Audio and motion data.

[0801] What it does: Compresses data and sends it to the server with low latency.

[0802] Output: Audio and movement data received by the server.

[0803] Step 7:

[0804] Server: Analyzes and learns voice and movement characteristics using a generative AI model.

[0805] Input: Received voice and movement data.

[0806] How it works: The generative AI model extracts and analyzes features, optimizing the model for conversion to text data.

[0807] Output: Optimized speech-to-motion translation model.

[0808] Step 8:

[0809] User: After completing the setup, select the target language and speak in Japanese.

[0810] Input: The target language selected on the app screen, the words you speak (e.g. "Hello, what's your name?").

[0811] Action: Select the target language and speak in Japanese.

[0812] Output: Spoken Japanese audio.

[0813] Step 9:

[0814] Device: A microphone captures the user's voice and a camera recognizes mouth movements.

[0815] Input: Spoken speech and mouth movements.

[0816] How it works: Captures audio waveforms in real time and records mouth movements with a camera.

[0817] Output: Captured audio and movement data.

[0818] Step 10:

[0819] Terminal: Sends captured audio and motion data to the server.

[0820] Input: Audio and motion data.

[0821] Operation: Real-time transmission of data.

[0822] Output: Audio and motion data received by the server.

[0823] Step 11:

[0824] Server: Analyzes the received voice data and movement data and converts it into Japanese text.

[0825] Input: Audio and motion data.

[0826] Actions: The generative AI model analyzes voice and actions and converts them into Japanese text data.

[0827] Output: Converted Japanese text data.

[0828] Step 12:

[0829] Server: Translates Japanese text into the target language.

[0830] Input: Japanese text.

[0831] Behavior: Translates to the specified target language (e.g., English).

[0832] Output: The translated text data (e.g. "Hello, what is your name?").

[0833] Step 13:

[0834] Server: Converts the translated text data into audio data and generates natural-sounding speech.

[0835] Input: Translated text data.

[0836] How it works: Generates natural-sounding speech using generative AI models.

[0837] Output: The generated audio data.

[0838] Step 14:

[0839] Terminal: Provides the generated foreign language speech to the user and plays it back through a speaker.

[0840] Input: The generated audio data.

[0841] Action: Plays audio through a playback device.

[0842] Output: Foreign language speech played through the speaker (e.g. "Hello, what is your name?").

[0843] Step 15:

[0844] User: If you have lost your vocal cords, select "Mouth Recognition Mode" from the app settings.

[0845] Input: Select "Mouth Recognition Mode" on the app's settings screen.

[0846] Behavior: Toggles selection mode.

[0847] Output: Selected mode.

[0848] Step 16:

[0849] User: Mimics speech using mouth movements only.

[0850] Input: Mouth movements (e.g. "please").

[0851] Action: Mimics mouth movements.

[0852] Output: Mouth movement data.

[0853] Step 17:

[0854] Device: The camera captures mouth movements and sends the data to the server.

[0855] Input: Mouth movement data.

[0856] Movement: A high-precision camera captures mouth movements and transmits the data.

[0857] Output: Lip movement data sent to the server.

[0858] Step 18:

[0859] Server: Analyzes the motion recognition data and converts it into Japanese text.

[0860] Input: Captured mouth movement data.

[0861] Movement: Movement recognition data is analyzed and converted into Japanese text.

[0862] Output: Japanese text (e.g. "Onegaishimasu").

[0863] Step 19:

[0864] Server: Translates Japanese text into the target language.

[0865] Input: Japanese text.

[0866] Behavior: Translates to the specified target language (e.g., English).

[0867] Output: The translated text data (e.g. "Please").

[0868] Step 20:

[0869] Server: Converts the translated text data into audio data and generates natural-sounding speech.

[0870] Input: Translated text data.

[0871] How it works: Generates natural-sounding speech using generative AI models.

[0872] Output: The generated audio data.

[0873] Step 21:

[0874] Terminal: Plays back the generated foreign language speech and provides it to the user.

[0875] Input: The generated audio data.

[0876] Action: Plays audio through a playback device.

[0877] Output: Foreign language voice (e.g. "Please") played through the speaker.

[0878] (Application example 1)

[0879] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0880] Conventional food delivery systems face a language barrier when customers who speak a foreign language place an order, making it difficult to accurately convey order information. This leads to frequent ordering errors and communication problems with customers, resulting in a decline in service quality. Furthermore, only a limited number of systems allow input using mouth movements in addition to voice input, making them inadequate for specific users. To solve these problems, a means was needed to translate and speak order details in real time across language barriers, enabling smooth communication.

[0881] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0882] In this invention, the server includes a voice recognition means, a motion recognition means, a means for analyzing the voice data and motion data, a means for providing the translated voice data as order information to the delivery person or the store, and a control means for integrating the above means. This enables accurate real-time transmission of order information between customers who speak a foreign language and the delivery person or the store, realizing smooth communication that transcends language barriers.

[0883] A "voice recognition means" is a device or technology that receives voice as input and converts it into digital data that can be analyzed.

[0884] "Movement recognition means" refers to a device or technology that captures a user's physical movements, particularly facial and mouth movements, using a sensor such as a camera, and converts them into digital data that can be analyzed.

[0885] "Means for analyzing voice data and movement data" refers to software or hardware that analyzes data based on input voice or movement and converts it into text data or other formats.

[0886] "Means for translating the analyzed data into another language" refers to software or algorithms for translating the analyzed text data into a target language.

[0887] "Means for converting translated data into audio" refers to a device or software that converts translated text data into audio data and outputs it as audio.

[0888] "Means for providing translated voice data as order information to delivery personnel and stores" refers to technology and devices that use translated voice data to transmit accurate order information to the delivery system.

[0889] "Control means" refers to software and hardware that integrates and manages the above means and controls the entire system to ensure smooth operation.

[0890] The present invention provides a system that recognizes speech and gestures in real time and translates them into other languages ​​for speech, which is particularly useful for food delivery services to assist customers who speak other languages ​​in placing orders.

[0891] System configuration

[0892] Hardware Configuration

[0893] The system includes the following hardware:

[0894] Device: Smartphone or tablet

[0895] Microphone: A device used to capture the user's voice.

[0896] Camera: A device for capturing the user's mouth movements.

[0897] Speaker: A device for playing the translated audio.

[0898] Software Configuration

[0899] The system uses the following software:

[0900] Speech Recognition Library: speech_recognition library

[0901] Action recognition library: OpenCV (cv2)

[0902] Translation API: Google Cloud Translation API

[0903] Speech synthesis API: gTTS (Google Text-to-Speech)

[0904] System Operation

[0905] Speech and gesture recognition

[0906] When a user speaks, the device's microphone captures the voice and the camera records the mouth movements. This information is converted into text data using a speech recognition library (speech_recognition) and a motion recognition library (OpenCV). The data captured by the speech recognition and motion recognition means is processed in real time.

[0907] Data analysis and translation

[0908] The server analyzes the voice and motion data in the cloud, converts the user's speech into text using the analysis method, and then translates the analyzed text into the target language using the Google Cloud Translation API.

[0909] Speech synthesis of translation data

[0910] The translated text data is converted into voice data using the gTTS (Google Text-to-Speech) API. The voice data generated using a voice synthesis tool is played back on the device's speaker. This allows users to smoothly place orders through food delivery services without having to learn a foreign language.

[0911] Specific examples

[0912] Order Scenarios

[0913] 1. User: Download the smartphone app and select the language you want to use (e.g., English).

[0914] 2. User: "Hello, I'd like one Margherita pizza please" in Japanese.

[0915] 3. Device: Captures voice and lip movements and sends the data to the server.

[0916] 4. Server: Translates the speech data to "Hello, one Margherita pizza please" and converts it to speech data.

[0917] 5. Terminal: Plays back the translated voice data and provides it to delivery personnel and stores.

[0918] Prompt Sentence Examples

[0919] "How can I develop a food delivery app that translates Japanese speech into English in real time and provides it in a way that delivery staff and restaurants can understand?"

[0920] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0921] Step 1:

[0922] Input: The user launches the app on their smartphone and selects the language they want to use (e.g., English).

[0923] Specific operation: The user selects the desired language on the app's settings screen and enters their information (name, address, desired language, etc.).

[0924] Output: Configuration information is sent from the device to the server.

[0925] Step 2:

[0926] Input: User speaks in Japanese (e.g., "Hello, I'd like one Margherita pizza please.").

[0927] What it does: The device's microphone captures the user's voice and the camera records their mouth movements.

[0928] Output: Audio and motion data are generated.

[0929] Step 3:

[0930] Input: Audio and motion data captured by the device.

[0931] Specific operation: Using the speech recognition library (speech_recognition), converts voice data into text data. It also analyzes mouth movement data recorded by a camera.

[0932] Output: Japanese text data is generated and sent to the server.

[0933] Step 4:

[0934] Input: Japanese text data received by the server.

[0935] What it does: It uses the Google Cloud Translation API in the cloud to translate text data into the desired language (e.g., English).

[0936] Output: The translated text data is generated.

[0937] Step 5:

[0938] Input: Translated text data.

[0939] What it does: Uses the gTTS (Google Text-to-Speech) API to convert the translated text data into audio data.

[0940] Output: The translated audio data is generated.

[0941] Step 6:

[0942] Input: Translated audio data.

[0943] Specific operation: The device's speaker plays the generated voice data, which is then provided to the delivery person or the store.

[0944] Output: The translated content is played back as audio, and accurate order information is communicated to the delivery person or store.

[0945] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0946] The present invention is a system that recognizes voice and actions in real time, translates them into other languages, and speaks in a way that reflects the user's emotions by linking the user, terminal, and server and combining them with an emotion engine. Specific embodiments of the present invention are described below.

[0947] System configuration

[0948] This system uses a dedicated application installed on a smartphone or tablet device (hereafter referred to as "device") and connects to a server located in the cloud. It also incorporates an emotion engine that analyzes the user's emotions from voice and movement data and reflects this in the translation results and speech generation. Using this app, users can communicate with speakers of other languages ​​without having to learn the language.

[0949] Initial Setup and User Registration

[0950] 1. User: First, install the app on your smartphone and launch it.

[0951] 2. Device: Displays the initial setup screen and prompts the user to enter necessary information such as their name, target language, and permission for sentiment analysis.

[0952] 3. Terminal: Sends the information entered by the user to the server.

[0953] 4. Server: Stores the received user information in a database and generates a unique user identifier.

[0954] Voice and movement learning phase

[0955] 1. User: Follow the instructions of the app and speak a few short Japanese phrases.

[0956] 2. Device: A microphone captures audio and a camera records mouth movements.

[0957] 3. Terminal: Sends voice data and mouth movement data to the server in real time.

[0958] 4. Server: The AI ​​that generates speech and motion recognition analyzes the user's characteristics and learns a model for converting speech and motion data into text data.

[0959] Emotion analysis using an emotion engine

[0960] 1. Server: Using the received voice data and movement data, the emotion engine analyzes the user's emotions.

[0961] 2. Server: Stores the analyzed emotion data along with the user's voice and movement characteristics.

[0962] Real-time speech recognition and translation

[0963] 1. User: After completing the setup, select the target language and speak in Japanese (e.g., "Hello, what's your name?").

[0964] 2. Device: The microphone captures audio and the camera recognizes mouth movements.

[0965] 3. Terminal: Transmits captured voice and movement data to the server in real time.

[0966] 4. Server: Analyzes the received voice data and converts it into Japanese text (e.g., "Hello, what's your name?").

[0967] 5. Server: Uses Japanese text and emotion data to translate into the target language (e.g., English) (e.g., "Hello, what is your name?").

[0968] 6. Server: Based on the translated text data and emotion data, it converts it into voice data and generates natural, emotionally-informed speech.

[0969] 7. Terminal: The generated foreign language speech is provided to the user and played back through the speaker.

[0970] Conversation assistance functions

[0971] 1. User: If you have lost your vocal cords, select mouth input mode.

[0972] 2. User: Imitate speech using mouth movements only (e.g., "please").

[0973] 3. Device: The camera captures mouth movements and sends the data to the server.

[0974] 4. Server: Analyzes the action recognition data and converts it into Japanese text (e.g., "Onegaishimasu").

[0975] 5. Server: Uses the Japanese text and emotion data to translate into the target language (e.g., English) (e.g., "Please").

[0976] 6. Server: Based on the translated text data and emotion data, it converts it into voice data and generates natural, emotionally-informed speech.

[0977] 7. Terminal: Receives the generated foreign language audio and plays it back through the speaker.

[0978] Specific examples

[0979] Example 1: Normal Japanese to English Conversation

[0980] 1. User: Open the app, select English mode and say "Hello, what's your name?"

[0981] 2. Device: Captures voice and movements and sends them to the server.

[0982] 3. Server: Translate this to "Hello, what is your name?" and incorporate sentiment data.

[0983] 4. Server: Converts the translation results into audio data and generates natural speech.

[0984] 5. Terminal: Plays the generated English audio and supports the conversation.

[0985] Example 2: Speech assistance for users who have lost their vocal cords

[0986] 1. User: Open the app and select mouth input mode.

[0987] 2. User: Imitate the word "please" using only mouth movements.

[0988] 3. Device: Captures the action and sends it to the server.

[0989] 4. Server: Translate this to "Please" and incorporate emotional data.

[0990] 5. Server: Converts the translation results into audio data and generates natural speech.

[0991] 6. Terminal: Plays the generated English audio and supports conversation.

[0992] In this way, the system of the present invention learns the user's voice characteristics and mouth movements and analyzes the user's emotions, enabling more natural and emotional communication. This not only effectively removes language barriers, but also provides a realistic conversation experience that reflects the user's emotions.

[0993] The processing flow will be explained below.

[0994] Step 1:

[0995] User: Install the dedicated app on your smartphone and launch it.

[0996] Step 2:

[0997] Device: Displays an initial setup screen and prompts the user to enter required information such as name, target language, and permission for sentiment analysis.

[0998] Step 3:

[0999] User: Enter the required information on the initial setup screen and press the "Next" button after completing the entry.

[1000] Step 4:

[1001] Terminal: Sends the information entered by the user to the server.

[1002] Step 5:

[1003] Server: Stores the received user information in a database and generates a unique user identifier.

[1004] Step 6:

[1005] Terminal: Displays prompts to the user to begin the voice and movement learning phase.

[1006] Step 7:

[1007] User: Follow the app's instructions and speak a few short Japanese phrases.

[1008] Step 8:

[1009] Device: A microphone captures your voice and a camera records your mouth movements.

[1010] Step 9:

[1011] Terminal: Sends voice and movement data to the server in real time.

[1012] Step 10:

[1013] Server: Analyzes the received voice and movement data and learns the characteristics of the user's voice and movements.

[1014] Step 11:

[1015] Server: The generated analysis model is linked to the user's account and saved.

[1016] Step 12:

[1017] Server: Analyzes the user's emotions using an emotion engine based on voice data and movement data.

[1018] Step 13:

[1019] Server: Stores the analyzed emotion data along with the user's voice and movement characteristics.

[1020] Step 14:

[1021] User: After completing the initial setup and learning phase, select the desired language on the app's main screen.

[1022] Step 15:

[1023] User: Speak in Japanese (e.g. "Hello, what's your name?").

[1024] Step 16:

[1025] Device: A microphone captures audio and a camera recognizes mouth movements.

[1026] Step 17:

[1027] Terminal: Transmits captured audio and motion data to the server in real time.

[1028] Step 18:

[1029] Server: Analyzes the received voice data and converts it into Japanese text (e.g., "Hello, what's your name?").

[1030] Step 19:

[1031] Server: Uses Japanese text and emotion data to translate into the target language (e.g., English) (e.g., "Hello, what is your name?").

[1032] Step 20:

[1033] Server: Based on the translated text data and emotion data, converts it into voice data and generates natural speech that reflects emotion.

[1034] Step 21:

[1035] Terminal: Receives the generated foreign language audio and plays it back through the speaker.

[1036] Step 22:

[1037] User: Listen to the foreign language audio and continue communicating with the other person.

[1038] Step 23:

[1039] Terminal: If an error is detected, an error message is displayed to the user.

[1040] Step 24:

[1041] User: Follow the error message and make any necessary corrections or try again.

[1042] Step 25:

[1043] User: If you have lost your vocal cords, select mouth input mode.

[1044] Step 26:

[1045] User: Mimic speech using mouth movements only (e.g., "please").

[1046] Step 27:

[1047] Device: The camera captures mouth movements and sends the data to the server.

[1048] Step 28:

[1049] Server: Analyzes the action recognition data and converts it into Japanese text (e.g., "Onegaishimasu").

[1050] Step 29:

[1051] Server: Uses Japanese text and emotion data to translate into the target language (e.g., English) (e.g., "Please").

[1052] Step 30:

[1053] Server: Based on the translated text data and emotion data, converts it into voice data and generates natural speech that reflects emotion.

[1054] Step 31:

[1055] Terminal: Receives the generated foreign language audio and plays it back through the speaker.

[1056] Step 32:

[1057] User: Listen to the foreign language audio and continue communicating with the other person.

[1058] Example 2

[1059] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1060] Conventional speech recognition and translation systems are unable to recognize speech and actions in real time and reflect emotions when translating into other languages, making it difficult to achieve natural, emotionally rich communication. Furthermore, they are unable to individually learn the speech and action characteristics of specific users, making it difficult to provide recognition accuracy tailored to each individual user. Furthermore, they lack the functionality to accommodate special situations, such as users who have lost their vocal cords.

[1061] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a voice recognition means, a motion recognition means, a means for analyzing voice data and motion data, a means for translating the analyzed data into another language, a means for converting the translated data into speech, a means for analyzing emotions and reflecting the emotions in translation and speech generation, and a control means for integrating the above means. This enables natural communication that reflects emotions in real time. Furthermore, the system can learn the voice and motion characteristics of specific users and provide recognition accuracy tailored to each individual user. Furthermore, by providing an input mode based on mouth movements, appropriate conversation assistance can be provided to users who have lost their vocal cords.

[1062] A "voice recognition means" is a device or mechanism that receives voice input and analyzes the voice signal to convert it into text data.

[1063] The "motion recognition means" is a device or mechanism that receives video input, analyzes the motions and gestures, and converts them into meaningful data.

[1064] The "means for analyzing voice data and movement data" refers to a device or mechanism that processes the acquired voice data and movement data, extracts its features, and performs further higher-level analysis.

[1065] The "means for translating parsed data into another language" is a device or mechanism for translating parsed text data into a target language.

[1066] The "means for converting translated data into speech" refers to a device or mechanism that converts the translated text data into speech signals and outputs them as speech.

[1067] The "means for analyzing emotions and reflecting them in translation and speech generation" is a device or mechanism that analyzes the emotional state of the user and reflects that emotional information in translation and speech generation.

[1068] The "control means for integrating the means" is a device or mechanism for efficiently coordinating and enabling a plurality of means to function.

[1069] A "means for learning the voice and movement characteristics of a specific user" is a device or mechanism that learns the voice and movement characteristics of a specific user using machine learning or the like, and generates a recognition model optimized for that user.

[1070] The present invention is a system in which a user, a terminal, and a server work together and combine an emotion engine to recognize voice and actions in real time, translate them into other languages, and then speak in a way that reflects the user's emotions.Specific embodiments of the present invention are described below.

[1071] System configuration

[1072] This system uses a dedicated application installed on a smartphone or tablet device (hereafter referred to as "device") and connects to a server located in the cloud. It also incorporates an emotion engine that analyzes the user's emotions from voice and movement data and reflects this in the translation results and speech generation. Using this app, users can communicate with speakers of other languages ​​without having to learn the language.

[1073] Initial Setup and User Registration

[1074] First, a user installs the app on their smartphone and launches it. The device displays an initial setup screen and prompts the user to enter necessary information such as their name, target language, and permission for sentiment analysis. The information entered by the user is sent by the device to a server, which then stores the received user information in a database and generates a unique user identifier.

[1075] Voice and movement learning phase

[1076] Next, the user follows the app's instructions to speak a few short Japanese phrases. The device captures the voice with a microphone and records the mouth movements with a camera. The captured voice and movement data are sent to the server in real time. The server uses a generative AI model to analyze the user's features for voice and movement recognition, and to train a model for converting the voice and movement data into text data.

[1077] Emotion analysis using an emotion engine

[1078] The server uses the received voice data and movement data to analyze the user's emotions with its emotion engine. The analyzed emotion data is then saved along with the user's voice and movement characteristics, allowing the user's emotional state to be properly reflected.

[1079] Real-time speech recognition and translation

[1080] After the user completes the setup, they select the target language and speak in Japanese (e.g., "Hello, what is your name?"). The device captures the voice with a microphone and recognizes mouth movements with a camera. The captured voice and movement data are sent to a server in real time, and the server analyzes the received voice data and converts it into Japanese text. The Japanese text and emotion data are then used to translate it into the target language (e.g., English) (e.g., "Hello, what is your name?"). The translated text data and emotion data are then converted into voice data, generating natural, emotionally-reflective speech. The generated foreign language voice is provided to the user by the device and played through the speaker.

[1081] Conversation assistance functions

[1082] If a user has lost their vocal cords, they can select the mouth movement input mode. The user mimics speech using only mouth movements (e.g., "Onegaishimasu"), and the device captures the mouth movements with a camera and sends the data to the server. The server analyzes the gesture recognition data and converts it into Japanese text (e.g., "Onegaishimasu"). Based on this, a translated target language (e.g., English) is generated (e.g., "Please") to create natural-sounding speech. The generated audio is played back from the device to support the conversation.

[1083] Specific examples

[1084] Example 1: Normal Japanese to English Conversation

[1085] When a user opens the app, selects English mode, and says, "Hello, what is your name?", the device captures the voice and movements and sends them to the server. The server translates this into "Hello, what is your name?" and incorporates emotion data. The translation result is converted into audio data, generating natural-sounding English speech, which the device then plays back to support the conversation.

[1086] Example 2: Speech assistance for users who have lost their vocal cords

[1087] When a user opens the app, selects the lip-activated input mode, and imitates the phrase "onegaishimasu" (please) using only their mouth movements, the device captures the movement and sends it to the server. The server translates this into "please" and incorporates emotional data. The translation result is converted into audio data, generating natural-sounding English speech, which the device then plays back to support the conversation.

[1088] In this way, the system of the present invention learns the user's voice characteristics and mouth movements and analyzes the user's emotions, enabling more natural and emotional communication. This not only effectively removes language barriers, but also provides a realistic conversation experience that reflects the user's emotions.

[1089] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1090] Step 1:

[1091] Initial Setup and User Registration

[1092] The user installs the dedicated app on their smartphone and launches the app.

[1093] The terminal displays an initial setup screen and prompts the user to enter information such as their name, target language, and permission for sentiment analysis.

[1094] The user enters information such as name, target language, and permission for sentiment analysis.

[1095] The terminal transmits the input information to the server (input: user information, output: data transmission to the server).

[1096] The server stores the received user information in a database and generates an identifier for each user (input: user information, output: identifier).

[1097] Step 2:

[1098] Voice and movement learning

[1099] The user speaks a short Japanese phrase as instructed by the app.

[1100] The device captures audio with a microphone and records mouth movements with a camera (input: audio, movement; output: captured data).

[1101] The terminal transmits the captured voice data and movement data to the server in real time (input: captured data, output: data transmission to the server).

[1102] The server analyzes the received data using a generative AI model (deep learning framework) and trains a model that converts voice data and movement data into text data (input: voice data, movement data; output: trained model).

[1103] Step 3:

[1104] Emotion analysis using an emotion engine

[1105] The server analyzes the received voice data and action data using an emotion engine to identify the user's emotion (input: voice data, action data, output: emotion data).

[1106] The server stores the analyzed emotion data in a database together with the user's voice characteristics and behavior characteristics (input: emotion data, output: saved data).

[1107] Step 4:

[1108] Real-time speech recognition and translation

[1109] The user selects the target language and speaks in Japanese (e.g., "Hello, what's your name?").

[1110] The device captures audio with a microphone and recognizes mouth movements with a camera (input: audio, movement; output: captured data).

[1111] The terminal transmits the captured voice data and motion data to the server in real time (input: captured data, output: data transmission to the server).

[1112] The server analyzes the received voice data and converts it into Japanese text (input: voice data, output: Japanese text).

[1113] The server uses the Japanese text and emotion data to translate into the target language (e.g., English: "Hello, what is your name?", input: Japanese text, emotion data, output: translated text).

[1114] The server converts the translated text data and emotional data into audio data, generating natural, emotionally-reflective speech (input: translated text, emotional data; output: foreign language audio).

[1115] The terminal plays the generated foreign language speech from a speaker (input: foreign language speech, output: speech playback).

[1116] Step 5:

[1117] Conversation assistance functions

[1118] If a user loses their vocal cords, they can select mouth input mode in the app.

[1119] The user mimics speech using only mouth movements (e.g., "please").

[1120] The device captures mouth movements with a camera and sends the data to the server (input: movement data, output: sending data to the server).

[1121] The server analyzes the action recognition data and converts it into Japanese text (input: action data, output: Japanese text).

[1122] The server uses the Japanese text and emotion data to translate into the target language (e.g., English: "Please", input: Japanese text, emotion data, output: translated text).

[1123] The server converts the translated text data and emotional data into audio data, generating natural, emotionally-reflective speech (input: translated text, emotional data; output: foreign language audio).

[1124] The terminal plays the generated foreign language speech from a speaker (input: foreign language speech, output: speech playback).

[1125] (Application example 2)

[1126] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1127] In modern society, smooth communication between multiple languages ​​remains a major challenge. Particularly in the food delivery field, it is difficult for foreign customers and the hearing impaired to easily place orders. Another problem is that the translated results do not reflect emotions, making it difficult to provide a natural conversation and user experience.

[1128] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1129] In this invention, the server includes means for analyzing voice data and motion data, means for analyzing emotion data and reflecting the emotion data in translation and speech generation, and means for analyzing, translating, and generating speech using a generative AI model. This makes it possible to capture voice and motion data in real time and provide translation results and speech that reflects emotion.

[1130] The "voice recognition means" is a means for capturing the voice spoken by the user and converting it into text data.

[1131] The "motion recognition means" is a means for capturing and analyzing the user's body movements, particularly the mouth movements.

[1132] The "means for analyzing voice data and motion data" refers to a means for processing the captured voice and motion data and extracting necessary information.

[1133] The "means for translating analyzed data into another language" refers to a means for automatically translating into another language based on the recognized speech and motion data.

[1134] The "means for converting translated data into speech" refers to a means for generating speech data from text data translated into another language.

[1135] The "means for analyzing emotion data and reflecting it in translation and speech generation" is a means for analyzing the user's emotion and reflecting it in translation and speech generation.

[1136] "Means that enable a standalone terminal and a server located on the cloud to work in cooperation" refers to means that enable data exchange and cooperative operation between multiple devices.

[1137] The "means for capturing voice and actions in real time and transmitting them to a server" refers to a means for recognizing the user's voice and actions in real time and immediately transmitting them to a server.

[1138] "Means for analyzing, translating, and generating speech data and movement data using a generative AI model" means means for analyzing speech and movement data and performing translation and speech generation processes using a machine learning model.

[1139] The "means for providing the user with the translation result and the speech that reflects the emotion" is a means for providing the user with the translated text and the speech that reflects the emotion.

[1140] This invention is a system that recognizes voice and actions in real time, translates them into other languages, and speaks in a way that reflects the user's emotions by linking the user, device, and server and combining them with an emotion engine. This system uses a dedicated application installed on a smartphone or tablet device and links with a server located in the cloud. Furthermore, it incorporates an emotion engine that analyzes the user's emotions from voice data and movement data and reflects them in the translation results and speech generation.

[1141] Initial Setup and User Registration

[1142] First, the user installs the application on their smartphone and follows the initial setup screen to enter necessary information such as their name, target language, permission for emotion analysis, etc. This information is sent from the device to the server, where it is stored in a database.

[1143] Voice and movement learning phase

[1144] To improve the accuracy of voice and movement data, users are prompted by the application to speak a few short Japanese phrases. The device then captures the voice with a microphone and records the mouth movements with a camera. This data is then sent in real time to a server, where the generative AI model is trained for voice and movement recognition.

[1145] Emotion analysis using an emotion engine

[1146] The server analyzes the user's emotions using the received voice and movement data. The analyzed emotion data is stored along with the user's voice and movement characteristics. This information is used in subsequent communications.

[1147] Real-time speech recognition and translation

[1148] When a user communicates in real time with a speaker of another language, the device's microphone captures the voice and the camera recognizes the mouth movements. The captured data is sent to a server, where the voice data is recognized and converted into Japanese text. The server then translates the Japanese text and emotion data into the target language and generates emotion-reflecting voice data using a generative AI model. The generated foreign language voice is then provided to the user from the device.

[1149] Conversation assistance functions

[1150] The system also provides a mouth-movement speech input mode for users who have lost their vocal cords. The user mimics speech using only their mouth movements, and the device captures and transmits the movements to the server. The server analyzes the movement data, translates it into the target language, and then generates speech that reflects the emotions.

[1151] Examples of concrete examples and prompts

[1152] Example 1:

[1153] Example: A user says, "I'd like a special hamburger and two cokes, please."

[1154] Example prompt: "I'd like a special hamburger and two colas, please." Translate this into English and speak it in a way that reflects the emotion.

[1155] Example 2:

[1156] Example: A user loses their vocal cords and uses mouth movements to mimic the saying "please."

[1157] Example prompt: Translate "please" into English and say it in a way that reflects your feelings.

[1158] The system is realized using cloud services such as Google Cloud Platform's AutoML, Translate API, and Text-to-Speech API. The generative AI model sends voice and movement data captured on the device to the server in real time and performs various data calculations to enable advanced translation and speech generation. This allows users to enjoy a natural and emotional conversation experience.

[1159] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1160] Step 1:

[1161] The user launches the application and enters the necessary information, such as their name, target language, and permission for sentiment analysis, on the initial setup screen. The entered information is sent from the device to the server, which stores it in a database and generates a user identifier. Once this initial setup process for user information is complete, the application is ready to proceed to the next step.

[1162] Step 2:

[1163] In the voice and movement learning phase, the user speaks a few short Japanese phrases according to the application's instructions. The device uses a microphone to capture voice data and a camera to record mouth movement data. The captured voice data and movement data are sent to the server in real time.

[1164] Step 3:

[1165] The server trains a generative AI model based on the received voice and motion data, learning the user's voice and motion characteristics. This process involves analyzing the data (voice recognition, motion recognition) and converting it into text. The resulting model is then used to perform voice and motion recognition optimized for each individual user.

[1166] Step 4:

[1167] The server analyzes the user's emotions from the voice data and movement data. The server stores the analyzed emotion data along with the user's voice characteristics and movement characteristics. For example, if the user says "please," the tone of voice and mouth movements are analyzed and stored as polite emotion data. This data will be used in a later step.

[1168] Step 5:

[1169] This is a real-time speech recognition and translation process. The user speaks Japanese into the device's microphone to translate the sentence into the target language. The device captures the speech and uses a camera to recognize mouth movements. This data is sent to the server in real time.

[1170] Step 6:

[1171] The server analyzes the received voice data and converts it into Japanese text. It then uses a generative AI model to translate the text based on the stored emotion data. For example, the Japanese text "Two special hamburgers and two Cokes, please" is translated into "Two special hamburgers and two Cokes, please."

[1172] Step 7:

[1173] Based on the translated text data and emotion data, the server uses a generative AI model to generate emotion-reflecting voice data, which is then sent to the device in real time.

[1174] Step 8:

[1175] The device receives the foreign language audio from the server and plays it back from the speaker, allowing the user to have a natural speaking experience and enabling multilingual communication.

[1176] Step 9:

[1177] This conversation assistance function for users who have lost their vocal cords imitates speech using only mouth movements. The user selects the mouth input mode and imitates speech using mouth movements. The device captures the movements and sends them to the server.

[1178] Step 10:

[1179] The server analyzes the motion data and converts it into Japanese text. It then uses a generative AI model to translate this text and emotion data into other languages ​​and generate audio data. The generated audio data is then sent to the device and played through the speaker.

[1180] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1181] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1182] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1183] [Third embodiment]

[1184] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1185] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1186] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1187] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1188] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1189] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1190] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1191] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1192] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1193] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1194] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1195] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1196] The present invention is a system in which a user, a terminal, and a server work together to recognize voices and actions in real time, convert them into another language, and then speak the voices. Specific embodiments of the present invention are described below.

[1197] System Overview

[1198] This system basically uses a dedicated application installed on a smartphone or tablet device (hereinafter referred to as "device") and connects to a server located in the cloud. Using this application, users can communicate with speakers of other languages ​​without having to learn the foreign language.

[1199] Initial Setup and User Registration

[1200] 1. User: First, install the app on your smartphone and launch it.

[1201] 2. Device: When the app launches, an initial setup screen appears, prompting the user to enter necessary information (such as name and desired language).

[1202] 3. Terminal: Sends the information entered by the user to the server.

[1203] 4. Server: Stores the received user information in a database and generates an identifier corresponding to each individual user.

[1204] Voice and movement learning phase

[1205] 1. User: Follow the instructions of the app and speak a few short Japanese phrases.

[1206] 2. Device: A microphone captures audio and a camera records mouth movements.

[1207] 3. Terminal: Sends voice data and mouth movement data to the server in real time.

[1208] 4. Server: The AI ​​that generates speech and motion recognition analyzes the user's characteristics and learns a model for converting speech and motion data into text data.

[1209] Real-time speech recognition and translation

[1210] 1. User: After completing the setup, select the target language and speak in Japanese (e.g., "Hello, what's your name?").

[1211] 2. Device: The microphone captures the user's voice and the camera recognizes mouth movements.

[1212] 3. Terminal: Sends the captured voice and movement data to the server.

[1213] 4. Server: Analyzes the received data and converts it into Japanese text.

[1214] 5. Server: Translate the Japanese text into the target language (e.g., English).

[1215] 6. Server: Converts the translated text data into audio data and generates natural-sounding speech.

[1216] 7. Terminal: The generated foreign language speech is provided to the user and played back through the speaker.

[1217] Conversation assistance functions

[1218] 1. User: If you have lost your vocal cords, select mouth input mode.

[1219] 2. User: Imitate speech using mouth movements only (e.g., "please").

[1220] 3. Device: The camera captures mouth movements and sends the data to the server.

[1221] 4. Server: Analyzes the motion recognition data and converts it into Japanese text.

[1222] 5. Server: Translate the Japanese text into the target language.

[1223] 6. Server: Converts the translated text data into audio data and generates natural-sounding speech.

[1224] 7. Terminal: Plays back the generated foreign language speech and provides it to the user.

[1225] Specific examples

[1226] Example 1: Normal Japanese to English Conversation

[1227] 1. User: Open the app, select English mode and say "Hello, what's your name?"

[1228] 2. Device: Captures voice and movements and sends them to the server.

[1229] 3. Server: Translate this into "Hello, what is your name?" and generate it as voice data.

[1230] 4. Terminal: Plays the generated English audio and supports the conversation.

[1231] Example 2: Speech assistance for users who have lost their vocal cords

[1232] 1. User: Open the app and select mouth input mode.

[1233] 2. User: Imitate the word "please" using only mouth movements.

[1234] 3. Device: Captures the action and sends it to the server.

[1235] 4. Server: Translate this into "Please" and generate it as voice data.

[1236] 5. Terminal: Plays the generated English audio and supports the conversation.

[1237] The system of the present invention learns the user's voice characteristics and mouth movements to enable natural conversations that transcend language barriers. It also has a function that notifies the user of an error if a problem occurs with voice capture or translation, prompting the user to take appropriate action. This allows the system to efficiently overcome language barriers, even in situations where communication is difficult, such as sudden overseas business trips.

[1238] The processing flow will be explained below.

[1239] Step 1:

[1240] User: Install the dedicated app on your smartphone and launch it.

[1241] Step 2:

[1242] Terminal: Displays an initial setup screen and prompts the user to enter necessary information such as name and desired language.

[1243] Step 3:

[1244] User: Enter the required information on the initial setup screen and press the "Next" button after completing the entry.

[1245] Step 4:

[1246] Terminal: Sends the information entered by the user to the server.

[1247] Step 5:

[1248] Server: Stores the received user information in a database and generates a unique user identifier.

[1249] Step 6:

[1250] Terminal: Displays prompts to the user to begin the voice and movement learning phase.

[1251] Step 7:

[1252] User: Follow the app's instructions and speak a few short Japanese phrases.

[1253] Step 8:

[1254] Device: A microphone captures your voice and a camera records your mouth movements.

[1255] Step 9:

[1256] Terminal: Sends voice and movement data to the server in real time.

[1257] Step 10:

[1258] Server: Analyzes the received voice and movement data and learns the characteristics of the user's voice and movements.

[1259] Step 11:

[1260] Server: The generated analysis model is linked to the user's account and saved.

[1261] Step 12:

[1262] User: After the initial setup and learning phase is complete, select your target language on the app's main screen.

[1263] Step 13:

[1264] User: Speak in Japanese (e.g. "Hello, what's your name?").

[1265] Step 14:

[1266] Device: A microphone captures the user's voice and a camera recognizes mouth movements.

[1267] Step 15:

[1268] Terminal: Transmits captured audio and motion data to the server in real time.

[1269] Step 16:

[1270] Server: Analyzes the received voice data and converts it into Japanese text (e.g., "Hello, what's your name?").

[1271] Step 17:

[1272] Server: Translate Japanese text into the target language (e.g., English) (e.g., "Hello, what is your name?").

[1273] Step 18:

[1274] Server: Converts the translated text data into audio data and generates natural-sounding speech.

[1275] Step 19:

[1276] Terminal: Receives the generated foreign language audio and plays it back through the speaker.

[1277] Step 20:

[1278] User: Listen to the foreign language audio and continue communication.

[1279] Step 21:

[1280] Terminal: If an error is detected, an error message is displayed to the user.

[1281] Step 22:

[1282] User: Follow the error message and make any necessary corrections or try again.

[1283] Step 23:

[1284] User: If you have lost your vocal cords, select mouth input mode.

[1285] Step 24:

[1286] User: Mimic speech using mouth movements only (e.g., "please").

[1287] Step 25:

[1288] Device: The camera captures mouth movements and sends the data to the server.

[1289] Step 26:

[1290] Server: Analyzes the action recognition data and converts it into Japanese text (e.g., "Onegaishimasu").

[1291] Step 27:

[1292] Server: Translate Japanese text into the target language (e.g., English) (e.g., "Please").

[1293] Step 28:

[1294] Server: Converts the translated text data into audio data and generates natural-sounding speech.

[1295] Step 29:

[1296] Terminal: Receives the generated foreign language audio and plays it back through the speaker.

[1297] Step 30:

[1298] User: Listen to the foreign language audio and continue communication.

[1299] Example 1

[1300] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1301] In today's globalized society, communication between people who speak different languages ​​is becoming increasingly important. However, the existence of language barriers makes smooth communication difficult. Furthermore, people with certain physical limitations (e.g., those who have lost their vocal cords) cannot use standard speech recognition technology, making communication even more difficult. To solve these challenges, there is a need for a system that can recognize voices and movements in real time, convert them into speech in other languages, and then speak them.

[1302] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1303] In this invention, the server includes a voice recognition unit, a motion recognition unit, a unit for analyzing voice data and motion data, a unit for translating the analyzed data into another language, and a unit for converting the translated data into voice. This allows the user to perform initial setup, and a unit for saving and managing the setup information, a unit for capturing voice and motion and transmitting them to the server in real time, and a unit for the server to provide the user with the analysis results, enabling smooth communication between people who speak different languages. Furthermore, even people with certain physical limitations can have natural conversations using mouth movements.

[1304] "Speech recognition means" is a general term for hardware and software for capturing speech as digital data and analyzing its content.

[1305] "Movement recognition means" is a general term for hardware and software for capturing the user's body movements, particularly mouth movements, and analyzing those movements.

[1306] The "means for analyzing voice data and movement data" is a mechanism for analyzing the captured voice data and movement data using an algorithm and converting it into text data.

[1307] "Means for translating analyzed data into another language" is a general term for hardware and software for translating analyzed text data into another specified language.

[1308] "Means for converting translated data into speech" is a general term for hardware and software for converting translated text data into speech data and generating natural-sounding speech.

[1309] The "control means for integrating the above means" is a control system for integrally operating the voice recognition means, the motion recognition means, the analysis means, the translation means, and the voice conversion means.

[1310] "Means for users to perform initial setup and save and manage setup information" is a general term for the interface and database that allows users to perform initial setup of the system and save and manage that information.

[1311] "Means for capturing voice and movement and transmitting it to a server in real time" is a general term for software and network communication means for acquiring voice and movement data, compressing it, and transmitting it to a server.

[1312] "Means by which the server provides the analysis results to the user" refers to a mechanism by which the server returns the analysis and translation results to the user's terminal and displays or plays them aloud.

[1313] The present invention is a system in which a user, a terminal, and a server work together to recognize voices and actions in real time, convert them into another language, and then speak the voices. Specific embodiments of the present invention are described below.

[1314] System Overview

[1315] This system basically uses a dedicated application installed on a smartphone or tablet device (hereinafter referred to as "device") and connects to a server located in the cloud. Using this application, users can communicate with speakers of other languages ​​without having to learn the foreign language.

[1316] Initial Setup and User Registration

[1317] 1. User: Installs the app on a smartphone and launches it. The app can be downloaded from the Google Play Store or the App Store.

[1318] 2. Device: When the app launches, an initial setup screen appears, prompting the user to enter information such as their name, preferred language, and email address. Once the information is entered, the device encrypts and sends it to the server.

[1319] 3. Server: Stores the received user information in a database and generates a unique identifier that is sent back to the device for future operations and data management.

[1320] Voice and movement learning phase

[1321] 1. User: Follow the instructions of the app to speak a few short Japanese phrases, such as "hello" and "thank you."

[1322] 2. Device: A microphone captures audio and a camera records lip movements. Audio and video are synchronized to collect precise data.

[1323] 3. Terminal: Collected voice and movement data is compressed in real time and sent to the server.

[1324] 4. Server: Analyzes and learns voice and movement features using generative AI models, and optimizes the model to convert this data into text.

[1325] Real-time speech recognition and translation

[1326] 1. User: After completing the setup, select the target language and speak in Japanese. For example, the user selects "English mode" and speaks "Hello, what is your name?"

[1327] 2. Device: The microphone captures the voice and the camera recognizes the mouth movements. This data is sent to the server in real time.

[1328] 3. Server: Analyzes the received voice and motion data and converts it into Japanese text. Then, translates the Japanese text into the target language (e.g., English). For example, "Hello, what is your name?" is converted to "Hello, what is your name?"

[1329] 4. Server: Converts the translated text data into audio data and generates natural-sounding speech.

[1330] 5. Terminal: The generated foreign language speech is provided to the user and played back through the speaker.

[1331] Conversation assistance functions

[1332] 1. User: If you have lost your vocal cords, select "Mouth Recognition Mode" from the app settings.

[1333] 2. User: Imitate speech using only mouth movements, for example, recreating the mouth movements for "please."

[1334] 3. Device: The camera captures mouth movements and sends the data to the server.

[1335] 4. Server: Analyzes the motion recognition data and converts it into Japanese text. Next, translates "onegaishimasu" into "Please."

[1336] 5. Server: Converts the translated text data into audio data and generates natural-sounding speech.

[1337] 6. Terminal: Plays back the generated foreign language speech and provides it to the user.

[1338] Examples and prompts

[1339] Example 1: Normal Japanese to English Conversation

[1340] 1. User: Open the app, select English mode and say "Hello, what's your name?"

[1341] 2. Device: Captures voice and movements and sends them to the server.

[1342] 3. Server: Translate this into "Hello, what is your name?" and generate it as voice data.

[1343] 4. Terminal: Plays the generated English audio and supports the conversation.

[1344] Example 2: Speech assistance for users who have lost their vocal cords

[1345] 1. User: Open the app and select mouth input mode.

[1346] 2. User: Imitate the word "please" using only mouth movements.

[1347] 3. Device: Captures the action and sends it to the server.

[1348] 4. Server: Translate this into "Please" and generate it as voice data.

[1349] 5. Terminal: Plays the generated English audio and supports the conversation.

[1350] As an example of a prompt sentence, if you enter "Translate and speak the Japanese greeting into English," the Japanese "Konnichiwa" will be translated and spoken as the English "Hello."

[1351] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1352] Step 1:

[1353] User: Installs the app on their smartphone and launches it.

[1354] Input: Download the app file from Google Play Store or App Store.

[1355] How it works: Press the app's install button, then tap the app icon to launch it after installation is complete.

[1356] Output: The initial setup screen is displayed on the terminal.

[1357] Step 2:

[1358] Device: An initial setup screen appears, prompting the user to enter information such as name, preferred language, and email address.

[1359] Input: The user enters their name, desired language, email address, etc.

[1360] Action: Enter the required information in the input form and press the "Submit" button.

[1361] Output: The input information is encrypted and sent from the device to the server.

[1362] Step 3:

[1363] Server: Stores the received user information in a database and generates and returns an identifier.

[1364] Input: User information data entered.

[1365] What it does: Connects to a database, stores the received data, and generates a unique identifier.

[1366] Output: The generated identifier is sent back to the terminal.

[1367] Step 4:

[1368] User: Follow the app's instructions and speak a few short Japanese phrases.

[1369] Input: A phrase to display on the app screen (e.g. "Hello", "Thank you").

[1370] Action: Reads the displayed text aloud.

[1371] Output: The user's voice and mouth movement data is generated.

[1372] Step 5:

[1373] Device: A microphone captures your voice and a camera records your mouth movements.

[1374] Input: The user's voice and mouth movements.

[1375] Movement: High-definition microphones and high-resolution cameras capture audio waveforms and movement data.

[1376] Output: Captured audio and movement data.

[1377] Step 6:

[1378] Terminal: The captured voice data and movement data are compressed in real time and sent to the server.

[1379] Input: Audio and motion data.

[1380] What it does: Compresses data and sends it to the server with low latency.

[1381] Output: Audio and movement data received by the server.

[1382] Step 7:

[1383] Server: Analyzes and learns voice and movement characteristics using a generative AI model.

[1384] Input: Received voice and movement data.

[1385] How it works: The generative AI model extracts and analyzes features, optimizing the model for conversion to text data.

[1386] Output: Optimized speech-to-motion translation model.

[1387] Step 8:

[1388] User: After completing the setup, select the target language and speak in Japanese.

[1389] Input: The target language selected on the app screen, the words you speak (e.g. "Hello, what's your name?").

[1390] Action: Select the target language and speak in Japanese.

[1391] Output: Spoken Japanese audio.

[1392] Step 9:

[1393] Device: A microphone captures the user's voice and a camera recognizes mouth movements.

[1394] Input: Spoken speech and mouth movements.

[1395] How it works: Captures audio waveforms in real time and records mouth movements with a camera.

[1396] Output: Captured audio and movement data.

[1397] Step 10:

[1398] Terminal: Sends captured audio and motion data to the server.

[1399] Input: Audio and motion data.

[1400] Operation: Real-time transmission of data.

[1401] Output: Audio and motion data received by the server.

[1402] Step 11:

[1403] Server: Analyzes the received voice data and movement data and converts it into Japanese text.

[1404] Input: Audio and motion data.

[1405] Actions: The generative AI model analyzes voice and actions and converts them into Japanese text data.

[1406] Output: Converted Japanese text data.

[1407] Step 12:

[1408] Server: Translates Japanese text into the target language.

[1409] Input: Japanese text.

[1410] Behavior: Translates to the specified target language (e.g., English).

[1411] Output: The translated text data (e.g. "Hello, what is your name?").

[1412] Step 13:

[1413] Server: Converts the translated text data into audio data and generates natural-sounding speech.

[1414] Input: Translated text data.

[1415] How it works: Generates natural-sounding speech using generative AI models.

[1416] Output: The generated audio data.

[1417] Step 14:

[1418] Terminal: Provides the generated foreign language speech to the user and plays it back through a speaker.

[1419] Input: The generated audio data.

[1420] Action: Plays audio through a playback device.

[1421] Output: Foreign language speech played through the speaker (e.g. "Hello, what is your name?").

[1422] Step 15:

[1423] User: If you have lost your vocal cords, select "Mouth Recognition Mode" from the app settings.

[1424] Input: Select "Mouth Recognition Mode" on the app's settings screen.

[1425] Behavior: Toggles selection mode.

[1426] Output: Selected mode.

[1427] Step 16:

[1428] User: Mimics speech using mouth movements only.

[1429] Input: Mouth movements (e.g. "please").

[1430] Action: Mimics mouth movements.

[1431] Output: Mouth movement data.

[1432] Step 17:

[1433] Device: The camera captures mouth movements and sends the data to the server.

[1434] Input: Mouth movement data.

[1435] Movement: A high-precision camera captures mouth movements and transmits the data.

[1436] Output: Lip movement data sent to the server.

[1437] Step 18:

[1438] Server: Analyzes the motion recognition data and converts it into Japanese text.

[1439] Input: Captured mouth movement data.

[1440] Movement: Movement recognition data is analyzed and converted into Japanese text.

[1441] Output: Japanese text (e.g. "Onegaishimasu").

[1442] Step 19:

[1443] Server: Translates Japanese text into the target language.

[1444] Input: Japanese text.

[1445] Behavior: Translates to the specified target language (e.g., English).

[1446] Output: The translated text data (e.g. "Please").

[1447] Step 20:

[1448] Server: Converts the translated text data into audio data and generates natural-sounding speech.

[1449] Input: Translated text data.

[1450] How it works: Generates natural-sounding speech using generative AI models.

[1451] Output: The generated audio data.

[1452] Step 21:

[1453] Terminal: Plays back the generated foreign language speech and provides it to the user.

[1454] Input: The generated audio data.

[1455] Action: Plays audio through a playback device.

[1456] Output: Foreign language voice (e.g. "Please") played through the speaker.

[1457] (Application example 1)

[1458] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1459] Conventional food delivery systems face a language barrier when customers who speak a foreign language place an order, making it difficult to accurately convey order information. This leads to frequent ordering errors and communication problems with customers, resulting in a decline in service quality. Furthermore, only a limited number of systems allow input using mouth movements in addition to voice input, making them inadequate for specific users. To solve these problems, a means was needed to translate and speak order details in real time across language barriers, enabling smooth communication.

[1460] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1461] In this invention, the server includes a voice recognition means, a motion recognition means, a means for analyzing the voice data and motion data, a means for providing the translated voice data as order information to the delivery person or the store, and a control means for integrating the above means. This enables accurate real-time transmission of order information between customers who speak a foreign language and the delivery person or the store, realizing smooth communication that transcends language barriers.

[1462] A "voice recognition means" is a device or technology that receives voice as input and converts it into digital data that can be analyzed.

[1463] "Movement recognition means" refers to a device or technology that captures a user's physical movements, particularly facial and mouth movements, using a sensor such as a camera, and converts them into digital data that can be analyzed.

[1464] "Means for analyzing voice data and movement data" refers to software or hardware that analyzes data based on input voice or movement and converts it into text data or other formats.

[1465] "Means for translating the analyzed data into another language" refers to software or algorithms for translating the analyzed text data into a target language.

[1466] "Means for converting translated data into audio" refers to a device or software that converts translated text data into audio data and outputs it as audio.

[1467] "Means for providing translated voice data as order information to delivery personnel and stores" refers to technology and devices that use translated voice data to transmit accurate order information to the delivery system.

[1468] "Control means" refers to software and hardware that integrates and manages the above means and controls the entire system to ensure smooth operation.

[1469] The present invention provides a system that recognizes speech and gestures in real time and translates them into other languages ​​for speech, which is particularly useful for food delivery services to assist customers who speak other languages ​​in placing orders.

[1470] System configuration

[1471] Hardware Configuration

[1472] The system includes the following hardware:

[1473] Device: Smartphone or tablet

[1474] Microphone: A device used to capture the user's voice.

[1475] Camera: A device for capturing the user's mouth movements.

[1476] Speaker: A device for playing the translated audio.

[1477] Software Configuration

[1478] The system uses the following software:

[1479] Speech Recognition Library: speech_recognition library

[1480] Action recognition library: OpenCV (cv2)

[1481] Translation API: Google Cloud Translation API

[1482] Speech synthesis API: gTTS (Google Text-to-Speech)

[1483] System Operation

[1484] Speech and gesture recognition

[1485] When a user speaks, the device's microphone captures the voice and the camera records the mouth movements. This information is converted into text data using a speech recognition library (speech_recognition) and a motion recognition library (OpenCV). The data captured by the speech recognition and motion recognition means is processed in real time.

[1486] Data analysis and translation

[1487] The server analyzes the voice and motion data in the cloud, converts the user's speech into text using the analysis method, and then translates the analyzed text into the target language using the Google Cloud Translation API.

[1488] Speech synthesis of translation data

[1489] The translated text data is converted into voice data using the gTTS (Google Text-to-Speech) API. The voice data generated using a voice synthesis tool is played back on the device's speaker. This allows users to smoothly place orders through food delivery services without having to learn a foreign language.

[1490] Specific examples

[1491] Order Scenarios

[1492] 1. User: Download the smartphone app and select the language you want to use (e.g., English).

[1493] 2. User: "Hello, I'd like one Margherita pizza please" in Japanese.

[1494] 3. Device: Captures voice and lip movements and sends the data to the server.

[1495] 4. Server: Translates the speech data to "Hello, one Margherita pizza please" and converts it to speech data.

[1496] 5. Terminal: Plays back the translated voice data and provides it to delivery personnel and stores.

[1497] Prompt Sentence Examples

[1498] "How can I develop a food delivery app that translates Japanese speech into English in real time and provides it in a way that delivery staff and restaurants can understand?"

[1499] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1500] Step 1:

[1501] Input: The user launches the app on their smartphone and selects the language they want to use (e.g., English).

[1502] Specific operation: The user selects the desired language on the app's settings screen and enters their information (name, address, desired language, etc.).

[1503] Output: Configuration information is sent from the device to the server.

[1504] Step 2:

[1505] Input: User speaks in Japanese (e.g., "Hello, I'd like one Margherita pizza please.").

[1506] What it does: The device's microphone captures the user's voice and the camera records their mouth movements.

[1507] Output: Audio and motion data are generated.

[1508] Step 3:

[1509] Input: Audio and motion data captured by the device.

[1510] Specific operation: Using the speech recognition library (speech_recognition), converts voice data into text data. It also analyzes mouth movement data recorded by a camera.

[1511] Output: Japanese text data is generated and sent to the server.

[1512] Step 4:

[1513] Input: Japanese text data received by the server.

[1514] What it does: It uses the Google Cloud Translation API in the cloud to translate text data into the desired language (e.g., English).

[1515] Output: The translated text data is generated.

[1516] Step 5:

[1517] Input: Translated text data.

[1518] What it does: Uses the gTTS (Google Text-to-Speech) API to convert the translated text data into audio data.

[1519] Output: The translated audio data is generated.

[1520] Step 6:

[1521] Input: Translated audio data.

[1522] Specific operation: The device's speaker plays the generated voice data, which is then provided to the delivery person or the store.

[1523] Output: The translated content is played back as audio, and accurate order information is communicated to the delivery person or store.

[1524] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1525] The present invention is a system that recognizes voice and actions in real time, translates them into other languages, and speaks in a way that reflects the user's emotions by linking the user, terminal, and server and combining them with an emotion engine. Specific embodiments of the present invention are described below.

[1526] System configuration

[1527] This system uses a dedicated application installed on a smartphone or tablet device (hereafter referred to as "device") and connects to a server located in the cloud. It also incorporates an emotion engine that analyzes the user's emotions from voice and movement data and reflects this in the translation results and speech generation. Using this app, users can communicate with speakers of other languages ​​without having to learn the language.

[1528] Initial Setup and User Registration

[1529] 1. User: First, install the app on your smartphone and launch it.

[1530] 2. Device: Displays the initial setup screen and prompts the user to enter necessary information such as their name, target language, and permission for sentiment analysis.

[1531] 3. Terminal: Sends the information entered by the user to the server.

[1532] 4. Server: Stores the received user information in a database and generates a unique user identifier.

[1533] Voice and movement learning phase

[1534] 1. User: Follow the instructions of the app and speak a few short Japanese phrases.

[1535] 2. Device: A microphone captures audio and a camera records mouth movements.

[1536] 3. Terminal: Sends voice data and mouth movement data to the server in real time.

[1537] 4. Server: The AI ​​that generates speech and motion recognition analyzes the user's characteristics and learns a model for converting speech and motion data into text data.

[1538] Emotion analysis using an emotion engine

[1539] 1. Server: Using the received voice data and movement data, the emotion engine analyzes the user's emotions.

[1540] 2. Server: Stores the analyzed emotion data along with the user's voice and movement characteristics.

[1541] Real-time speech recognition and translation

[1542] 1. User: After completing the setup, select the target language and speak in Japanese (e.g., "Hello, what's your name?").

[1543] 2. Device: The microphone captures audio and the camera recognizes mouth movements.

[1544] 3. Terminal: Transmits captured voice and movement data to the server in real time.

[1545] 4. Server: Analyzes the received voice data and converts it into Japanese text (e.g., "Hello, what's your name?").

[1546] 5. Server: Uses Japanese text and emotion data to translate into the target language (e.g., English) (e.g., "Hello, what is your name?").

[1547] 6. Server: Based on the translated text data and emotion data, it converts it into voice data and generates natural, emotionally-informed speech.

[1548] 7. Terminal: The generated foreign language speech is provided to the user and played back through the speaker.

[1549] Conversation assistance functions

[1550] 1. User: If you have lost your vocal cords, select mouth input mode.

[1551] 2. User: Imitate speech using mouth movements only (e.g., "please").

[1552] 3. Device: The camera captures mouth movements and sends the data to the server.

[1553] 4. Server: Analyzes the action recognition data and converts it into Japanese text (e.g., "Onegaishimasu").

[1554] 5. Server: Uses the Japanese text and emotion data to translate into the target language (e.g., English) (e.g., "Please").

[1555] 6. Server: Based on the translated text data and emotion data, it converts it into voice data and generates natural, emotionally-informed speech.

[1556] 7. Terminal: Receives the generated foreign language audio and plays it back through the speaker.

[1557] Specific examples

[1558] Example 1: Normal Japanese to English Conversation

[1559] 1. User: Open the app, select English mode and say "Hello, what's your name?"

[1560] 2. Device: Captures voice and movements and sends them to the server.

[1561] 3. Server: Translate this to "Hello, what is your name?" and incorporate sentiment data.

[1562] 4. Server: Converts the translation results into audio data and generates natural speech.

[1563] 5. Terminal: Plays the generated English audio and supports the conversation.

[1564] Example 2: Speech assistance for users who have lost their vocal cords

[1565] 1. User: Open the app and select mouth input mode.

[1566] 2. User: Imitate the word "please" using only mouth movements.

[1567] 3. Device: Captures the action and sends it to the server.

[1568] 4. Server: Translate this to "Please" and incorporate emotional data.

[1569] 5. Server: Converts the translation results into audio data and generates natural speech.

[1570] 6. Terminal: Plays the generated English audio and supports conversation.

[1571] In this way, the system of the present invention learns the user's voice characteristics and mouth movements and analyzes the user's emotions, enabling more natural and emotional communication. This not only effectively removes language barriers, but also provides a realistic conversation experience that reflects the user's emotions.

[1572] The processing flow will be explained below.

[1573] Step 1:

[1574] User: Install the dedicated app on your smartphone and launch it.

[1575] Step 2:

[1576] Device: Displays an initial setup screen and prompts the user to enter required information such as name, target language, and permission for sentiment analysis.

[1577] Step 3:

[1578] User: Enter the required information on the initial setup screen and press the "Next" button after completing the entry.

[1579] Step 4:

[1580] Terminal: Sends the information entered by the user to the server.

[1581] Step 5:

[1582] Server: Stores the received user information in a database and generates a unique user identifier.

[1583] Step 6:

[1584] Terminal: Displays prompts to the user to begin the voice and movement learning phase.

[1585] Step 7:

[1586] User: Follow the app's instructions and speak a few short Japanese phrases.

[1587] Step 8:

[1588] Device: A microphone captures your voice and a camera records your mouth movements.

[1589] Step 9:

[1590] Terminal: Sends voice and movement data to the server in real time.

[1591] Step 10:

[1592] Server: Analyzes the received voice and movement data and learns the characteristics of the user's voice and movements.

[1593] Step 11:

[1594] Server: The generated analysis model is linked to the user's account and saved.

[1595] Step 12:

[1596] Server: Analyzes the user's emotions using an emotion engine based on voice data and movement data.

[1597] Step 13:

[1598] Server: Stores the analyzed emotion data along with the user's voice and movement characteristics.

[1599] Step 14:

[1600] User: After completing the initial setup and learning phase, select the desired language on the app's main screen.

[1601] Step 15:

[1602] User: Speak in Japanese (e.g. "Hello, what's your name?").

[1603] Step 16:

[1604] Device: A microphone captures audio and a camera recognizes mouth movements.

[1605] Step 17:

[1606] Terminal: Transmits captured audio and motion data to the server in real time.

[1607] Step 18:

[1608] Server: Analyzes the received voice data and converts it into Japanese text (e.g., "Hello, what's your name?").

[1609] Step 19:

[1610] Server: Uses Japanese text and emotion data to translate into the target language (e.g., English) (e.g., "Hello, what is your name?").

[1611] Step 20:

[1612] Server: Based on the translated text data and emotion data, converts it into voice data and generates natural speech that reflects emotion.

[1613] Step 21:

[1614] Terminal: Receives the generated foreign language audio and plays it back through the speaker.

[1615] Step 22:

[1616] User: Listen to the foreign language audio and continue communicating with the other person.

[1617] Step 23:

[1618] Terminal: If an error is detected, an error message is displayed to the user.

[1619] Step 24:

[1620] User: Follow the error message and make any necessary corrections or try again.

[1621] Step 25:

[1622] User: If you have lost your vocal cords, select mouth input mode.

[1623] Step 26:

[1624] User: Mimic speech using mouth movements only (e.g., "please").

[1625] Step 27:

[1626] Device: The camera captures mouth movements and sends the data to the server.

[1627] Step 28:

[1628] Server: Analyzes the action recognition data and converts it into Japanese text (e.g., "Onegaishimasu").

[1629] Step 29:

[1630] Server: Uses Japanese text and emotion data to translate into the target language (e.g., English) (e.g., "Please").

[1631] Step 30:

[1632] Server: Based on the translated text data and emotion data, converts it into voice data and generates natural speech that reflects emotion.

[1633] Step 31:

[1634] Terminal: Receives the generated foreign language audio and plays it back through the speaker.

[1635] Step 32:

[1636] User: Listen to the foreign language audio and continue communicating with the other person.

[1637] Example 2

[1638] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1639] Conventional speech recognition and translation systems are unable to recognize speech and actions in real time and reflect emotions when translating into other languages, making it difficult to achieve natural, emotionally rich communication. Furthermore, they are unable to individually learn the speech and action characteristics of specific users, making it difficult to provide recognition accuracy tailored to each individual user. Furthermore, they lack the functionality to accommodate special situations, such as users who have lost their vocal cords.

[1640] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a voice recognition means, a motion recognition means, a means for analyzing voice data and motion data, a means for translating the analyzed data into another language, a means for converting the translated data into speech, a means for analyzing emotions and reflecting the emotions in translation and speech generation, and a control means for integrating the above means. This enables natural communication that reflects emotions in real time. Furthermore, the system can learn the voice and motion characteristics of specific users and provide recognition accuracy tailored to each individual user. Furthermore, by providing an input mode based on mouth movements, appropriate conversation assistance can be provided to users who have lost their vocal cords.

[1641] A "voice recognition means" is a device or mechanism that receives voice input and analyzes the voice signal to convert it into text data.

[1642] The "motion recognition means" is a device or mechanism that receives video input, analyzes the motions and gestures, and converts them into meaningful data.

[1643] The "means for analyzing voice data and movement data" refers to a device or mechanism that processes the acquired voice data and movement data, extracts its features, and performs further higher-level analysis.

[1644] The "means for translating parsed data into another language" is a device or mechanism for translating parsed text data into a target language.

[1645] The "means for converting translated data into speech" refers to a device or mechanism that converts the translated text data into speech signals and outputs them as speech.

[1646] The "means for analyzing emotions and reflecting them in translation and speech generation" is a device or mechanism that analyzes the emotional state of the user and reflects that emotional information in translation and speech generation.

[1647] The "control means for integrating the means" is a device or mechanism for efficiently coordinating and enabling a plurality of means to function.

[1648] A "means for learning the voice and movement characteristics of a specific user" is a device or mechanism that learns the voice and movement characteristics of a specific user using machine learning or the like, and generates a recognition model optimized for that user.

[1649] The present invention is a system in which a user, a terminal, and a server work together and combine an emotion engine to recognize voice and actions in real time, translate them into other languages, and then speak in a way that reflects the user's emotions.Specific embodiments of the present invention are described below.

[1650] System configuration

[1651] This system uses a dedicated application installed on a smartphone or tablet device (hereafter referred to as "device") and connects to a server located in the cloud. It also incorporates an emotion engine that analyzes the user's emotions from voice and movement data and reflects this in the translation results and speech generation. Using this app, users can communicate with speakers of other languages ​​without having to learn the language.

[1652] Initial Setup and User Registration

[1653] First, a user installs the app on their smartphone and launches it. The device displays an initial setup screen and prompts the user to enter necessary information such as their name, target language, and permission for sentiment analysis. The information entered by the user is sent by the device to a server, which then stores the received user information in a database and generates a unique user identifier.

[1654] Voice and movement learning phase

[1655] Next, the user follows the app's instructions to speak a few short Japanese phrases. The device captures the voice with a microphone and records the mouth movements with a camera. The captured voice and movement data are sent to the server in real time. The server uses a generative AI model to analyze the user's features for voice and movement recognition, and to train a model for converting the voice and movement data into text data.

[1656] Emotion analysis using an emotion engine

[1657] The server uses the received voice data and movement data to analyze the user's emotions with its emotion engine. The analyzed emotion data is then saved along with the user's voice and movement characteristics, allowing the user's emotional state to be properly reflected.

[1658] Real-time speech recognition and translation

[1659] After the user completes the setup, they select the target language and speak in Japanese (e.g., "Hello, what is your name?"). The device captures the voice with a microphone and recognizes mouth movements with a camera. The captured voice and movement data are sent to a server in real time, and the server analyzes the received voice data and converts it into Japanese text. The Japanese text and emotion data are then used to translate it into the target language (e.g., English) (e.g., "Hello, what is your name?"). The translated text data and emotion data are then converted into voice data, generating natural, emotionally-reflective speech. The generated foreign language voice is provided to the user by the device and played through the speaker.

[1660] Conversation assistance functions

[1661] If a user has lost their vocal cords, they can select the mouth movement input mode. The user mimics speech using only mouth movements (e.g., "Onegaishimasu"), and the device captures the mouth movements with a camera and sends the data to the server. The server analyzes the gesture recognition data and converts it into Japanese text (e.g., "Onegaishimasu"). Based on this, a translated target language (e.g., English) is generated (e.g., "Please") to create natural-sounding speech. The generated audio is played back from the device to support the conversation.

[1662] Specific examples

[1663] Example 1: Normal Japanese to English Conversation

[1664] When a user opens the app, selects English mode, and says, "Hello, what is your name?", the device captures the voice and movements and sends them to the server. The server translates this into "Hello, what is your name?" and incorporates emotion data. The translation result is converted into audio data, generating natural-sounding English speech, which the device then plays back to support the conversation.

[1665] Example 2: Speech assistance for users who have lost their vocal cords

[1666] When a user opens the app, selects the lip-activated input mode, and imitates the phrase "onegaishimasu" (please) using only their mouth movements, the device captures the movement and sends it to the server. The server translates this into "please" and incorporates emotional data. The translation result is converted into audio data, generating natural-sounding English speech, which the device then plays back to support the conversation.

[1667] In this way, the system of the present invention learns the user's voice characteristics and mouth movements and analyzes the user's emotions, enabling more natural and emotional communication. This not only effectively removes language barriers, but also provides a realistic conversation experience that reflects the user's emotions.

[1668] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1669] Step 1:

[1670] Initial Setup and User Registration

[1671] The user installs the dedicated app on their smartphone and launches the app.

[1672] The terminal displays an initial setup screen and prompts the user to enter information such as their name, target language, and permission for sentiment analysis.

[1673] The user enters information such as name, target language, and permission for sentiment analysis.

[1674] The terminal transmits the input information to the server (input: user information, output: data transmission to the server).

[1675] The server stores the received user information in a database and generates an identifier for each user (input: user information, output: identifier).

[1676] Step 2:

[1677] Voice and movement learning

[1678] The user speaks a short Japanese phrase as instructed by the app.

[1679] The device captures audio with a microphone and records mouth movements with a camera (input: audio, movement; output: captured data).

[1680] The terminal transmits the captured voice data and movement data to the server in real time (input: captured data, output: data transmission to the server).

[1681] The server analyzes the received data using a generative AI model (deep learning framework) and trains a model that converts voice data and movement data into text data (input: voice data, movement data; output: trained model).

[1682] Step 3:

[1683] Emotion analysis using an emotion engine

[1684] The server analyzes the received voice data and action data using an emotion engine to identify the user's emotion (input: voice data, action data, output: emotion data).

[1685] The server stores the analyzed emotion data in a database together with the user's voice characteristics and behavior characteristics (input: emotion data, output: saved data).

[1686] Step 4:

[1687] Real-time speech recognition and translation

[1688] The user selects the target language and speaks in Japanese (e.g., "Hello, what's your name?").

[1689] The device captures audio with a microphone and recognizes mouth movements with a camera (input: audio, movement; output: captured data).

[1690] The terminal transmits the captured voice data and motion data to the server in real time (input: captured data, output: data transmission to the server).

[1691] The server analyzes the received voice data and converts it into Japanese text (input: voice data, output: Japanese text).

[1692] The server uses the Japanese text and emotion data to translate into the target language (e.g., English: "Hello, what is your name?", input: Japanese text, emotion data, output: translated text).

[1693] The server converts the translated text data and emotional data into audio data, generating natural, emotionally-reflective speech (input: translated text, emotional data; output: foreign language audio).

[1694] The terminal plays the generated foreign language speech from a speaker (input: foreign language speech, output: speech playback).

[1695] Step 5:

[1696] Conversation assistance functions

[1697] If a user loses their vocal cords, they can select mouth input mode in the app.

[1698] The user mimics speech using only mouth movements (e.g., "please").

[1699] The device captures mouth movements with a camera and sends the data to the server (input: movement data, output: sending data to the server).

[1700] The server analyzes the action recognition data and converts it into Japanese text (input: action data, output: Japanese text).

[1701] The server uses the Japanese text and emotion data to translate into the target language (e.g., English: "Please", input: Japanese text, emotion data, output: translated text).

[1702] The server converts the translated text data and emotional data into audio data, generating natural, emotionally-reflective speech (input: translated text, emotional data; output: foreign language audio).

[1703] The terminal plays the generated foreign language speech from a speaker (input: foreign language speech, output: speech playback).

[1704] (Application example 2)

[1705] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1706] In modern society, smooth communication between multiple languages ​​remains a major challenge. Particularly in the food delivery field, it is difficult for foreign customers and the hearing impaired to easily place orders. Another problem is that the translated results do not reflect emotions, making it difficult to provide a natural conversation and user experience.

[1707] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1708] In this invention, the server includes means for analyzing voice data and motion data, means for analyzing emotion data and reflecting the emotion data in translation and speech generation, and means for analyzing, translating, and generating speech using a generative AI model. This makes it possible to capture voice and motion data in real time and provide translation results and speech that reflects emotion.

[1709] The "voice recognition means" is a means for capturing the voice spoken by the user and converting it into text data.

[1710] The "motion recognition means" is a means for capturing and analyzing the user's body movements, particularly the mouth movements.

[1711] The "means for analyzing voice data and motion data" refers to a means for processing the captured voice and motion data and extracting necessary information.

[1712] The "means for translating analyzed data into another language" refers to a means for automatically translating into another language based on the recognized speech and motion data.

[1713] The "means for converting translated data into speech" refers to a means for generating speech data from text data translated into another language.

[1714] The "means for analyzing emotion data and reflecting it in translation and speech generation" is a means for analyzing the user's emotion and reflecting it in translation and speech generation.

[1715] "Means that enable a standalone terminal and a server located on the cloud to work in cooperation" refers to means that enable data exchange and cooperative operation between multiple devices.

[1716] The "means for capturing voice and actions in real time and transmitting them to a server" refers to a means for recognizing the user's voice and actions in real time and immediately transmitting them to a server.

[1717] "Means for analyzing, translating, and generating speech data and movement data using a generative AI model" means means for analyzing speech and movement data and performing translation and speech generation processes using a machine learning model.

[1718] The "means for providing the user with the translation result and the speech that reflects the emotion" is a means for providing the user with the translated text and the speech that reflects the emotion.

[1719] This invention is a system that recognizes voice and actions in real time, translates them into other languages, and speaks in a way that reflects the user's emotions by linking the user, device, and server and combining them with an emotion engine. This system uses a dedicated application installed on a smartphone or tablet device and links with a server located in the cloud. Furthermore, it incorporates an emotion engine that analyzes the user's emotions from voice data and movement data and reflects them in the translation results and speech generation.

[1720] Initial Setup and User Registration

[1721] First, the user installs the application on their smartphone and follows the initial setup screen to enter necessary information such as their name, target language, permission for emotion analysis, etc. This information is sent from the device to the server, where it is stored in a database.

[1722] Voice and movement learning phase

[1723] To improve the accuracy of voice and movement data, users are prompted by the application to speak a few short Japanese phrases. The device then captures the voice with a microphone and records the mouth movements with a camera. This data is then sent in real time to a server, where the generative AI model is trained for voice and movement recognition.

[1724] Emotion analysis using an emotion engine

[1725] The server analyzes the user's emotions using the received voice and movement data. The analyzed emotion data is stored along with the user's voice and movement characteristics. This information is used in subsequent communications.

[1726] Real-time speech recognition and translation

[1727] When a user communicates in real time with a speaker of another language, the device's microphone captures the voice and the camera recognizes the mouth movements. The captured data is sent to a server, where the voice data is recognized and converted into Japanese text. The server then translates the Japanese text and emotion data into the target language and generates emotion-reflecting voice data using a generative AI model. The generated foreign language voice is then provided to the user from the device.

[1728] Conversation assistance functions

[1729] The system also provides a mouth-movement speech input mode for users who have lost their vocal cords. The user mimics speech using only their mouth movements, and the device captures and transmits the movements to the server. The server analyzes the movement data, translates it into the target language, and then generates speech that reflects the emotions.

[1730] Examples of concrete examples and prompts

[1731] Example 1:

[1732] Example: A user says, "I'd like a special hamburger and two cokes, please."

[1733] Example prompt: "I'd like a special hamburger and two colas, please." Translate this into English and speak it in a way that reflects the emotion.

[1734] Example 2:

[1735] Example: A user loses their vocal cords and uses mouth movements to mimic the saying "please."

[1736] Example prompt: Translate "please" into English and say it in a way that reflects your feelings.

[1737] The system is realized using cloud services such as Google Cloud Platform's AutoML, Translate API, and Text-to-Speech API. The generative AI model sends voice and movement data captured on the device to the server in real time and performs various data calculations to enable advanced translation and speech generation. This allows users to enjoy a natural and emotional conversation experience.

[1738] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1739] Step 1:

[1740] The user launches the application and enters the necessary information, such as their name, target language, and permission for sentiment analysis, on the initial setup screen. The entered information is sent from the device to the server, which stores it in a database and generates a user identifier. Once this initial setup process for user information is complete, the application is ready to proceed to the next step.

[1741] Step 2:

[1742] In the voice and movement learning phase, the user speaks a few short Japanese phrases according to the application's instructions. The device uses a microphone to capture voice data and a camera to record mouth movement data. The captured voice data and movement data are sent to the server in real time.

[1743] Step 3:

[1744] The server trains a generative AI model based on the received voice and motion data, learning the user's voice and motion characteristics. This process involves analyzing the data (voice recognition, motion recognition) and converting it into text. The resulting model is then used to perform voice and motion recognition optimized for each individual user.

[1745] Step 4:

[1746] The server analyzes the user's emotions from the voice data and movement data. The server stores the analyzed emotion data along with the user's voice characteristics and movement characteristics. For example, if the user says "please," the tone of voice and mouth movements are analyzed and stored as polite emotion data. This data will be used in a later step.

[1747] Step 5:

[1748] This is a real-time speech recognition and translation process. The user speaks Japanese into the device's microphone to translate the sentence into the target language. The device captures the speech and uses a camera to recognize mouth movements. This data is sent to the server in real time.

[1749] Step 6:

[1750] The server analyzes the received voice data and converts it into Japanese text. It then uses a generative AI model to translate the text based on the stored emotion data. For example, the Japanese text "Two special hamburgers and two Cokes, please" is translated into "Two special hamburgers and two Cokes, please."

[1751] Step 7:

[1752] Based on the translated text data and emotion data, the server uses a generative AI model to generate emotion-reflecting voice data, which is then sent to the device in real time.

[1753] Step 8:

[1754] The device receives the foreign language audio from the server and plays it back from the speaker, allowing the user to have a natural speaking experience and enabling multilingual communication.

[1755] Step 9:

[1756] This conversation assistance function for users who have lost their vocal cords imitates speech using only mouth movements. The user selects the mouth input mode and imitates speech using mouth movements. The device captures the movements and sends them to the server.

[1757] Step 10:

[1758] The server analyzes the motion data and converts it into Japanese text. It then uses a generative AI model to translate this text and emotion data into other languages ​​and generate audio data. The generated audio data is then sent to the device and played through the speaker.

[1759] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1760] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1761] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1762] [Fourth embodiment]

[1763] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1764] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1765] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1766] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1767] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1768] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1769] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1770] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1771] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1772] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1773] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1774] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1775] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1776] The present invention is a system in which a user, a terminal, and a server work together to recognize voices and actions in real time, convert them into another language, and then speak the voices. Specific embodiments of the present invention are described below.

[1777] System Overview

[1778] This system basically uses a dedicated application installed on a smartphone or tablet device (hereinafter referred to as "device") and connects to a server located in the cloud. Using this application, users can communicate with speakers of other languages ​​without having to learn the foreign language.

[1779] Initial Setup and User Registration

[1780] 1. User: First, install the app on your smartphone and launch it.

[1781] 2. Device: When the app launches, an initial setup screen appears, prompting the user to enter necessary information (such as name and desired language).

[1782] 3. Terminal: Sends the information entered by the user to the server.

[1783] 4. Server: Stores the received user information in a database and generates an identifier corresponding to each individual user.

[1784] Voice and movement learning phase

[1785] 1. User: Follow the instructions of the app and speak a few short Japanese phrases.

[1786] 2. Device: A microphone captures audio and a camera records mouth movements.

[1787] 3. Terminal: Sends voice data and mouth movement data to the server in real time.

[1788] 4. Server: The AI ​​that generates speech and motion recognition analyzes the user's characteristics and learns a model for converting speech and motion data into text data.

[1789] Real-time speech recognition and translation

[1790] 1. User: After completing the setup, select the target language and speak in Japanese (e.g., "Hello, what's your name?").

[1791] 2. Device: The microphone captures the user's voice and the camera recognizes mouth movements.

[1792] 3. Terminal: Sends the captured voice and movement data to the server.

[1793] 4. Server: Analyzes the received data and converts it into Japanese text.

[1794] 5. Server: Translate the Japanese text into the target language (e.g., English).

[1795] 6. Server: Converts the translated text data into audio data and generates natural-sounding speech.

[1796] 7. Terminal: The generated foreign language speech is provided to the user and played back through the speaker.

[1797] Conversation assistance functions

[1798] 1. User: If you have lost your vocal cords, select mouth input mode.

[1799] 2. User: Imitate speech using mouth movements only (e.g., "please").

[1800] 3. Device: The camera captures mouth movements and sends the data to the server.

[1801] 4. Server: Analyzes the motion recognition data and converts it into Japanese text.

[1802] 5. Server: Translate the Japanese text into the target language.

[1803] 6. Server: Converts the translated text data into audio data and generates natural-sounding speech.

[1804] 7. Terminal: Plays back the generated foreign language speech and provides it to the user.

[1805] Specific examples

[1806] Example 1: Normal Japanese to English Conversation

[1807] 1. User: Open the app, select English mode and say "Hello, what's your name?"

[1808] 2. Device: Captures voice and movements and sends them to the server.

[1809] 3. Server: Translate this into "Hello, what is your name?" and generate it as voice data.

[1810] 4. Terminal: Plays the generated English audio and supports the conversation.

[1811] Example 2: Speech assistance for users who have lost their vocal cords

[1812] 1. User: Open the app and select mouth input mode.

[1813] 2. User: Imitate the word "please" using only mouth movements.

[1814] 3. Device: Captures the action and sends it to the server.

[1815] 4. Server: Translate this into "Please" and generate it as voice data.

[1816] 5. Terminal: Plays the generated English audio and supports the conversation.

[1817] The system of the present invention learns the user's voice characteristics and mouth movements to enable natural conversations that transcend language barriers. It also has a function that notifies the user of an error if a problem occurs with voice capture or translation, prompting the user to take appropriate action. This allows the system to efficiently overcome language barriers, even in situations where communication is difficult, such as sudden overseas business trips.

[1818] The processing flow will be explained below.

[1819] Step 1:

[1820] User: Install the dedicated app on your smartphone and launch it.

[1821] Step 2:

[1822] Terminal: Displays an initial setup screen and prompts the user to enter necessary information such as name and desired language.

[1823] Step 3:

[1824] User: Enter the required information on the initial setup screen and press the "Next" button after completing the entry.

[1825] Step 4:

[1826] Terminal: Sends the information entered by the user to the server.

[1827] Step 5:

[1828] Server: Stores the received user information in a database and generates a unique user identifier.

[1829] Step 6:

[1830] Terminal: Displays prompts to the user to begin the voice and movement learning phase.

[1831] Step 7:

[1832] User: Follow the app's instructions and speak a few short Japanese phrases.

[1833] Step 8:

[1834] Device: A microphone captures your voice and a camera records your mouth movements.

[1835] Step 9:

[1836] Terminal: Sends voice and movement data to the server in real time.

[1837] Step 10:

[1838] Server: Analyzes the received voice and movement data and learns the characteristics of the user's voice and movements.

[1839] Step 11:

[1840] Server: The generated analysis model is linked to the user's account and saved.

[1841] Step 12:

[1842] User: After the initial setup and learning phase is complete, select your target language on the app's main screen.

[1843] Step 13:

[1844] User: Speak in Japanese (e.g. "Hello, what's your name?").

[1845] Step 14:

[1846] Device: A microphone captures the user's voice and a camera recognizes mouth movements.

[1847] Step 15:

[1848] Terminal: Transmits captured audio and motion data to the server in real time.

[1849] Step 16:

[1850] Server: Analyzes the received voice data and converts it into Japanese text (e.g., "Hello, what's your name?").

[1851] Step 17:

[1852] Server: Translate Japanese text into the target language (e.g., English) (e.g., "Hello, what is your name?").

[1853] Step 18:

[1854] Server: Converts the translated text data into audio data and generates natural-sounding speech.

[1855] Step 19:

[1856] Terminal: Receives the generated foreign language audio and plays it back through the speaker.

[1857] Step 20:

[1858] User: Listen to the foreign language audio and continue communication.

[1859] Step 21:

[1860] Terminal: If an error is detected, an error message is displayed to the user.

[1861] Step 22:

[1862] User: Follow the error message and make any necessary corrections or try again.

[1863] Step 23:

[1864] User: If you have lost your vocal cords, select mouth input mode.

[1865] Step 24:

[1866] User: Mimic speech using mouth movements only (e.g., "please").

[1867] Step 25:

[1868] Device: The camera captures mouth movements and sends the data to the server.

[1869] Step 26:

[1870] Server: Analyzes the action recognition data and converts it into Japanese text (e.g., "Onegaishimasu").

[1871] Step 27:

[1872] Server: Translate Japanese text into the target language (e.g., English) (e.g., "Please").

[1873] Step 28:

[1874] Server: Converts the translated text data into audio data and generates natural-sounding speech.

[1875] Step 29:

[1876] Terminal: Receives the generated foreign language audio and plays it back through the speaker.

[1877] Step 30:

[1878] User: Listen to the foreign language audio and continue communication.

[1879] Example 1

[1880] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1881] In today's globalized society, communication between people who speak different languages ​​is becoming increasingly important. However, the existence of language barriers makes smooth communication difficult. Furthermore, people with certain physical limitations (e.g., those who have lost their vocal cords) cannot use standard speech recognition technology, making communication even more difficult. To solve these challenges, there is a need for a system that can recognize voices and movements in real time, convert them into speech in other languages, and then speak them.

[1882] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1883] In this invention, the server includes a voice recognition unit, a motion recognition unit, a unit for analyzing voice data and motion data, a unit for translating the analyzed data into another language, and a unit for converting the translated data into voice. This allows the user to perform initial setup, and a unit for saving and managing the setup information, a unit for capturing voice and motion and transmitting them to the server in real time, and a unit for the server to provide the user with the analysis results, enabling smooth communication between people who speak different languages. Furthermore, even people with certain physical limitations can have natural conversations using mouth movements.

[1884] "Speech recognition means" is a general term for hardware and software for capturing speech as digital data and analyzing its content.

[1885] "Movement recognition means" is a general term for hardware and software for capturing the user's body movements, particularly mouth movements, and analyzing those movements.

[1886] The "means for analyzing voice data and movement data" is a mechanism for analyzing the captured voice data and movement data using an algorithm and converting it into text data.

[1887] "Means for translating analyzed data into another language" is a general term for hardware and software for translating analyzed text data into another specified language.

[1888] "Means for converting translated data into speech" is a general term for hardware and software for converting translated text data into speech data and generating natural-sounding speech.

[1889] The "control means for integrating the above means" is a control system for integrally operating the voice recognition means, the motion recognition means, the analysis means, the translation means, and the voice conversion means.

[1890] "Means for users to perform initial setup and save and manage setup information" is a general term for the interface and database that allows users to perform initial setup of the system and save and manage that information.

[1891] "Means for capturing voice and movement and transmitting it to a server in real time" is a general term for software and network communication means for acquiring voice and movement data, compressing it, and transmitting it to a server.

[1892] "Means by which the server provides the analysis results to the user" refers to a mechanism by which the server returns the analysis and translation results to the user's terminal and displays or plays them aloud.

[1893] The present invention is a system in which a user, a terminal, and a server work together to recognize voices and actions in real time, convert them into another language, and then speak the voices. Specific embodiments of the present invention are described below.

[1894] System Overview

[1895] This system basically uses a dedicated application installed on a smartphone or tablet device (hereinafter referred to as "device") and connects to a server located in the cloud. Using this application, users can communicate with speakers of other languages ​​without having to learn the foreign language.

[1896] Initial Setup and User Registration

[1897] 1. User: Installs the app on a smartphone and launches it. The app can be downloaded from the Google Play Store or the App Store.

[1898] 2. Device: When the app launches, an initial setup screen appears, prompting the user to enter information such as their name, preferred language, and email address. Once the information is entered, the device encrypts and sends it to the server.

[1899] 3. Server: Stores the received user information in a database and generates a unique identifier that is sent back to the device for future operations and data management.

[1900] Voice and movement learning phase

[1901] 1. User: Follow the instructions of the app to speak a few short Japanese phrases, such as "hello" and "thank you."

[1902] 2. Device: A microphone captures audio and a camera records lip movements. Audio and video are synchronized to collect precise data.

[1903] 3. Terminal: Collected voice and movement data is compressed in real time and sent to the server.

[1904] 4. Server: Analyzes and learns voice and movement features using generative AI models, and optimizes the model to convert this data into text.

[1905] Real-time speech recognition and translation

[1906] 1. User: After completing the setup, select the target language and speak in Japanese. For example, the user selects "English mode" and speaks "Hello, what is your name?"

[1907] 2. Device: The microphone captures the voice and the camera recognizes the mouth movements. This data is sent to the server in real time.

[1908] 3. Server: Analyzes the received voice and motion data and converts it into Japanese text. Then, translates the Japanese text into the target language (e.g., English). For example, "Hello, what is your name?" is converted to "Hello, what is your name?"

[1909] 4. Server: Converts the translated text data into audio data and generates natural-sounding speech.

[1910] 5. Terminal: The generated foreign language speech is provided to the user and played back through the speaker.

[1911] Conversation assistance functions

[1912] 1. User: If you have lost your vocal cords, select "Mouth Recognition Mode" from the app settings.

[1913] 2. User: Imitate speech using only mouth movements, for example, recreating the mouth movements for "please."

[1914] 3. Device: The camera captures mouth movements and sends the data to the server.

[1915] 4. Server: Analyzes the motion recognition data and converts it into Japanese text. Next, translates "onegaishimasu" into "Please."

[1916] 5. Server: Converts the translated text data into audio data and generates natural-sounding speech.

[1917] 6. Terminal: Plays back the generated foreign language speech and provides it to the user.

[1918] Examples and prompts

[1919] Example 1: Normal Japanese to English Conversation

[1920] 1. User: Open the app, select English mode and say "Hello, what's your name?"

[1921] 2. Device: Captures voice and movements and sends them to the server.

[1922] 3. Server: Translate this into "Hello, what is your name?" and generate it as voice data.

[1923] 4. Terminal: Plays the generated English audio and supports the conversation.

[1924] Example 2: Speech assistance for users who have lost their vocal cords

[1925] 1. User: Open the app and select mouth input mode.

[1926] 2. User: Imitate the word "please" using only mouth movements.

[1927] 3. Device: Captures the action and sends it to the server.

[1928] 4. Server: Translate this into "Please" and generate it as voice data.

[1929] 5. Terminal: Plays the generated English audio and supports the conversation.

[1930] As an example of a prompt sentence, if you enter "Translate and speak the Japanese greeting into English," the Japanese "Konnichiwa" will be translated and spoken as the English "Hello."

[1931] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1932] Step 1:

[1933] User: Installs the app on their smartphone and launches it.

[1934] Input: Download the app file from Google Play Store or App Store.

[1935] How it works: Press the app's install button, then tap the app icon to launch it after installation is complete.

[1936] Output: The initial setup screen is displayed on the terminal.

[1937] Step 2:

[1938] Device: An initial setup screen appears, prompting the user to enter information such as name, preferred language, and email address.

[1939] Input: The user enters their name, desired language, email address, etc.

[1940] Action: Enter the required information in the input form and press the "Submit" button.

[1941] Output: The input information is encrypted and sent from the device to the server.

[1942] Step 3:

[1943] Server: Stores the received user information in a database and generates and returns an identifier.

[1944] Input: User information data entered.

[1945] What it does: Connects to a database, stores the received data, and generates a unique identifier.

[1946] Output: The generated identifier is sent back to the terminal.

[1947] Step 4:

[1948] User: Follow the app's instructions and speak a few short Japanese phrases.

[1949] Input: A phrase to display on the app screen (e.g. "Hello", "Thank you").

[1950] Action: Reads the displayed text aloud.

[1951] Output: The user's voice and mouth movement data is generated.

[1952] Step 5:

[1953] Device: A microphone captures your voice and a camera records your mouth movements.

[1954] Input: The user's voice and mouth movements.

[1955] Movement: High-definition microphones and high-resolution cameras capture audio waveforms and movement data.

[1956] Output: Captured audio and movement data.

[1957] Step 6:

[1958] Terminal: The captured voice data and movement data are compressed in real time and sent to the server.

[1959] Input: Audio and motion data.

[1960] What it does: Compresses data and sends it to the server with low latency.

[1961] Output: Audio and movement data received by the server.

[1962] Step 7:

[1963] Server: Analyzes and learns voice and movement characteristics using a generative AI model.

[1964] Input: Received voice and movement data.

[1965] How it works: The generative AI model extracts and analyzes features, optimizing the model for conversion to text data.

[1966] Output: Optimized speech-to-motion translation model.

[1967] Step 8:

[1968] User: After completing the setup, select the target language and speak in Japanese.

[1969] Input: The target language selected on the app screen, the words you speak (e.g. "Hello, what's your name?").

[1970] Action: Select the target language and speak in Japanese.

[1971] Output: Spoken Japanese audio.

[1972] Step 9:

[1973] Device: A microphone captures the user's voice and a camera recognizes mouth movements.

[1974] Input: Spoken speech and mouth movements.

[1975] How it works: Captures audio waveforms in real time and records mouth movements with a camera.

[1976] Output: Captured audio and movement data.

[1977] Step 10:

[1978] Terminal: Sends captured audio and motion data to the server.

[1979] Input: Audio and motion data.

[1980] Operation: Real-time transmission of data.

[1981] Output: Audio and motion data received by the server.

[1982] Step 11:

[1983] Server: Analyzes the received voice data and movement data and converts it into Japanese text.

[1984] Input: Audio and motion data.

[1985] Actions: The generative AI model analyzes voice and actions and converts them into Japanese text data.

[1986] Output: Converted Japanese text data.

[1987] Step 12:

[1988] Server: Translates Japanese text into the target language.

[1989] Input: Japanese text.

[1990] Behavior: Translates to the specified target language (e.g., English).

[1991] Output: The translated text data (e.g. "Hello, what is your name?").

[1992] Step 13:

[1993] Server: Converts the translated text data into audio data and generates natural-sounding speech.

[1994] Input: Translated text data.

[1995] How it works: Generates natural-sounding speech using generative AI models.

[1996] Output: The generated audio data.

[1997] Step 14:

[1998] Terminal: Provides the generated foreign language speech to the user and plays it back through a speaker.

[1999] Input: The generated audio data.

[2000] Action: Plays audio through a playback device.

[2001] Output: Foreign language speech played through the speaker (e.g. "Hello, what is your name?").

[2002] Step 15:

[2003] User: If you have lost your vocal cords, select "Mouth Recognition Mode" from the app settings.

[2004] Input: Select "Mouth Recognition Mode" on the app's settings screen.

[2005] Behavior: Toggles selection mode.

[2006] Output: Selected mode.

[2007] Step 16:

[2008] User: Mimics speech using mouth movements only.

[2009] Input: Mouth movements (e.g. "please").

[2010] Action: Mimics mouth movements.

[2011] Output: Mouth movement data.

[2012] Step 17:

[2013] Device: The camera captures mouth movements and sends the data to the server.

[2014] Input: Mouth movement data.

[2015] Movement: A high-precision camera captures mouth movements and transmits the data.

[2016] Output: Lip movement data sent to the server.

[2017] Step 18:

[2018] Server: Analyzes the motion recognition data and converts it into Japanese text.

[2019] Input: Captured mouth movement data.

[2020] Movement: Movement recognition data is analyzed and converted into Japanese text.

[2021] Output: Japanese text (e.g. "Onegaishimasu").

[2022] Step 19:

[2023] Server: Translates Japanese text into the target language.

[2024] Input: Japanese text.

[2025] Behavior: Translates to the specified target language (e.g., English).

[2026] Output: The translated text data (e.g. "Please").

[2027] Step 20:

[2028] Server: Converts the translated text data into audio data and generates natural-sounding speech.

[2029] Input: Translated text data.

[2030] How it works: Generates natural-sounding speech using generative AI models.

[2031] Output: The generated audio data.

[2032] Step 21:

[2033] Terminal: Plays back the generated foreign language speech and provides it to the user.

[2034] Input: The generated audio data.

[2035] Action: Plays audio through a playback device.

[2036] Output: Foreign language voice (e.g. "Please") played through the speaker.

[2037] (Application example 1)

[2038] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2039] Conventional food delivery systems face a language barrier when customers who speak a foreign language place an order, making it difficult to accurately convey order information. This leads to frequent ordering errors and communication problems with customers, resulting in a decline in service quality. Furthermore, only a limited number of systems allow input using mouth movements in addition to voice input, making them inadequate for specific users. To solve these problems, a means was needed to translate and speak order details in real time across language barriers, enabling smooth communication.

[2040] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2041] In this invention, the server includes a voice recognition means, a motion recognition means, a means for analyzing the voice data and motion data, a means for providing the translated voice data as order information to the delivery person or the store, and a control means for integrating the above means. This enables accurate real-time transmission of order information between customers who speak a foreign language and the delivery person or the store, realizing smooth communication that transcends language barriers.

[2042] A "voice recognition means" is a device or technology that receives voice as input and converts it into digital data that can be analyzed.

[2043] "Movement recognition means" refers to a device or technology that captures a user's physical movements, particularly facial and mouth movements, using a sensor such as a camera, and converts them into digital data that can be analyzed.

[2044] "Means for analyzing voice data and movement data" refers to software or hardware that analyzes data based on input voice or movement and converts it into text data or other formats.

[2045] "Means for translating the analyzed data into another language" refers to software or algorithms for translating the analyzed text data into a target language.

[2046] "Means for converting translated data into audio" refers to a device or software that converts translated text data into audio data and outputs it as audio.

[2047] "Means for providing translated voice data as order information to delivery personnel and stores" refers to technology and devices that use translated voice data to transmit accurate order information to the delivery system.

[2048] "Control means" refers to software and hardware that integrates and manages the above means and controls the entire system to ensure smooth operation.

[2049] The present invention provides a system that recognizes speech and gestures in real time and translates them into other languages ​​for speech, which is particularly useful for food delivery services to assist customers who speak other languages ​​in placing orders.

[2050] System configuration

[2051] Hardware Configuration

[2052] The system includes the following hardware:

[2053] Device: Smartphone or tablet

[2054] Microphone: A device used to capture the user's voice.

[2055] Camera: A device for capturing the user's mouth movements.

[2056] Speaker: A device for playing the translated audio.

[2057] Software Configuration

[2058] The system uses the following software:

[2059] Speech Recognition Library: speech_recognition library

[2060] Action recognition library: OpenCV (cv2)

[2061] Translation API: Google Cloud Translation API

[2062] Speech synthesis API: gTTS (Google Text-to-Speech)

[2063] System Operation

[2064] Speech and gesture recognition

[2065] When a user speaks, the device's microphone captures the voice and the camera records the mouth movements. This information is converted into text data using a speech recognition library (speech_recognition) and a motion recognition library (OpenCV). The data captured by the speech recognition and motion recognition means is processed in real time.

[2066] Data analysis and translation

[2067] The server analyzes the voice and motion data in the cloud, converts the user's speech into text using the analysis method, and then translates the analyzed text into the target language using the Google Cloud Translation API.

[2068] Speech synthesis of translation data

[2069] The translated text data is converted into voice data using the gTTS (Google Text-to-Speech) API. The voice data generated using a voice synthesis tool is played back on the device's speaker. This allows users to smoothly place orders through food delivery services without having to learn a foreign language.

[2070] Specific examples

[2071] Order Scenarios

[2072] 1. User: Download the smartphone app and select the language you want to use (e.g., English).

[2073] 2. User: "Hello, I'd like one Margherita pizza please" in Japanese.

[2074] 3. Device: Captures voice and lip movements and sends the data to the server.

[2075] 4. Server: Translates the speech data to "Hello, one Margherita pizza please" and converts it to speech data.

[2076] 5. Terminal: Plays back the translated voice data and provides it to delivery personnel and stores.

[2077] Prompt Sentence Examples

[2078] "How can I develop a food delivery app that translates Japanese speech into English in real time and provides it in a way that delivery staff and restaurants can understand?"

[2079] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2080] Step 1:

[2081] Input: The user launches the app on their smartphone and selects the language they want to use (e.g., English).

[2082] Specific operation: The user selects the desired language on the app's settings screen and enters their information (name, address, desired language, etc.).

[2083] Output: Configuration information is sent from the device to the server.

[2084] Step 2:

[2085] Input: User speaks in Japanese (e.g., "Hello, I'd like one Margherita pizza please.").

[2086] What it does: The device's microphone captures the user's voice and the camera records their mouth movements.

[2087] Output: Audio and motion data are generated.

[2088] Step 3:

[2089] Input: Audio and motion data captured by the device.

[2090] Specific operation: Using the speech recognition library (speech_recognition), converts voice data into text data. It also analyzes mouth movement data recorded by a camera.

[2091] Output: Japanese text data is generated and sent to the server.

[2092] Step 4:

[2093] Input: Japanese text data received by the server.

[2094] What it does: It uses the Google Cloud Translation API in the cloud to translate text data into the desired language (e.g., English).

[2095] Output: The translated text data is generated.

[2096] Step 5:

[2097] Input: Translated text data.

[2098] What it does: Uses the gTTS (Google Text-to-Speech) API to convert the translated text data into audio data.

[2099] Output: The translated audio data is generated.

[2100] Step 6:

[2101] Input: Translated audio data.

[2102] Specific operation: The device's speaker plays the generated voice data, which is then provided to the delivery person or the store.

[2103] Output: The translated content is played back as audio, and accurate order information is communicated to the delivery person or store.

[2104] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2105] The present invention is a system that recognizes voice and actions in real time, translates them into other languages, and speaks in a way that reflects the user's emotions by linking the user, terminal, and server and combining them with an emotion engine. Specific embodiments of the present invention are described below.

[2106] System configuration

[2107] This system uses a dedicated application installed on a smartphone or tablet device (hereafter referred to as "device") and connects to a server located in the cloud. It also incorporates an emotion engine that analyzes the user's emotions from voice and movement data and reflects this in the translation results and speech generation. Using this app, users can communicate with speakers of other languages ​​without having to learn the language.

[2108] Initial Setup and User Registration

[2109] 1. User: First, install the app on your smartphone and launch it.

[2110] 2. Device: Displays the initial setup screen and prompts the user to enter necessary information such as their name, target language, and permission for sentiment analysis.

[2111] 3. Terminal: Sends the information entered by the user to the server.

[2112] 4. Server: Stores the received user information in a database and generates a unique user identifier.

[2113] Voice and movement learning phase

[2114] 1. User: Follow the instructions of the app and speak a few short Japanese phrases.

[2115] 2. Device: A microphone captures audio and a camera records mouth movements.

[2116] 3. Terminal: Sends voice data and mouth movement data to the server in real time.

[2117] 4. Server: The AI ​​that generates speech and motion recognition analyzes the user's characteristics and learns a model for converting speech and motion data into text data.

[2118] Emotion analysis using an emotion engine

[2119] 1. Server: Using the received voice data and movement data, the emotion engine analyzes the user's emotions.

[2120] 2. Server: Stores the analyzed emotion data along with the user's voice and movement characteristics.

[2121] Real-time speech recognition and translation

[2122] 1. User: After completing the setup, select the target language and speak in Japanese (e.g., "Hello, what's your name?").

[2123] 2. Device: The microphone captures audio and the camera recognizes mouth movements.

[2124] 3. Terminal: Transmits captured voice and movement data to the server in real time.

[2125] 4. Server: Analyzes the received voice data and converts it into Japanese text (e.g., "Hello, what's your name?").

[2126] 5. Server: Uses Japanese text and emotion data to translate into the target language (e.g., English) (e.g., "Hello, what is your name?").

[2127] 6. Server: Based on the translated text data and emotion data, it converts it into voice data and generates natural, emotionally-informed speech.

[2128] 7. Terminal: The generated foreign language speech is provided to the user and played back through the speaker.

[2129] Conversation assistance functions

[2130] 1. User: If you have lost your vocal cords, select mouth input mode.

[2131] 2. User: Imitate speech using mouth movements only (e.g., "please").

[2132] 3. Device: The camera captures mouth movements and sends the data to the server.

[2133] 4. Server: Analyzes the action recognition data and converts it into Japanese text (e.g., "Onegaishimasu").

[2134] 5. Server: Uses the Japanese text and emotion data to translate into the target language (e.g., English) (e.g., "Please").

[2135] 6. Server: Based on the translated text data and emotion data, it converts it into voice data and generates natural, emotionally-informed speech.

[2136] 7. Terminal: Receives the generated foreign language audio and plays it back through the speaker.

[2137] Specific examples

[2138] Example 1: Normal Japanese to English Conversation

[2139] 1. User: Open the app, select English mode and say "Hello, what's your name?"

[2140] 2. Device: Captures voice and movements and sends them to the server.

[2141] 3. Server: Translate this to "Hello, what is your name?" and incorporate sentiment data.

[2142] 4. Server: Converts the translation results into audio data and generates natural speech.

[2143] 5. Terminal: Plays the generated English audio and supports the conversation.

[2144] Example 2: Speech assistance for users who have lost their vocal cords

[2145] 1. User: Open the app and select mouth input mode.

[2146] 2. User: Imitate the word "please" using only mouth movements.

[2147] 3. Device: Captures the action and sends it to the server.

[2148] 4. Server: Translate this to "Please" and incorporate emotional data.

[2149] 5. Server: Converts the translation results into audio data and generates natural speech.

[2150] 6. Terminal: Plays the generated English audio and supports conversation.

[2151] In this way, the system of the present invention learns the user's voice characteristics and mouth movements and analyzes the user's emotions, enabling more natural and emotional communication. This not only effectively removes language barriers, but also provides a realistic conversation experience that reflects the user's emotions.

[2152] The processing flow will be explained below.

[2153] Step 1:

[2154] User: Install the dedicated app on your smartphone and launch it.

[2155] Step 2:

[2156] Device: Displays an initial setup screen and prompts the user to enter required information such as name, target language, and permission for sentiment analysis.

[2157] Step 3:

[2158] User: Enter the required information on the initial setup screen and press the "Next" button after completing the entry.

[2159] Step 4:

[2160] Terminal: Sends the information entered by the user to the server.

[2161] Step 5:

[2162] Server: Stores the received user information in a database and generates a unique user identifier.

[2163] Step 6:

[2164] Terminal: Displays prompts to the user to begin the voice and movement learning phase.

[2165] Step 7:

[2166] User: Follow the app's instructions and speak a few short Japanese phrases.

[2167] Step 8:

[2168] Device: A microphone captures your voice and a camera records your mouth movements.

[2169] Step 9:

[2170] Terminal: Sends voice and movement data to the server in real time.

[2171] Step 10:

[2172] Server: Analyzes the received voice and movement data and learns the characteristics of the user's voice and movements.

[2173] Step 11:

[2174] Server: The generated analysis model is linked to the user's account and saved.

[2175] Step 12:

[2176] Server: Analyzes the user's emotions using an emotion engine based on voice data and movement data.

[2177] Step 13:

[2178] Server: Stores the analyzed emotion data along with the user's voice and movement characteristics.

[2179] Step 14:

[2180] User: After completing the initial setup and learning phase, select the desired language on the app's main screen.

[2181] Step 15:

[2182] User: Speak in Japanese (e.g. "Hello, what's your name?").

[2183] Step 16:

[2184] Device: A microphone captures audio and a camera recognizes mouth movements.

[2185] Step 17:

[2186] Terminal: Transmits captured audio and motion data to the server in real time.

[2187] Step 18:

[2188] Server: Analyzes the received voice data and converts it into Japanese text (e.g., "Hello, what's your name?").

[2189] Step 19:

[2190] Server: Uses Japanese text and emotion data to translate into the target language (e.g., English) (e.g., "Hello, what is your name?").

[2191] Step 20:

[2192] Server: Based on the translated text data and emotion data, converts it into voice data and generates natural speech that reflects emotion.

[2193] Step 21:

[2194] Terminal: Receives the generated foreign language audio and plays it back through the speaker.

[2195] Step 22:

[2196] User: Listen to the foreign language audio and continue communicating with the other person.

[2197] Step 23:

[2198] Terminal: If an error is detected, an error message is displayed to the user.

[2199] Step 24:

[2200] User: Follow the error message and make any necessary corrections or try again.

[2201] Step 25:

[2202] User: If you have lost your vocal cords, select mouth input mode.

[2203] Step 26:

[2204] User: Mimic speech using mouth movements only (e.g., "please").

[2205] Step 27:

[2206] Device: The camera captures mouth movements and sends the data to the server.

[2207] Step 28:

[2208] Server: Analyzes the action recognition data and converts it into Japanese text (e.g., "Onegaishimasu").

[2209] Step 29:

[2210] Server: Uses Japanese text and emotion data to translate into the target language (e.g., English) (e.g., "Please").

[2211] Step 30:

[2212] Server: Based on the translated text data and emotion data, converts it into voice data and generates natural speech that reflects emotion.

[2213] Step 31:

[2214] Terminal: Receives the generated foreign language audio and plays it back through the speaker.

[2215] Step 32:

[2216] User: Listen to the foreign language audio and continue communicating with the other person.

[2217] Example 2

[2218] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2219] Conventional speech recognition and translation systems are unable to recognize speech and actions in real time and reflect emotions when translating into other languages, making it difficult to achieve natural, emotionally rich communication. Furthermore, they are unable to individually learn the speech and action characteristics of specific users, making it difficult to provide recognition accuracy tailored to each individual user. Furthermore, they lack the functionality to accommodate special situations, such as users who have lost their vocal cords.

[2220] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a voice recognition means, a motion recognition means, a means for analyzing voice data and motion data, a means for translating the analyzed data into another language, a means for converting the translated data into speech, a means for analyzing emotions and reflecting the emotions in translation and speech generation, and a control means for integrating the above means. This enables natural communication that reflects emotions in real time. Furthermore, the system can learn the voice and motion characteristics of specific users and provide recognition accuracy tailored to each individual user. Furthermore, by providing an input mode based on mouth movements, appropriate conversation assistance can be provided to users who have lost their vocal cords.

[2221] A "voice recognition means" is a device or mechanism that receives voice input and analyzes the voice signal to convert it into text data.

[2222] The "motion recognition means" is a device or mechanism that receives video input, analyzes the motions and gestures, and converts them into meaningful data.

[2223] The "means for analyzing voice data and movement data" refers to a device or mechanism that processes the acquired voice data and movement data, extracts its features, and performs further higher-level analysis.

[2224] The "means for translating parsed data into another language" is a device or mechanism for translating parsed text data into a target language.

[2225] The "means for converting translated data into speech" refers to a device or mechanism that converts the translated text data into speech signals and outputs them as speech.

[2226] The "means for analyzing emotions and reflecting them in translation and speech generation" is a device or mechanism that analyzes the emotional state of the user and reflects that emotional information in translation and speech generation.

[2227] The "control means for integrating the means" is a device or mechanism for efficiently coordinating and enabling a plurality of means to function.

[2228] A "means for learning the voice and movement characteristics of a specific user" is a device or mechanism that learns the voice and movement characteristics of a specific user using machine learning or the like, and generates a recognition model optimized for that user.

[2229] The present invention is a system in which a user, a terminal, and a server work together and combine an emotion engine to recognize voice and actions in real time, translate them into other languages, and then speak in a way that reflects the user's emotions.Specific embodiments of the present invention are described below.

[2230] System configuration

[2231] This system uses a dedicated application installed on a smartphone or tablet device (hereafter referred to as "device") and connects to a server located in the cloud. It also incorporates an emotion engine that analyzes the user's emotions from voice and movement data and reflects this in the translation results and speech generation. Using this app, users can communicate with speakers of other languages ​​without having to learn the language.

[2232] Initial Setup and User Registration

[2233] First, a user installs the app on their smartphone and launches it. The device displays an initial setup screen and prompts the user to enter necessary information such as their name, target language, and permission for sentiment analysis. The information entered by the user is sent by the device to a server, which then stores the received user information in a database and generates a unique user identifier.

[2234] Voice and movement learning phase

[2235] Next, the user follows the app's instructions to speak a few short Japanese phrases. The device captures the voice with a microphone and records the mouth movements with a camera. The captured voice and movement data are sent to the server in real time. The server uses a generative AI model to analyze the user's features for voice and movement recognition, and to train a model for converting the voice and movement data into text data.

[2236] Emotion analysis using an emotion engine

[2237] The server uses the received voice data and movement data to analyze the user's emotions with its emotion engine. The analyzed emotion data is then saved along with the user's voice and movement characteristics, allowing the user's emotional state to be properly reflected.

[2238] Real-time speech recognition and translation

[2239] After the user completes the setup, they select the target language and speak in Japanese (e.g., "Hello, what is your name?"). The device captures the voice with a microphone and recognizes mouth movements with a camera. The captured voice and movement data are sent to a server in real time, and the server analyzes the received voice data and converts it into Japanese text. The Japanese text and emotion data are then used to translate it into the target language (e.g., English) (e.g., "Hello, what is your name?"). The translated text data and emotion data are then converted into voice data, generating natural, emotionally-reflective speech. The generated foreign language voice is provided to the user by the device and played through the speaker.

[2240] Conversation assistance functions

[2241] If a user has lost their vocal cords, they can select the mouth movement input mode. The user mimics speech using only mouth movements (e.g., "Onegaishimasu"), and the device captures the mouth movements with a camera and sends the data to the server. The server analyzes the gesture recognition data and converts it into Japanese text (e.g., "Onegaishimasu"). Based on this, a translated target language (e.g., English) is generated (e.g., "Please") to create natural-sounding speech. The generated audio is played back from the device to support the conversation.

[2242] Specific examples

[2243] Example 1: Normal Japanese to English Conversation

[2244] When a user opens the app, selects English mode, and says, "Hello, what is your name?", the device captures the voice and movements and sends them to the server. The server translates this into "Hello, what is your name?" and incorporates emotion data. The translation result is converted into audio data, generating natural-sounding English speech, which the device then plays back to support the conversation.

[2245] Example 2: Speech assistance for users who have lost their vocal cords

[2246] When a user opens the app, selects the lip-activated input mode, and imitates the phrase "onegaishimasu" (please) using only their mouth movements, the device captures the movement and sends it to the server. The server translates this into "please" and incorporates emotional data. The translation result is converted into audio data, generating natural-sounding English speech, which the device then plays back to support the conversation.

[2247] In this way, the system of the present invention learns the user's voice characteristics and mouth movements and analyzes the user's emotions, enabling more natural and emotional communication. This not only effectively removes language barriers, but also provides a realistic conversation experience that reflects the user's emotions.

[2248] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2249] Step 1:

[2250] Initial Setup and User Registration

[2251] The user installs the dedicated app on their smartphone and launches the app.

[2252] The terminal displays an initial setup screen and prompts the user to enter information such as their name, target language, and permission for sentiment analysis.

[2253] The user enters information such as name, target language, and permission for sentiment analysis.

[2254] The terminal transmits the input information to the server (input: user information, output: data transmission to the server).

[2255] The server stores the received user information in a database and generates an identifier for each user (input: user information, output: identifier).

[2256] Step 2:

[2257] Voice and movement learning

[2258] The user speaks a short Japanese phrase as instructed by the app.

[2259] The device captures audio with a microphone and records mouth movements with a camera (input: audio, movement; output: captured data).

[2260] The terminal transmits the captured voice data and movement data to the server in real time (input: captured data, output: data transmission to the server).

[2261] The server analyzes the received data using a generative AI model (deep learning framework) and trains a model that converts voice data and movement data into text data (input: voice data, movement data; output: trained model).

[2262] Step 3:

[2263] Emotion analysis using an emotion engine

[2264] The server analyzes the received voice data and action data using an emotion engine to identify the user's emotion (input: voice data, action data, output: emotion data).

[2265] The server stores the analyzed emotion data in a database together with the user's voice characteristics and behavior characteristics (input: emotion data, output: saved data).

[2266] Step 4:

[2267] Real-time speech recognition and translation

[2268] The user selects the target language and speaks in Japanese (e.g., "Hello, what's your name?").

[2269] The device captures audio with a microphone and recognizes mouth movements with a camera (input: audio, movement; output: captured data).

[2270] The terminal transmits the captured voice data and motion data to the server in real time (input: captured data, output: data transmission to the server).

[2271] The server analyzes the received voice data and converts it into Japanese text (input: voice data, output: Japanese text).

[2272] The server uses the Japanese text and emotion data to translate into the target language (e.g., English: "Hello, what is your name?", input: Japanese text, emotion data, output: translated text).

[2273] The server converts the translated text data and emotional data into audio data, generating natural, emotionally-reflective speech (input: translated text, emotional data; output: foreign language audio).

[2274] The terminal plays the generated foreign language speech from a speaker (input: foreign language speech, output: speech playback).

[2275] Step 5:

[2276] Conversation assistance functions

[2277] If a user loses their vocal cords, they can select mouth input mode in the app.

[2278] The user mimics speech using only mouth movements (e.g., "please").

[2279] The device captures mouth movements with a camera and sends the data to the server (input: movement data, output: sending data to the server).

[2280] The server analyzes the action recognition data and converts it into Japanese text (input: action data, output: Japanese text).

[2281] The server uses the Japanese text and emotion data to translate into the target language (e.g., English: "Please", input: Japanese text, emotion data, output: translated text).

[2282] The server converts the translated text data and emotional data into audio data, generating natural, emotionally-reflective speech (input: translated text, emotional data; output: foreign language audio).

[2283] The terminal plays the generated foreign language speech from a speaker (input: foreign language speech, output: speech playback).

[2284] (Application example 2)

[2285] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2286] In modern society, smooth communication between multiple languages ​​remains a major challenge. Particularly in the food delivery field, it is difficult for foreign customers and the hearing impaired to easily place orders. Another problem is that the translated results do not reflect emotions, making it difficult to provide a natural conversation and user experience.

[2287] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2288] In this invention, the server includes means for analyzing voice data and motion data, means for analyzing emotion data and reflecting the emotion data in translation and speech generation, and means for analyzing, translating, and generating speech using a generative AI model. This makes it possible to capture voice and motion data in real time and provide translation results and speech that reflects emotion.

[2289] The "voice recognition means" is a means for capturing the voice spoken by the user and converting it into text data.

[2290] The "motion recognition means" is a means for capturing and analyzing the user's body movements, particularly the mouth movements.

[2291] The "means for analyzing voice data and motion data" refers to a means for processing the captured voice and motion data and extracting necessary information.

[2292] The "means for translating analyzed data into another language" refers to a means for automatically translating into another language based on the recognized speech and motion data.

[2293] The "means for converting translated data into speech" refers to a means for generating speech data from text data translated into another language.

[2294] The "means for analyzing emotion data and reflecting it in translation and speech generation" is a means for analyzing the user's emotion and reflecting it in translation and speech generation.

[2295] "Means that enable a standalone terminal and a server located on the cloud to work in cooperation" refers to means that enable data exchange and cooperative operation between multiple devices.

[2296] The "means for capturing voice and actions in real time and transmitting them to a server" refers to a means for recognizing the user's voice and actions in real time and immediately transmitting them to a server.

[2297] "Means for analyzing, translating, and generating speech data and movement data using a generative AI model" means means for analyzing speech and movement data and performing translation and speech generation processes using a machine learning model.

[2298] The "means for providing the user with the translation result and the speech that reflects the emotion" is a means for providing the user with the translated text and the speech that reflects the emotion.

[2299] This invention is a system that recognizes voice and actions in real time, translates them into other languages, and speaks in a way that reflects the user's emotions by linking the user, device, and server and combining them with an emotion engine. This system uses a dedicated application installed on a smartphone or tablet device and links with a server located in the cloud. Furthermore, it incorporates an emotion engine that analyzes the user's emotions from voice data and movement data and reflects them in the translation results and speech generation.

[2300] Initial Setup and User Registration

[2301] First, the user installs the application on their smartphone and follows the initial setup screen to enter necessary information such as their name, target language, permission for emotion analysis, etc. This information is sent from the device to the server, where it is stored in a database.

[2302] Voice and movement learning phase

[2303] To improve the accuracy of voice and movement data, users are prompted by the application to speak a few short Japanese phrases. The device then captures the voice with a microphone and records the mouth movements with a camera. This data is then sent in real time to a server, where the generative AI model is trained for voice and movement recognition.

[2304] Emotion analysis using an emotion engine

[2305] The server analyzes the user's emotions using the received voice and movement data. The analyzed emotion data is stored along with the user's voice and movement characteristics. This information is used in subsequent communications.

[2306] Real-time speech recognition and translation

[2307] When a user communicates in real time with a speaker of another language, the device's microphone captures the voice and the camera recognizes the mouth movements. The captured data is sent to a server, where the voice data is recognized and converted into Japanese text. The server then translates the Japanese text and emotion data into the target language and generates emotion-reflecting voice data using a generative AI model. The generated foreign language voice is then provided to the user from the device.

[2308] Conversation assistance functions

[2309] The system also provides a mouth-movement speech input mode for users who have lost their vocal cords. The user mimics speech using only their mouth movements, and the device captures and transmits the movements to the server. The server analyzes the movement data, translates it into the target language, and then generates speech that reflects the emotions.

[2310] Examples of concrete examples and prompts

[2311] Example 1:

[2312] Example: A user says, "I'd like a special hamburger and two cokes, please."

[2313] Example prompt: "I'd like a special hamburger and two colas, please." Translate this into English and speak it in a way that reflects the emotion.

[2314] Example 2:

[2315] Example: A user loses their vocal cords and uses mouth movements to mimic the saying "please."

[2316] Example prompt: Translate "please" into English and say it in a way that reflects your feelings.

[2317] The system is realized using cloud services such as Google Cloud Platform's AutoML, Translate API, and Text-to-Speech API. The generative AI model sends voice and movement data captured on the device to the server in real time and performs various data calculations to enable advanced translation and speech generation. This allows users to enjoy a natural and emotional conversation experience.

[2318] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2319] Step 1:

[2320] The user launches the application and enters the necessary information, such as their name, target language, and permission for sentiment analysis, on the initial setup screen. The entered information is sent from the device to the server, which stores it in a database and generates a user identifier. Once this initial setup process for user information is complete, the application is ready to proceed to the next step.

[2321] Step 2:

[2322] In the voice and movement learning phase, the user speaks a few short Japanese phrases according to the application's instructions. The device uses a microphone to capture voice data and a camera to record mouth movement data. The captured voice data and movement data are sent to the server in real time.

[2323] Step 3:

[2324] The server trains a generative AI model based on the received voice and motion data, learning the user's voice and motion characteristics. This process involves analyzing the data (voice recognition, motion recognition) and converting it into text. The resulting model is then used to perform voice and motion recognition optimized for each individual user.

[2325] Step 4:

[2326] The server analyzes the user's emotions from the voice data and movement data. The server stores the analyzed emotion data along with the user's voice characteristics and movement characteristics. For example, if the user says "please," the tone of voice and mouth movements are analyzed and stored as polite emotion data. This data will be used in a later step.

[2327] Step 5:

[2328] This is a real-time speech recognition and translation process. The user speaks Japanese into the device's microphone to translate the sentence into the target language. The device captures the speech and uses a camera to recognize mouth movements. This data is sent to the server in real time.

[2329] Step 6:

[2330] The server analyzes the received voice data and converts it into Japanese text. It then uses a generative AI model to translate the text based on the stored emotion data. For example, the Japanese text "Two special hamburgers and two Cokes, please" is translated into "Two special hamburgers and two Cokes, please."

[2331] Step 7:

[2332] Based on the translated text data and emotion data, the server uses a generative AI model to generate emotion-reflecting voice data, which is then sent to the device in real time.

[2333] Step 8:

[2334] The device receives the foreign language audio from the server and plays it back from the speaker, allowing the user to have a natural speaking experience and enabling multilingual communication.

[2335] Step 9:

[2336] This conversation assistance function for users who have lost their vocal cords imitates speech using only mouth movements. The user selects the mouth input mode and imitates speech using mouth movements. The device captures the movements and sends them to the server.

[2337] Step 10:

[2338] The server analyzes the motion data and converts it into Japanese text. It then uses a generative AI model to translate this text and emotion data into other languages ​​and generate audio data. The generated audio data is then sent to the device and played through the speaker.

[2339] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2340] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2341] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2342] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2343] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2344] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2345] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2346] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2347] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2348] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2349] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2350] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2351] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2352] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the s...

Claims

1. a voice recognition means; A motion recognition means; means for analyzing voice data and motion data; a means for translating the parsed data into other languages; means for converting the translated data into speech; control means for integrating said means; A system including:

2. 2. The system of claim 1, further comprising means for learning the vocal cord behavior of a particular user in recognizing voice data.

3. 2. The system of claim 1, further comprising means for learning a particular user's mouth movements in recognizing the motion data.

4. 10. The system of claim 1, further comprising means for generating and playing translated speech data in real time.

5. 10. The system of claim 1, further comprising means for providing error notification if a problem occurs in capturing audio or motion data.

6. 2. The system of claim 1, further comprising means adapted to different dialects and accents within the same language as well as foreign languages.

Images