System

Abstract

A system is provided.SOLUTION: A system comprising: means for setting a first language and a second language for selection by a user; means for capturing audio data from a first terminal; means for converting the captured audio data into text data; means for translating the converted text data into the selected second language; and means for displaying the translated text data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Language barriers are a major problem when people who speak different languages ​​communicate. This problem often hinders smooth communication in business and everyday life. Conventional language translation tools and applications lack real-timeness and accuracy, making them inadequate for situations requiring immediate communication. Furthermore, they are often cumbersome to use, leaving room for improvement in terms of usability. [Means for solving the problem]

[0005] In order to solve the above-mentioned problems, the present invention provides the following means. First, it includes means for setting a first language and a second language for the user to select. Next, it provides means for capturing voice data from a first terminal and converting the captured voice data into text data. It also includes means for translating the converted text data into the selected second language. This translated text data is displayed in real time by display means. This allows users to communicate smoothly across language barriers. It is also possible to capture voice data from a second terminal and translate and display the data into the user's first language through a similar process. By integrating these means, real-time, two-way language translation becomes possible, significantly improving usability.

[0006] "User" refers to a person who uses the system to make a video call.

[0007] "Primary language" refers to the language that a user chooses to express their speech.

[0008] The "second language" refers to the language selected by the user to translate and display the content of the other person's speech.

[0009] "Terminal" refers to the device (e.g., smartphone, tablet, computer, etc.) that a user uses to make a video call.

[0010] "Voice data" refers to data that is a digital recording of the voice spoken by the user or the other party.

[0011] "Text data" refers to the character information of voice data converted by a voice recognition engine.

[0012] A "voice recognition engine" refers to software or hardware technology for converting received voice data into text data.

[0013] "Translation AI" refers to artificial intelligence technology that translates text data from one language to another.

[0014] "Display means" refers to the technology or device for displaying translated text data on the video call screen.

[0015] "Capture" refers to taking in audio data through an input device such as a microphone.

[0016] "Real-time" means that the entire process from user speech to translation and display is carried out instantly. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The system based on this invention is a video calling service that allows users to smoothly communicate with people who speak different languages. Specifically, it has the function of translating languages ​​in real time using voice recognition technology and translation AI and displaying the translation content. The specific operation form of the system and its components are described below.

[0039] System Components

[0040] 1. User terminal: The device that a user uses to make a video call (e.g., smartphone, tablet, computer, etc.).

[0041] 2. Speech recognition engine: Software or hardware that converts a user's voice data into text data.

[0042] 3. Translation AI: Artificial intelligence technology for translating text data from one language to another.

[0043] 4. Server: The central device that controls the capture of voice data, text conversion, translation, and sending and receiving of the results.

[0044] 5. Display means: Technology or equipment for displaying translated text data on the video call screen.

[0045] System Operation Overview

[0046] 1. Language selection: First, the user selects their own language (first language) and the other person's language (second language) on the device's settings screen.

[0047] 2. Starting a video call: The user presses a button to start a video call, which sends a connection request to the server, which then sends this request to the other device and establishes the call.

[0048] 3. Voice capture: When the user starts speaking, the device captures voice data through the microphone and transmits it to the server in real time.

[0049] 4. Speech recognition and text conversion: The server passes the received voice data to a speech recognition engine and converts it into text data.

[0050] 5. Text translation: The converted text data is passed from the server to a translation AI, where it is translated into a second language.

[0051] 6. Displaying translated text: The translated text data is sent to the user's and the other party's devices via the server and displayed on the video call screen.

[0052] Specific examples

[0053] For example, consider the case where user A selects Japanese and the other user, user B, selects English. When user A says, "Hello, how are you?", the speech is captured on the device and sent to the server. The server uses a speech recognition engine to convert this speech into text, "Hello, how are you?" This text is sent to a translation AI, which translates it into English as "Hello, how are you?" The translated text is sent to the devices of user A and user B via the server and displayed on the video call screen.

[0054] Similarly, when User B says "I am fine, thank you," the voice data is sent to the server and converted into text by a speech recognition engine. This text is then translated into Japanese by a translation AI as "I am fine, thank you." The translated text is then sent via the server again to the devices of User A and User B and displayed.

[0055] By repeating this process, users and their contacts can communicate in real time, overcoming language barriers. This system is designed to improve usability and enable smooth two-way communication.

[0056] The processing flow will be explained below.

[0057] Program processing flow details

[0058] Step 1:

[0059] The user launches the application on their device and transitions to the main screen, at which point the account information is loaded.

[0060] Step 2:

[0061] The user accesses the language selection screen and sets their own language (first language) and the other party's language (second language). The terminal then sends the selected language information to the server.

[0062] Step 3:

[0063] To start a video call, the user presses a video call start button, and the terminal sends the request to the server.

[0064] Step 4:

[0065] The server receives the user's request and sends a connection request to the other party's terminal. If the other party accepts the request, the call is established.

[0066] Step 5:

[0067] When a user starts speaking, the device captures the voice data through the microphone, and the captured voice data is sent to the server in real time.

[0068] Step 6:

[0069] The server inputs the received voice data into a voice recognition engine and converts it into text data.

[0070] Step 7:

[0071] The server passes the converted text data to a translation AI, which translates it into a second language.

[0072] Step 8:

[0073] The server transmits the translated text data to the user terminal and the other terminal.

[0074] Step 9:

[0075] The terminal displays the received translated text on the video call screen.

[0076] Step 10:

[0077] When the other person begins to speak, the other person's device also captures the voice data through the microphone and sends it to the server.

[0078] Step 11:

[0079] The server inputs the other party's voice data into a voice recognition engine and converts it into text data.

[0080] Step 12:

[0081] The server passes the converted text data to a translation AI, which translates it into the first language.

[0082] Step 13:

[0083] The server transmits the translated text data to the user terminal and the other terminal.

[0084] Step 14:

[0085] The user terminal displays the received translated text on the video call screen.

[0086] This process flow enables users to communicate with others in real time, overcoming language barriers. Both users and others can receive information in a language they can instantly understand, ensuring smooth dialogue.

[0087] Example 1

[0088] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0089] It is difficult for users who speak different languages ​​to communicate smoothly. Real-time interpretation is particularly required during video calls, but achieving this is technically complex and expensive. Furthermore, existing translation systems have problems such as low translation accuracy and a lack of real-time capabilities, resulting in a poor user experience. The present invention aims to solve these problems by providing a system that enables users who speak different languages ​​to communicate smoothly in real time.

[0090] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0091] In this invention, the server includes means for setting a first language and a second language for selection by the user, means for capturing voice data from the first terminal, means for converting the captured voice data into text data, means for translating the converted text data into the selected second language, means for displaying the translated text data, means for sending a connection request and establishing a video call, means for transmitting and receiving voice data via the server, means for converting the voice data into text data using a voice recognition engine, means for translating the text data into a specified language in real time using a translation AI, and means for displaying the translated text data on the video call screen, thereby enabling users to smoothly communicate in real time with parties who speak different languages.

[0092] "User" means a person or end user who makes a video call using the System.

[0093] "Primary language" refers to the language that a user selects within the system to use.

[0094] "Second language" refers to the language used by the person selected by the user within the system.

[0095] A "terminal" is a device for making video calls, and includes a smartphone, tablet, computer, etc.

[0096] "Voice data" refers to a digital recording of a user's speech.

[0097] "Text data" is voice data converted into character information.

[0098] A "voice recognition engine" is software or hardware that converts voice data into text data.

[0099] "Translation AI" is an artificial intelligence technology that translates text data into other languages.

[0100] The "server" is a central device that manages the processing of the entire system, and transmits and receives voice data, converts text, translates, and transmits the results.

[0101] The "video call screen" is a part of the user interface for displaying translated text data during a video call.

[0102] A "connection request" is a request signal sent from a server to start a video call.

[0103] "Capture" refers to the act of acquiring audio data using an input device such as a terminal's microphone.

[0104] The system based on this invention is a video calling service that enables users who speak different languages ​​to communicate smoothly. This system uses speech recognition technology and translation AI to translate languages ​​in real time and display the translation results. The detailed configuration and operation of the system are described below.

[0105] System Components

[0106] 1. User Device

[0107] This is a device that allows users to make video calls, such as a smartphone, tablet, or computer. These devices must have a microphone for voice input and a screen to display the translation results.

[0108] 2. Speech Recognition Engine

[0109] Software or hardware that converts a user's voice data into text data. Examples include Google Cloud Speech-to-Text and Amazon Transcribe.

[0110] 3. Translation AI

[0111] AI technology that translates text data from one language to another. Applications include DeepL, Google Cloud Translation API, and Microsoft Translator.

[0112] 4. Server

[0113] A central device that controls the capture of voice data, text conversion, translation, and sending and receiving of the results. Examples include AWS EC2 and Google Cloud Compute Engine.

[0114] 5. Display means

[0115] This refers to the technology and equipment used to display translated text data on the video call screen, either on the screen of the user's device or on the user interface (UI) of a dedicated application.

[0116] System Operation Overview

[0117] Before starting a video call, users set their preferred language (primary language) and the other party's preferred language (secondary language). Once the call is started, the following process is performed each time the user speaks:

[0118] 1. Audio Capture

[0119] When a user starts speaking, the voice data is captured through the device's microphone and sent to the server in real time.

[0120] 2. Speech Recognition and Text Conversion

[0121] The server passes the received voice data to a voice recognition engine and converts it into text data. For example, if user A says "Hello, how are you?", this voice becomes the text data "Hello, how are you?"

[0122] 3. Text Translation

[0123] The converted text data is passed to a translation AI and translated into the specified second language. In the example above, the text "Hello, how are you?" is translated into English as "Hello, how are you?"

[0124] 4. Displaying translated text

[0125] The translated text data is sent to the user's device and the other party's device via the server and displayed on the video call screen.

[0126] Specific examples

[0127] For example:

[0128] Consider the case where user A selects Japanese and the other user, user B, selects English. When user A says, "Hello, how are you?", the speech is captured on the device and sent to the server. The server uses a speech recognition engine to convert this speech into text, "Hello, how are you?" This text is sent to a translation AI, which translates it into English as "Hello, how are you?" The translated text is sent to the devices of user A and user B via the server and displayed on the video call screen.

[0129] Similarly, when User B says, "I am fine, thank you," the voice data is sent to the server and converted into text by a speech recognition engine. This text is then translated into Japanese by a translation AI as "I am fine, thank you." The translated text is then sent via the server again to the devices of User A and User B and displayed. By repeating this process, users can communicate in real time across language barriers.

[0130] Prompt Sentence Examples

[0131] Here are some example prompts to input to a generative AI model:

[0132] "Let's develop a video chat system with highly accurate speech recognition and real-time translation capabilities. Please tell us the specific steps involved. Please explain the roles of the user device, speech recognition engine, translation AI, server, and display means, and provide a detailed description of the processing steps, including setup and connection."

[0133] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0134] Step 1: Select your language

[0135] User action: The user selects their own language (first language) and the other person's language (second language) on the system settings screen.

[0136] Input: User-driven language selection through the interface.

[0137] Output: The system saves the primary and secondary language settings.

[0138] Step 2: Start a video call

[0139] User action: The user presses the start video call button.

[0140] Server operation: The server sends a connection request to the other device and establishes a video call connection.

[0141] Input: A request to start a video call initiated by the user.

[0142] Output: The server sends a connection request to the other user's device, and a video call connection is established.

[0143] Step 3: Capture audio

[0144] Device operation: When the user starts speaking, the device captures voice data in real time through the microphone.

[0145] Input: Audio capture via audio input device (microphone).

[0146] Output: The captured audio data is saved as internal data.

[0147] Step 4: Sending audio data to the server

[0148] Device operation: Captured audio data is sent to the server in real time.

[0149] Input: Audio capture data.

[0150] Output: The audio data is sent to the server.

[0151] Step 5: Speech recognition and text conversion

[0152] Server operation: The server passes the received voice data to a speech recognition engine and converts it into text data, for example, using Google Cloud Speech-to-Text.

[0153] Input: The audio data received by the server.

[0154] Output: Data converted into text by the speech recognition engine.

[0155] Step 6: Translate the text

[0156] Server operation: The converted text data is passed to a translation AI, which translates it into the specified second language, for example, using DeepL or the Google Cloud Translation API.

[0157] Input: Text data converted by speech recognition.

[0158] Output: Text data translated into a second language by the translation AI.

[0159] Step 7: Send and view translation results

[0160] Server operation: Translated text data is sent to the user and the other party's device.

[0161] Device operation: The device displays the received translated text data on the video call screen.

[0162] Input: The translated text data sent by the server.

[0163] Output: The translated text data is displayed on the user's and the recipient's devices.

[0164] As a concrete example, when user A says "Hello, how are you?", the speech is captured on the device and sent to the server. The server uses a speech recognition engine to convert this speech into text "Hello, how are you?", which is then passed to a translation AI that translates it into English as "Hello, how are you?" The translated text is sent to the devices of user A and user B and displayed on the video call screen.

[0165] (Application example 1)

[0166] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0167] In stores, it is difficult for foreign tourists and store staff who speak different languages ​​to communicate smoothly. Current tools and translation functions struggle to translate accurately in real time, resulting in frequent interruptions in conversations. This leads to a decline in the quality of customer service and impacts customer satisfaction. Therefore, there is a need for an effective communication support system with real-time translation functionality that can be used in stores.

[0168] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0169] In this invention, the server includes means for setting a first language and a second language for selection by a user, means for capturing voice data from a first terminal, means for converting the captured voice data into text data, means for translating the converted text data into the selected second language, means for displaying the translated text data, means for being installed in smart glasses or a smartphone and displaying the translated text in real time, and means for enabling smooth communication with customers in a store using the smart glasses or the smartphone, thereby enabling store clerks and foreign tourists to communicate in real time without worrying about language differences.

[0170] "Primary language" refers to the original language spoken by the user.

[0171] "Second language" refers to the language the user wishes to understand or the language the other person speaks.

[0172] A "user terminal" is a device used by a user, such as smart glasses or a smartphone.

[0173] "Voice data" refers to digital data of the voice spoken by the user.

[0174] "Text data" refers to character data obtained by converting voice data.

[0175] "Translation tools" refers to artificial intelligence technologies and software used to convert text data into different languages.

[0176] "Server" refers to a central system that processes and manages data.

[0177] "Display means" refers to the functionality for displaying text or information on devices such as smart glasses or smartphones.

[0178] "Real-time translation" refers to the process of providing translation results almost instantly after speech is spoken.

[0179] "Smart glasses" refers to a glasses-type device that displays information using AR technology.

[0180] A "smartphone" refers to a mobile phone with advanced computing and connectivity capabilities.

[0181] "Customer" refers to a shopper who visits a store or a person who uses a service.

[0182] "Store clerk" refers to an employee who deals with customers in a store.

[0183] "Translated text" refers to the result of translating text written in a source language into another language.

[0184] The system for implementing this invention is a real-time translation system for smooth communication between store clerks and foreign tourists. This system includes a user terminal, a voice recognition engine, a translation AI, and a server.

[0185] Program processing overview

[0186] The server performs a series of processes: capturing voice data from the user's device, converting it into text data, translating it into the selected second language, and displaying the translated text data. Specifically, via an application installed on smart glasses or a smartphone, the voice data is converted into text data using a voice recognition engine (e.g., Google Speech-to-Text API), and the text data is translated using a translation AI (e.g., DeepL API). The server manages this entire process and displays the translation results on the user's device in real time.

[0187] Hardware and software used

[0188] Hardware

[0189] Smart glasses (e.g., glasses-type devices using AR technology)

[0190] Smartphone

[0191] software

[0192] Speech recognition engine (e.g. Google Speech-to-Text API)

[0193] Translation AI (e.g. DeepL API)

[0194] Cloud server (e.g. AWS or Google Cloud)

[0195] Data processing and calculation flow

[0196] 1. Language Selection

[0197] The store clerk selects a foreign language that corresponds to his or her own language on the user terminal.

[0198] 2. Audio Capture

[0199] When a customer speaks into the microphone on their smart glasses or smartphone, voice data is captured.

[0200] 3. Voice Recognition

[0201] The captured voice data is sent to a cloud server and converted into text data using the Google Speech-to-Text API.

[0202] 4. Text Translation

[0203] The obtained text data is translated into the store clerk's language using the DeepL API.

[0204] 5. Displaying the translation results

[0205] The translated text is displayed in real time on smart glasses or smartphones, allowing store clerks to instantly understand the customer's questions and what they are saying.

[0206] Specific examples

[0207] For example, consider a case where a customer speaks Japanese and a store clerk speaks English. When the customer asks, "Where is this product?", the voice is captured by the microphone on the smart glasses or smartphone. The captured voice data is sent to a server and converted into text data, "Where is this product located?", using the Google Speech-to-Text API. This text data is then translated into English, "Where is this product located?", using the DeepL API, and the result is instantly displayed on the display of the smart glasses or smartphone. The store clerk can then look at the English translation and provide the customer with appropriate guidance.

[0208] Prompt Sentence Examples

[0209] Test the performance of your application by inputting prompt sentences like the following into your generative AI model:

[0210] Generate the following HTML code:

[0211] 1. JavaScript code to capture customer utterances.

[0212] 2. Send the captured audio data to the Google Speech-to-Text API.

[0213] 3. The resulting text is passed to the DeepL API for translation.

[0214] 4. HTML and CSS for displaying the translation results on the screen in real time.

[0215] Below are the details of the APIs we will use:

[0216] Google Speech-to-Text API: https: / / cloud.google.com / speech-to-text

[0217] DeepL API: https: / / www.deepl.com / pro-api

[0218] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0219] Step 1:

[0220] Language selection

[0221] The user selects their own language (first language) and the target foreign language (second language) on the device (smart glasses or smartphone).

[0222] Input: The primary and secondary languages ​​selected by the user.

[0223] Data processing: The selected language information is sent from the terminal to the server, and the initial connection settings are made.

[0224] Output: Language setting information stored on the server.

[0225] Step 2:

[0226] Audio Capture

[0227] When a customer speaks into the microphone on their smart glasses or smartphone, the voice data is captured by the device.

[0228] Input: Customer utterance (audio data).

[0229] Data processing: Voice data is collected in real time by the device and sent to the server.

[0230] Output: The audio data sent to the server.

[0231] Step 3:

[0232] Voice Recognition

[0233] The server receives the transmitted voice data and converts it into text data using a speech recognition engine (e.g., Google Speech-to-Text API).

[0234] Input: Audio data sent to the server.

[0235] Data calculation: The process of converting voice to text using a voice recognition engine.

[0236] Output: Text data (first language).

[0237] Step 4:

[0238] Text Translation

[0239] The server passes the text data to a translation AI (e.g., DeepL API) to translate it into the selected second language.

[0240] Input: Text data (primary language) generated by the speech recognition engine.

[0241] Data calculation: Translation processing of text data using translation AI.

[0242] Output: Translated text data (second language).

[0243] Step 5:

[0244] Displaying translation results

[0245] The server sends the translated text data to the user's device, where it is displayed on smart glasses or a smartphone.

[0246] Input: Translated text data (second language).

[0247] Data processing: Formatting text data and sending it to the user's device.

[0248] Output: Translated text displayed on smart glasses or smartphone.

[0249] Specific operation example

[0250] If a user selects English as their first language and Japanese as their second language, that information is sent to the server. When a customer speaks a question in Japanese, the voice data is captured by the microphone in the smart glasses and sent to the server. The server uses the Google Speech-to-Text API to convert the voice data into text data such as "Where is this product located?" This text data is then translated into English as "Where is this product located?" using the DeepL API and finally displayed on the smart glasses. The store clerk can view this translation result and immediately respond to the customer.

[0251] Prompt Sentence Examples

[0252] Generate the following HTML code:

[0253] 1. JavaScript code to capture customer utterances.

[0254] 2. Send the captured audio data to the Google Speech-to-Text API.

[0255] 3. The resulting text is passed to the DeepL API for translation.

[0256] 4. HTML and CSS for displaying the translation results on the screen in real time.

[0257] Below are the details of the APIs we will use:

[0258] Google Speech-to-Text API: https: / / cloud.google.com / speech-to-text

[0259] DeepL API: https: / / www.deepl.com / pro-api

[0260] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0261] The system based on this invention not only provides a video call service that translates languages ​​in real time, but also has the ability to recognize and display the user's emotions. Specifically, in addition to speech recognition technology and translation AI, it uses an emotion engine to recognize the user's emotions when speaking and displays that information on the video call screen, thereby achieving richer communication. The specific operating form of the system and its components are described below.

[0262] System Components

[0263] 1. User terminal: The device that a user uses to make a video call (e.g., smartphone, tablet, computer, etc.).

[0264] 2. Speech recognition engine: Software or hardware that converts a user's voice data into text data.

[0265] 3. Translation AI: Artificial intelligence technology for translating text data from one language to another.

[0266] 4. Emotion Engine: Technology for recognizing user emotions based on voice and text data.

[0267] 5. Server: The central device that controls voice data capture, text conversion, translation, emotion recognition, and sending and receiving the results.

[0268] 6. Display means: Technology and devices for displaying translated text data and emotion data on the video call screen.

[0269] System Operation Overview

[0270] 1. Language selection: First, the user selects their own language (first language) and the other person's language (second language) on the device's settings screen. The selected language information is sent to the server.

[0271] 2. Starting a video call: The user presses a button to start a video call, which sends a connection request to the server. The server then sends a request to the other device to establish the call.

[0272] 3. Voice capture: When the user starts speaking, the device captures voice data through the microphone and transmits it to the server in real time.

[0273] 4. Speech recognition and text conversion: The server inputs the received voice data into a speech recognition engine and converts it into text data.

[0274] 5. Text translation: The converted text data is passed from the server to a translation AI, which translates it into a second language.

[0275] 6. Emotion Recognition: The emotion engine recognizes the user's emotions based on the voice and text data.

[0276] 7. Displaying translated text and emotion data: The server sends the translated text data and the recognized emotion data to the user terminal and the other terminal, and displays them on the video call screen.

[0277] Specific examples

[0278] For example, consider the case where user A selects Japanese and the other user, user B, selects English. When user A says, "Hello, how are you?", the speech is captured on the device and sent to the server. The server uses a speech recognition engine to convert this speech into text, "Hello, how are you?" This text is sent to the translation AI, which translates it into English as "Hello, how are you?" Furthermore, this speech data and text data undergo emotion recognition by an emotion engine, which recognizes the emotion "friendly." The translated text and emotion data are sent to the devices of user A and user B via the server and displayed on the video call screen.

[0279] Similarly, when User B says, "I am fine, thank you," the voice data is also sent to the server and converted into text by the speech recognition engine. This text is then translated into Japanese by the translation AI as "I am fine, thank you." In this process, the emotion engine recognizes the emotion "happy." The translated text and emotion data are then sent again via the server to the devices of User A and User B and displayed.

[0280] This process allows users to communicate in real time, with greater depth and depth of emotion. By conveying emotions as well as translation, this system can build deeper understanding and bonds.

[0281] The processing flow will be explained below.

[0282] Program processing flow details

[0283] Step 1:

[0284] The user launches the application on their device and transitions to the main screen, at which point the account information is loaded.

[0285] Step 2:

[0286] The user accesses the language selection screen and sets their own language (first language) and the other party's language (second language). The terminal then sends the selected language information to the server.

[0287] Step 3:

[0288] To start a video call, the user presses a video call start button, and the terminal sends the request to the server.

[0289] Step 4:

[0290] The server receives the user's request and sends a connection request to the other party's terminal. If the other party accepts the request, the call is established.

[0291] Step 5:

[0292] When a user starts speaking, the device captures the voice data through the microphone, and the captured voice data is sent to the server in real time.

[0293] Step 6:

[0294] The server inputs the received voice data into a voice recognition engine and converts it into text data.

[0295] Step 7:

[0296] The server passes the converted text data to a translation AI, which translates it into a second language.

[0297] Step 8:

[0298] The server inputs the translated text data into an emotion engine to recognize the user's emotion.

[0299] Step 9:

[0300] The server transmits the translated text data and the recognized emotion data to the user terminal and the other terminal.

[0301] Step 10:

[0302] The device displays the received translated text and emotion data on the video call screen.

[0303] Step 11:

[0304] When the other person begins to speak, the other person's device also captures the voice data through the microphone and sends it to the server.

[0305] Step 12:

[0306] The server inputs the other party's voice data into a voice recognition engine and converts it into text data.

[0307] Step 13:

[0308] The server passes the converted text data to a translation AI, which translates it into the first language.

[0309] Step 14:

[0310] The server inputs the translated text data into an emotion engine to recognize the other person's emotions.

[0311] Step 15:

[0312] The server transmits the translated text data and the recognized emotion data to the user terminal and the other terminal.

[0313] Step 16:

[0314] The user terminal displays the received translated text and emotion data on the video call screen.

[0315] This process flow allows users to communicate with others in real time, overcoming language barriers and including emotions. Both users and others can receive information in language and emotions that they can instantly understand, resulting in smooth dialogue and deep understanding.

[0316] Example 2

[0317] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0318] Conventional real-time translation systems can convert voice data into text data and translate it, but they lack a means to convey user emotions. This makes it difficult for users to understand each other's emotions, which can lead to a decline in the quality of communication. For example, not being able to read emotions can lead to misunderstandings and miscommunication.

[0319] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for setting a first language and a second language for the user to select, means for capturing voice data from the first terminal, means for converting the captured voice data into text data, means for translating the converted text data into the selected second language, means for displaying the translated text data, means for recognizing the user's emotion based on the voice data and text data, and means for displaying the recognized emotion data on the video call screen. This allows the user to translate and convey emotions in real time, thereby achieving richer communication.

[0320] "Users" are people who use this system to make video calls.

[0321] "Primary language" refers to the user's preferred language.

[0322] A "secondary language" is another language that is used in a video call with a user-selected party.

[0323] A "terminal" is a device (e.g., a smartphone, tablet, computer) that a user uses to make a video call.

[0324] A "server" is a central device that controls voice data capture, text conversion, translation, emotion recognition, and result transmission and reception.

[0325] "Voice data" refers to digital voice information obtained by capturing a user's speech through a microphone.

[0326] "Text data" refers to character information converted from voice data using voice recognition technology.

[0327] "Translation" is the process of converting text data written in one language into text data written in another language.

[0328] "Emotion recognition" refers to a technology that analyzes and identifies a user's emotions based on voice and text data.

[0329] A "video call screen" refers to a display screen that displays video, audio, and supplemental information between users during a video call.

[0330] The system based on this invention has a function to recognize and display the user's emotions in addition to a video call service that translates languages ​​in real time. The main components of this system include a user terminal, a voice recognition engine, a translation AI, an emotion engine, a server, and a display means.

[0331] Hardware and software used

[0332] User device:

[0333] A user terminal is a device used by a user to make video calls, and can be a smartphone, tablet, computer, etc. This terminal has a built-in microphone and speaker, and has a video call application installed.

[0334] Speech Recognition Engine:

[0335] The speech recognition engine is used to convert the user's voice data into text data. Specifically, speech recognition technology such as Google Cloud Speech-to-Text is used.

[0336] Translation AI:

[0337] Artificial intelligence technology is used to translate the converted text data from one language to another, such as the Google Translate API.

[0338] Emotion Engine:

[0339] This technology is used to recognize user emotions based on voice and text data. Specifically, IBM Watson Tone Analyzer is used.

[0340] server:

[0341] The server is a central device that controls voice capture, text conversion, translation, emotion recognition, and sending and receiving the results. Typically, a cloud server (e.g., an AWS EC2 instance) is used.

[0342] Display means:

[0343] The display means is software or a device for displaying the translated text data and emotion data on the video call screen. A display built into the user terminal is used.

[0344] Specific examples

[0345] For example, consider the case where user A selects Japanese as their first language and the other user, user B, selects English as their second language. When user A says, "Hello, how are you?", the speech is captured on the device and sent to the server. The server uses a speech recognition engine to convert this speech into text, "Hello, how are you?" This text is sent to a translation AI, which translates it into English as "Hello, how are you?" This voice data and text data are then analyzed by an emotion engine, which recognizes the emotion "friendly." The translated text and emotion data are sent to the devices of user A and user B via the server and displayed on the video call screen.

[0346] Similarly, when User B says, "I am fine, thank you," the voice data is also sent to the server and converted into text by the speech recognition engine. This text is then translated into Japanese by the translation AI as "I am fine, thank you." In this process, the emotion engine recognizes the emotion "happy." The translated text and emotion data are then sent again via the server to the devices of User A and User B and displayed.

[0347] Prompt Sentence Examples

[0348] "Create a program that converts voice data into text in real time, translates that text into a specified language, recognizes emotions, and displays both translations and emotion data. The technologies used are a speech recognition engine, a translation AI, and an emotion engine."

[0349] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0350] Step 1: Select your language

[0351] The user selects their own language (first language) and the other person's language (second language) on the settings screen of a video calling app.

[0352] Input: The primary and secondary languages ​​selected by the user.

[0353] Output: The selected language information is sent to the server and stored.

[0354] Specific operation: User A selects Japanese as the first language and User B selects English as the second language. The device sends this information to the server, which stores it in a database.

[0355] Step 2: Start a video call

[0356] When a user presses the start button for a video call, the terminal sends a connection request to the server.

[0357] Input: Video call initiation request.

[0358] Output: The server sends a connection request to the other user's device and the video call is established.

[0359] Specific operation: When User A presses the "Start call" button, the device sends a connection request to the server. The server then sends a connection request to User B's device to establish a video call.

[0360] Step 3: Capture and send audio

[0361] When the user starts speaking, the device's microphone captures the voice data and transmits it to the server in real time.

[0362] Input: User speech.

[0363] Output: The captured audio data is sent to the server.

[0364] Specific operation: When user A says "Hello, how are you?", the device's microphone captures the voice and sends it to the server.

[0365] Step 4: Speech recognition and text conversion

[0366] The server inputs the received voice data into a voice recognition engine and converts it into text data.

[0367] Input: Captured audio data.

[0368] Output: The audio data is converted to text data.

[0369] Specific behavior: The server sends the audio data to Google Cloud Speech-to-Text, and the text "Hello, how are you?" is obtained.

[0370] Step 5: Translate the text

[0371] The server sends the text data obtained through voice recognition to a translation AI, which translates it into the specified second language.

[0372] Input: Text data obtained by speech recognition.

[0373] Output: The text data is translated into the second language.

[0374] Specific operation: The server sends text data to the Google Translate API, and "Hello, how are you?" is translated into "Hello, how are you?"

[0375] Step 6: Recognize emotions

[0376] The server inputs the voice data and text data into an emotion engine to recognize the user's emotion.

[0377] Input: Audio data and text data.

[0378] Output: Emotion data.

[0379] How it works: The server sends voice and text data to IBM Watson Tone Analyzer, which recognizes the emotion "friendly."

[0380] Step 7: Send and display translated text and emotion data

[0381] The server sends the translated text and recognized emotion data to both user terminals and displays them on the video call screen.

[0382] Input: translated text data and recognized emotion data.

[0383] Output: Display of translated text and emotion data on the device.

[0384] Specific operation: The server sends the translated text "Hello, how are you?" and the emotion data "friendly" to the devices of user A and user B, and displays them on their respective video call screens.

[0385] (Application example 2)

[0386] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0387] While systems already exist to enable smooth communication between users who speak different languages, recognizing emotions and responding quickly based on them is crucial, especially in security services where emergency response is required. However, existing systems lack the ability to recognize and display emotions, making communication in emergencies difficult. There is also a need for technology that can display emotions in real time on a video call screen while supporting multiple languages.

[0388] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0389] In this invention, the server includes means for setting a first language and a second language for selection by the user, means for capturing voice data from the first terminal, means for converting the captured voice data into text data, means for translating the converted text data into the selected second language, means for recognizing the user's emotion based on the voice data and text data, and means for displaying the recognized emotion data on the video call screen. This enables smooth communication between users who speak different languages, and makes it possible to quickly recognize the user's emotion and respond based on it, particularly in security service situations where emergency response is required.

[0390] The "means for setting a first language and a second language for the user to select" is a mechanism for the user making a video call to select the language to use and register that information in the system.

[0391] The "means for capturing voice data from the first terminal" is a function for collecting speech as voice data using a device (smartphone, tablet, computer, etc.) used by a user.

[0392] The "means for converting captured voice data into text data" is a technique for converting collected voice data into text data using voice recognition technology.

[0393] "Means for translating the converted text data into a selected second language" refers to a translation technique for converting the converted text data in a first language into another selected language.

[0394] The "means for recognizing a user's emotions based on voice data and text data" is a technology for analyzing collected voice and text data and determining the user's emotions at the time of speech.

[0395] The "means for displaying recognized emotion data on the video call screen" is a function for displaying the emotion determined by the emotion engine in real time on the terminal screen of the user making the video call.

[0396] The system based on this invention aims to improve emergency calls in security services, and is specifically designed to realize real-time translation between users who speak different languages ​​and emotion recognition.

[0397] System Components

[0398] 1. User Device

[0399] A device that allows users to make video calls. Typical examples include smartphones, tablets, and computers. These devices can capture audio and transmit it to a server in real time.

[0400] 2. Speech Recognition Engine

[0401] This is a technology that converts captured voice data into text data. A representative example of this software is "SpeechRecognition."

[0402] 3. Translation AI

[0403] It is a technology that translates text data from one language to another. By using "Google Trans," real-time translation between multiple languages ​​is possible.

[0404] 4. Emotion Engine

[0405] This is a technology for recognizing user emotions based on voice and text data. "EmotionRecognizer" is a representative example.

[0406] 5. Video Call System

[0407] This is a system for managing video calls between users. It includes OpenCV for video processing and a custom module for managing video calls.

[0408] 6. Server

[0409] It is the central device that controls voice data capture, text conversion, translation, emotion recognition, and sending and receiving the results.

[0410] System Operation Overview

[0411] 1. Language Selection

[0412] Users select their own language (first language) and the other person's language (second language) on the device settings screen of their smartphone or other device.

[0413] 2. Start a video call

[0414] The user presses a button to start a video call, which sends a connection request to the server, which then sends a request to the other device, and the video call is established.

[0415] 3. Audio Capture

[0416] When the user starts speaking, the device captures voice data through the microphone and transmits it to the server in real time.

[0417] 4. Speech Recognition and Text Conversion

[0418] The server inputs the received voice data into a voice recognition engine and converts it into text data.

[0419] 5. Text Translation

[0420] The converted text data is passed from the server to a translation AI and translated into a second language.

[0421] 6. Emotional Recognition

[0422] The emotion engine recognizes the user's emotion based on the voice data and text data.

[0423] 7. Displaying translated text and sentiment data

[0424] The server transmits the translated text data and the recognized emotion data to the user terminal and the other terminal, and displays them on the video call screen.

[0425] The specific hardware and software used

[0426] Hardware

[0427] Smartphones, tablets, and computers

[0428] software

[0429] OpenCV: Image processing for video calls

[0430] SpeechRecognition: Speech recognition

[0431] GoogleTrans: Real-time translation

[0432] EmotionRecognizer: Emotion Recognition from Audio Data

[0433] Custom module for video chat system: Call management and communication with the server

[0434] Specific examples

[0435] For example, if a security staff member on-site says, "We've spotted a suspicious person. What should we do?", this data is instantly captured and converted into text data by a speech recognition engine. The translation AI then translates this text and displays it in English. At the same time, the emotion engine recognizes this as "tension," and the control center immediately begins emergency response.

[0436] Prompt Sentence Examples

[0437] "Translate emergency calls in Japanese from security staff into English in real time, and recognize and display the emotions expressed. The emotion engine recognizes emotions from voice and text data, and displays those emotions on the video call screen."

[0438] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0439] Step 1:

[0440] The user selects a primary and secondary language on the device's settings screen. The user's selected language information is sent from the device to the server, allowing the server to identify the language pair for translation.

[0441] Step 2:

[0442] To start a video call, a user presses a video call start button on the terminal, and the terminal sends this request to the server, which then sends a call start request to the other terminal and establishes the call.

[0443] Step 3:

[0444] When a user starts speaking, voice data is captured through the device's microphone. This voice data is sent to the server in real time. The server then converts the input voice data into text data using a voice recognition engine.

[0445] Step 4:

[0446] The server passes the converted text data to a translation AI, which translates it into the selected second language. The input is the converted text data, and the output is the translated text data. This is done by the translation AI (Googletrans).

[0447] Step 5:

[0448] Using the emotion engine, the server recognizes the user's emotion based on the voice and text data. The input is the voice and text data, and the output is the recognized emotion data. This is done by EmotionRecognizer.

[0449] Step 6:

[0450] The translated text data and the recognized emotion data are sent from the server to the user's device and the other device. The device displays this data on the screen. As an output, the text translated into the other user's language and the emotion display are reflected on the video call screen in real time.

[0451] Step 7:

[0452] The user checks the translated text and emotion data displayed on the screen. This process allows users to communicate in real time with people who speak different languages ​​and progress through the dialogue while understanding the other person's emotions.

[0453] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0454] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0455] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0456] [Second embodiment]

[0457] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0458] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0459] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0460] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0461] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0462] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0463] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0464] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0465] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0466] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0467] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0468] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0469] The system based on this invention is a video calling service that allows users to smoothly communicate with people who speak different languages. Specifically, it has the function of translating languages ​​in real time using voice recognition technology and translation AI and displaying the translation content. The specific operation form of the system and its components are described below.

[0470] System Components

[0471] 1. User terminal: The device that a user uses to make a video call (e.g., smartphone, tablet, computer, etc.).

[0472] 2. Speech recognition engine: Software or hardware that converts a user's voice data into text data.

[0473] 3. Translation AI: Artificial intelligence technology for translating text data from one language to another.

[0474] 4. Server: The central device that controls the capture of voice data, text conversion, translation, and sending and receiving of the results.

[0475] 5. Display means: Technology or equipment for displaying translated text data on the video call screen.

[0476] System Operation Overview

[0477] 1. Language selection: First, the user selects their own language (first language) and the other person's language (second language) on the device's settings screen.

[0478] 2. Starting a video call: The user presses a button to start a video call, which sends a connection request to the server, which then sends this request to the other device and establishes the call.

[0479] 3. Voice capture: When the user starts speaking, the device captures voice data through the microphone and transmits it to the server in real time.

[0480] 4. Speech recognition and text conversion: The server passes the received voice data to a speech recognition engine and converts it into text data.

[0481] 5. Text translation: The converted text data is passed from the server to a translation AI, where it is translated into a second language.

[0482] 6. Displaying translated text: The translated text data is sent to the user's and the other party's devices via the server and displayed on the video call screen.

[0483] Specific examples

[0484] For example, consider the case where user A selects Japanese and the other user, user B, selects English. When user A says, "Hello, how are you?", the speech is captured on the device and sent to the server. The server uses a speech recognition engine to convert this speech into text, "Hello, how are you?" This text is sent to a translation AI, which translates it into English as "Hello, how are you?" The translated text is sent to the devices of user A and user B via the server and displayed on the video call screen.

[0485] Similarly, when User B says "I am fine, thank you," the voice data is sent to the server and converted into text by a speech recognition engine. This text is then translated into Japanese by a translation AI as "I am fine, thank you." The translated text is then sent via the server again to the devices of User A and User B and displayed.

[0486] By repeating this process, users and their contacts can communicate in real time, overcoming language barriers. This system is designed to improve usability and enable smooth two-way communication.

[0487] The processing flow will be explained below.

[0488] Program processing flow details

[0489] Step 1:

[0490] The user launches the application on their device and transitions to the main screen, at which point the account information is loaded.

[0491] Step 2:

[0492] The user accesses the language selection screen and sets their own language (first language) and the other party's language (second language). The terminal then sends the selected language information to the server.

[0493] Step 3:

[0494] To start a video call, the user presses a video call start button, and the terminal sends the request to the server.

[0495] Step 4:

[0496] The server receives the user's request and sends a connection request to the other party's terminal. If the other party accepts the request, the call is established.

[0497] Step 5:

[0498] When a user starts speaking, the device captures the voice data through the microphone, and the captured voice data is sent to the server in real time.

[0499] Step 6:

[0500] The server inputs the received voice data into a voice recognition engine and converts it into text data.

[0501] Step 7:

[0502] The server passes the converted text data to a translation AI, which translates it into a second language.

[0503] Step 8:

[0504] The server transmits the translated text data to the user terminal and the other terminal.

[0505] Step 9:

[0506] The terminal displays the received translated text on the video call screen.

[0507] Step 10:

[0508] When the other person begins to speak, the other person's device also captures the voice data through the microphone and sends it to the server.

[0509] Step 11:

[0510] The server inputs the other party's voice data into a voice recognition engine and converts it into text data.

[0511] Step 12:

[0512] The server passes the converted text data to a translation AI, which translates it into the first language.

[0513] Step 13:

[0514] The server transmits the translated text data to the user terminal and the other terminal.

[0515] Step 14:

[0516] The user terminal displays the received translated text on the video call screen.

[0517] This process flow enables users to communicate with others in real time, overcoming language barriers. Both users and others can receive information in a language they can instantly understand, ensuring smooth dialogue.

[0518] Example 1

[0519] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0520] It is difficult for users who speak different languages ​​to communicate smoothly. Real-time interpretation is particularly required during video calls, but achieving this is technically complex and expensive. Furthermore, existing translation systems have problems such as low translation accuracy and a lack of real-time capabilities, resulting in a poor user experience. The present invention aims to solve these problems by providing a system that enables users who speak different languages ​​to communicate smoothly in real time.

[0521] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0522] In this invention, the server includes means for setting a first language and a second language for selection by the user, means for capturing voice data from the first terminal, means for converting the captured voice data into text data, means for translating the converted text data into the selected second language, means for displaying the translated text data, means for sending a connection request and establishing a video call, means for transmitting and receiving voice data via the server, means for converting the voice data into text data using a voice recognition engine, means for translating the text data into a specified language in real time using a translation AI, and means for displaying the translated text data on the video call screen, thereby enabling users to smoothly communicate in real time with parties who speak different languages.

[0523] "User" means a person or end user who makes a video call using the System.

[0524] "Primary language" refers to the language that a user selects within the system to use.

[0525] "Second language" refers to the language used by the person selected by the user within the system.

[0526] A "terminal" is a device for making video calls, and includes a smartphone, tablet, computer, etc.

[0527] "Voice data" refers to a digital recording of a user's speech.

[0528] "Text data" is voice data converted into character information.

[0529] A "voice recognition engine" is software or hardware that converts voice data into text data.

[0530] "Translation AI" is an artificial intelligence technology that translates text data into other languages.

[0531] The "server" is a central device that manages the processing of the entire system, and transmits and receives voice data, converts text, translates, and transmits the results.

[0532] The "video call screen" is a part of the user interface for displaying translated text data during a video call.

[0533] A "connection request" is a request signal sent from a server to start a video call.

[0534] "Capture" refers to the act of acquiring audio data using an input device such as a terminal's microphone.

[0535] The system based on this invention is a video calling service that enables users who speak different languages ​​to communicate smoothly. This system uses speech recognition technology and translation AI to translate languages ​​in real time and display the translation results. The detailed configuration and operation of the system are described below.

[0536] System Components

[0537] 1. User Device

[0538] This is a device that allows users to make video calls, such as a smartphone, tablet, or computer. These devices must have a microphone for voice input and a screen to display the translation results.

[0539] 2. Speech Recognition Engine

[0540] Software or hardware that converts a user's voice data into text data. Examples include Google Cloud Speech-to-Text and Amazon Transcribe.

[0541] 3. Translation AI

[0542] AI technology that translates text data from one language to another. Applications include DeepL, Google Cloud Translation API, and Microsoft Translator.

[0543] 4. Server

[0544] A central device that controls the capture of voice data, text conversion, translation, and sending and receiving of the results. Examples include AWS EC2 and Google Cloud Compute Engine.

[0545] 5. Display means

[0546] This refers to the technology and equipment used to display translated text data on the video call screen, either on the screen of the user's device or on the user interface (UI) of a dedicated application.

[0547] System Operation Overview

[0548] Before starting a video call, users set their preferred language (primary language) and the other party's preferred language (secondary language). Once the call is started, the following process is performed each time the user speaks:

[0549] 1. Audio Capture

[0550] When a user starts speaking, the voice data is captured through the device's microphone and sent to the server in real time.

[0551] 2. Speech Recognition and Text Conversion

[0552] The server passes the received voice data to a voice recognition engine and converts it into text data. For example, if user A says "Hello, how are you?", this voice becomes the text data "Hello, how are you?"

[0553] 3. Text Translation

[0554] The converted text data is passed to a translation AI and translated into the specified second language. In the example above, the text "Hello, how are you?" is translated into English as "Hello, how are you?"

[0555] 4. Displaying translated text

[0556] The translated text data is sent to the user's device and the other party's device via the server and displayed on the video call screen.

[0557] Specific examples

[0558] For example:

[0559] Consider the case where user A selects Japanese and the other user, user B, selects English. When user A says, "Hello, how are you?", the speech is captured on the device and sent to the server. The server uses a speech recognition engine to convert this speech into text, "Hello, how are you?" This text is sent to a translation AI, which translates it into English as "Hello, how are you?" The translated text is sent to the devices of user A and user B via the server and displayed on the video call screen.

[0560] Similarly, when User B says, "I am fine, thank you," the voice data is sent to the server and converted into text by a speech recognition engine. This text is then translated into Japanese by a translation AI as "I am fine, thank you." The translated text is then sent via the server again to the devices of User A and User B and displayed. By repeating this process, users can communicate in real time across language barriers.

[0561] Prompt Sentence Examples

[0562] Here are some example prompts to input to a generative AI model:

[0563] "Let's develop a video chat system with highly accurate speech recognition and real-time translation capabilities. Please tell us the specific steps involved. Please explain the roles of the user device, speech recognition engine, translation AI, server, and display means, and provide a detailed description of the processing steps, including setup and connection."

[0564] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0565] Step 1: Select your language

[0566] User action: The user selects their own language (first language) and the other person's language (second language) on the system settings screen.

[0567] Input: User-driven language selection through the interface.

[0568] Output: The system saves the primary and secondary language settings.

[0569] Step 2: Start a video call

[0570] User action: The user presses the start video call button.

[0571] Server operation: The server sends a connection request to the other device and establishes a video call connection.

[0572] Input: A request to start a video call initiated by the user.

[0573] Output: The server sends a connection request to the other user's device, and a video call connection is established.

[0574] Step 3: Capture audio

[0575] Device operation: When the user starts speaking, the device captures voice data in real time through the microphone.

[0576] Input: Audio capture via audio input device (microphone).

[0577] Output: The captured audio data is saved as internal data.

[0578] Step 4: Sending audio data to the server

[0579] Device operation: Captured audio data is sent to the server in real time.

[0580] Input: Audio capture data.

[0581] Output: The audio data is sent to the server.

[0582] Step 5: Speech recognition and text conversion

[0583] Server operation: The server passes the received voice data to a speech recognition engine and converts it into text data, for example, using Google Cloud Speech-to-Text.

[0584] Input: The audio data received by the server.

[0585] Output: Data converted into text by the speech recognition engine.

[0586] Step 6: Translate the text

[0587] Server operation: The converted text data is passed to a translation AI, which translates it into the specified second language, for example, using DeepL or the Google Cloud Translation API.

[0588] Input: Text data converted by speech recognition.

[0589] Output: Text data translated into a second language by the translation AI.

[0590] Step 7: Send and view translation results

[0591] Server operation: Translated text data is sent to the user and the other party's device.

[0592] Device operation: The device displays the received translated text data on the video call screen.

[0593] Input: The translated text data sent by the server.

[0594] Output: The translated text data is displayed on the user's and the recipient's devices.

[0595] As a concrete example, when user A says "Hello, how are you?", the speech is captured on the device and sent to the server. The server uses a speech recognition engine to convert this speech into text "Hello, how are you?", which is then passed to a translation AI that translates it into English as "Hello, how are you?" The translated text is sent to the devices of user A and user B and displayed on the video call screen.

[0596] (Application example 1)

[0597] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0598] In stores, it is difficult for foreign tourists and store staff who speak different languages ​​to communicate smoothly. Current tools and translation functions struggle to translate accurately in real time, resulting in frequent interruptions in conversations. This leads to a decline in the quality of customer service and impacts customer satisfaction. Therefore, there is a need for an effective communication support system with real-time translation functionality that can be used in stores.

[0599] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0600] In this invention, the server includes means for setting a first language and a second language for selection by a user, means for capturing voice data from a first terminal, means for converting the captured voice data into text data, means for translating the converted text data into the selected second language, means for displaying the translated text data, means for being installed in smart glasses or a smartphone and displaying the translated text in real time, and means for enabling smooth communication with customers in a store using the smart glasses or the smartphone, thereby enabling store clerks and foreign tourists to communicate in real time without worrying about language differences.

[0601] "Primary language" refers to the original language spoken by the user.

[0602] "Second language" refers to the language the user wishes to understand or the language the other person speaks.

[0603] A "user terminal" is a device used by a user, such as smart glasses or a smartphone.

[0604] "Voice data" refers to digital data of the voice spoken by the user.

[0605] "Text data" refers to character data obtained by converting voice data.

[0606] "Translation tools" refers to artificial intelligence technologies and software used to convert text data into different languages.

[0607] "Server" refers to a central system that processes and manages data.

[0608] "Display means" refers to the functionality for displaying text or information on devices such as smart glasses or smartphones.

[0609] "Real-time translation" refers to the process of providing translation results almost instantly after speech is spoken.

[0610] "Smart glasses" refers to a glasses-type device that displays information using AR technology.

[0611] A "smartphone" refers to a mobile phone with advanced computing and connectivity capabilities.

[0612] "Customer" refers to a shopper who visits a store or a person who uses a service.

[0613] "Store clerk" refers to an employee who deals with customers in a store.

[0614] "Translated text" refers to the result of translating text written in a source language into another language.

[0615] The system for implementing this invention is a real-time translation system for smooth communication between store clerks and foreign tourists. This system includes a user terminal, a voice recognition engine, a translation AI, and a server.

[0616] Program processing overview

[0617] The server performs a series of processes: capturing voice data from the user's device, converting it into text data, translating it into the selected second language, and displaying the translated text data. Specifically, via an application installed on smart glasses or a smartphone, the voice data is converted into text data using a voice recognition engine (e.g., Google Speech-to-Text API), and the text data is translated using a translation AI (e.g., DeepL API). The server manages this entire process and displays the translation results on the user's device in real time.

[0618] Hardware and software used

[0619] Hardware

[0620] Smart glasses (e.g., glasses-type devices using AR technology)

[0621] Smartphone

[0622] software

[0623] Speech recognition engine (e.g. Google Speech-to-Text API)

[0624] Translation AI (e.g. DeepL API)

[0625] Cloud server (e.g. AWS or Google Cloud)

[0626] Data processing and calculation flow

[0627] 1. Language Selection

[0628] The store clerk selects a foreign language that corresponds to his or her own language on the user terminal.

[0629] 2. Audio Capture

[0630] When a customer speaks into the microphone on their smart glasses or smartphone, voice data is captured.

[0631] 3. Voice Recognition

[0632] The captured voice data is sent to a cloud server and converted into text data using the Google Speech-to-Text API.

[0633] 4. Text Translation

[0634] The obtained text data is translated into the store clerk's language using the DeepL API.

[0635] 5. Displaying the translation results

[0636] The translated text is displayed in real time on smart glasses or smartphones, allowing store clerks to instantly understand the customer's questions and what they are saying.

[0637] Specific examples

[0638] For example, consider a case where a customer speaks Japanese and a store clerk speaks English. When the customer asks, "Where is this product?", the voice is captured by the microphone on the smart glasses or smartphone. The captured voice data is sent to a server and converted into text data, "Where is this product located?", using the Google Speech-to-Text API. This text data is then translated into English, "Where is this product located?", using the DeepL API, and the result is instantly displayed on the display of the smart glasses or smartphone. The store clerk can then look at the English translation and provide the customer with appropriate guidance.

[0639] Prompt Sentence Examples

[0640] Test the performance of your application by inputting prompt sentences like the following into your generative AI model:

[0641] Generate the following HTML code:

[0642] 1. JavaScript code to capture customer utterances.

[0643] 2. Send the captured audio data to the Google Speech-to-Text API.

[0644] 3. The resulting text is passed to the DeepL API for translation.

[0645] 4. HTML and CSS for displaying the translation results on the screen in real time.

[0646] Below are the details of the APIs we will use:

[0647] Google Speech-to-Text API: https: / / cloud.google.com / speech-to-text

[0648] DeepL API: https: / / www.deepl.com / pro-api

[0649] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0650] Step 1:

[0651] Language selection

[0652] The user selects their own language (first language) and the target foreign language (second language) on the device (smart glasses or smartphone).

[0653] Input: The primary and secondary languages ​​selected by the user.

[0654] Data processing: The selected language information is sent from the terminal to the server, and the initial connection settings are made.

[0655] Output: Language setting information stored on the server.

[0656] Step 2:

[0657] Audio Capture

[0658] When a customer speaks into the microphone on their smart glasses or smartphone, the voice data is captured by the device.

[0659] Input: Customer utterance (audio data).

[0660] Data processing: Voice data is collected in real time by the device and sent to the server.

[0661] Output: The audio data sent to the server.

[0662] Step 3:

[0663] Voice Recognition

[0664] The server receives the transmitted voice data and converts it into text data using a speech recognition engine (e.g., Google Speech-to-Text API).

[0665] Input: Audio data sent to the server.

[0666] Data calculation: The process of converting voice to text using a voice recognition engine.

[0667] Output: Text data (first language).

[0668] Step 4:

[0669] Text Translation

[0670] The server passes the text data to a translation AI (e.g., DeepL API) to translate it into the selected second language.

[0671] Input: Text data (primary language) generated by the speech recognition engine.

[0672] Data calculation: Translation processing of text data using translation AI.

[0673] Output: Translated text data (second language).

[0674] Step 5:

[0675] Displaying translation results

[0676] The server sends the translated text data to the user's device, where it is displayed on smart glasses or a smartphone.

[0677] Input: Translated text data (second language).

[0678] Data processing: Formatting text data and sending it to the user's device.

[0679] Output: Translated text displayed on smart glasses or smartphone.

[0680] Specific operation example

[0681] If a user selects English as their first language and Japanese as their second language, that information is sent to the server. When a customer speaks a question in Japanese, the voice data is captured by the microphone in the smart glasses and sent to the server. The server uses the Google Speech-to-Text API to convert the voice data into text data such as "Where is this product located?" This text data is then translated into English as "Where is this product located?" using the DeepL API and finally displayed on the smart glasses. The store clerk can view this translation result and immediately respond to the customer.

[0682] Prompt Sentence Examples

[0683] Generate the following HTML code:

[0684] 1. JavaScript code to capture customer utterances.

[0685] 2. Send the captured audio data to the Google Speech-to-Text API.

[0686] 3. The resulting text is passed to the DeepL API for translation.

[0687] 4. HTML and CSS for displaying the translation results on the screen in real time.

[0688] Below are the details of the APIs we will use:

[0689] Google Speech-to-Text API: https: / / cloud.google.com / speech-to-text

[0690] DeepL API: https: / / www.deepl.com / pro-api

[0691] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0692] The system based on this invention not only provides a video call service that translates languages ​​in real time, but also has the ability to recognize and display the user's emotions. Specifically, in addition to speech recognition technology and translation AI, it uses an emotion engine to recognize the user's emotions when speaking and displays that information on the video call screen, thereby achieving richer communication. The specific operating form of the system and its components are described below.

[0693] System Components

[0694] 1. User terminal: The device that a user uses to make a video call (e.g., smartphone, tablet, computer, etc.).

[0695] 2. Speech recognition engine: Software or hardware that converts a user's voice data into text data.

[0696] 3. Translation AI: Artificial intelligence technology for translating text data from one language to another.

[0697] 4. Emotion Engine: Technology for recognizing user emotions based on voice and text data.

[0698] 5. Server: The central device that controls voice data capture, text conversion, translation, emotion recognition, and sending and receiving the results.

[0699] 6. Display means: Technology and devices for displaying translated text data and emotion data on the video call screen.

[0700] System Operation Overview

[0701] 1. Language selection: First, the user selects their own language (first language) and the other person's language (second language) on the device's settings screen. The selected language information is sent to the server.

[0702] 2. Starting a video call: The user presses a button to start a video call, which sends a connection request to the server. The server then sends a request to the other device to establish the call.

[0703] 3. Voice capture: When the user starts speaking, the device captures voice data through the microphone and transmits it to the server in real time.

[0704] 4. Speech recognition and text conversion: The server inputs the received voice data into a speech recognition engine and converts it into text data.

[0705] 5. Text translation: The converted text data is passed from the server to a translation AI, which translates it into a second language.

[0706] 6. Emotion Recognition: The emotion engine recognizes the user's emotions based on the voice and text data.

[0707] 7. Displaying translated text and emotion data: The server sends the translated text data and the recognized emotion data to the user terminal and the other terminal, and displays them on the video call screen.

[0708] Specific examples

[0709] For example, consider the case where user A selects Japanese and the other user, user B, selects English. When user A says, "Hello, how are you?", the speech is captured on the device and sent to the server. The server uses a speech recognition engine to convert this speech into text, "Hello, how are you?" This text is sent to the translation AI, which translates it into English as "Hello, how are you?" Furthermore, this speech data and text data undergo emotion recognition by an emotion engine, which recognizes the emotion "friendly." The translated text and emotion data are sent to the devices of user A and user B via the server and displayed on the video call screen.

[0710] Similarly, when User B says, "I am fine, thank you," the voice data is also sent to the server and converted into text by the speech recognition engine. This text is then translated into Japanese by the translation AI as "I am fine, thank you." In this process, the emotion engine recognizes the emotion "happy." The translated text and emotion data are then sent again via the server to the devices of User A and User B and displayed.

[0711] This process allows users to communicate in real time, with greater depth and depth of emotion. By conveying emotions as well as translation, this system can build deeper understanding and bonds.

[0712] The processing flow will be explained below.

[0713] Program processing flow details

[0714] Step 1:

[0715] The user launches the application on their device and transitions to the main screen, at which point the account information is loaded.

[0716] Step 2:

[0717] The user accesses the language selection screen and sets their own language (first language) and the other party's language (second language). The terminal then sends the selected language information to the server.

[0718] Step 3:

[0719] To start a video call, the user presses a video call start button, and the terminal sends the request to the server.

[0720] Step 4:

[0721] The server receives the user's request and sends a connection request to the other party's terminal. If the other party accepts the request, the call is established.

[0722] Step 5:

[0723] When a user starts speaking, the device captures the voice data through the microphone, and the captured voice data is sent to the server in real time.

[0724] Step 6:

[0725] The server inputs the received voice data into a voice recognition engine and converts it into text data.

[0726] Step 7:

[0727] The server passes the converted text data to a translation AI, which translates it into a second language.

[0728] Step 8:

[0729] The server inputs the translated text data into an emotion engine to recognize the user's emotion.

[0730] Step 9:

[0731] The server transmits the translated text data and the recognized emotion data to the user terminal and the other terminal.

[0732] Step 10:

[0733] The device displays the received translated text and emotion data on the video call screen.

[0734] Step 11:

[0735] When the other person begins to speak, the other person's device also captures the voice data through the microphone and sends it to the server.

[0736] Step 12:

[0737] The server inputs the other party's voice data into a voice recognition engine and converts it into text data.

[0738] Step 13:

[0739] The server passes the converted text data to a translation AI, which translates it into the first language.

[0740] Step 14:

[0741] The server inputs the translated text data into an emotion engine to recognize the other person's emotions.

[0742] Step 15:

[0743] The server transmits the translated text data and the recognized emotion data to the user terminal and the other terminal.

[0744] Step 16:

[0745] The user terminal displays the received translated text and emotion data on the video call screen.

[0746] This process flow allows users to communicate with others in real time, overcoming language barriers and including emotions. Both users and others can receive information in language and emotions that they can instantly understand, resulting in smooth dialogue and deep understanding.

[0747] Example 2

[0748] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0749] Conventional real-time translation systems can convert voice data into text data and translate it, but they lack a means to convey user emotions. This makes it difficult for users to understand each other's emotions, which can lead to a decline in the quality of communication. For example, not being able to read emotions can lead to misunderstandings and miscommunication.

[0750] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for setting a first language and a second language for the user to select, means for capturing voice data from the first terminal, means for converting the captured voice data into text data, means for translating the converted text data into the selected second language, means for displaying the translated text data, means for recognizing the user's emotion based on the voice data and text data, and means for displaying the recognized emotion data on the video call screen. This allows the user to translate and convey emotions in real time, thereby achieving richer communication.

[0751] "Users" are people who use this system to make video calls.

[0752] "Primary language" refers to the user's preferred language.

[0753] A "secondary language" is another language that is used in a video call with a user-selected party.

[0754] A "terminal" is a device (e.g., a smartphone, tablet, computer) that a user uses to make a video call.

[0755] A "server" is a central device that controls voice data capture, text conversion, translation, emotion recognition, and result transmission and reception.

[0756] "Voice data" refers to digital voice information obtained by capturing a user's speech through a microphone.

[0757] "Text data" refers to character information converted from voice data using voice recognition technology.

[0758] "Translation" is the process of converting text data written in one language into text data written in another language.

[0759] "Emotion recognition" refers to a technology that analyzes and identifies a user's emotions based on voice and text data.

[0760] A "video call screen" refers to a display screen that displays video, audio, and supplemental information between users during a video call.

[0761] The system based on this invention has a function to recognize and display the user's emotions in addition to a video call service that translates languages ​​in real time. The main components of this system include a user terminal, a voice recognition engine, a translation AI, an emotion engine, a server, and a display means.

[0762] Hardware and software used

[0763] User device:

[0764] A user terminal is a device used by a user to make video calls, and can be a smartphone, tablet, computer, etc. This terminal has a built-in microphone and speaker, and has a video call application installed.

[0765] Speech Recognition Engine:

[0766] The speech recognition engine is used to convert the user's voice data into text data. Specifically, speech recognition technology such as Google Cloud Speech-to-Text is used.

[0767] Translation AI:

[0768] Artificial intelligence technology is used to translate the converted text data from one language to another, such as the Google Translate API.

[0769] Emotion Engine:

[0770] This technology is used to recognize user emotions based on voice and text data. Specifically, IBM Watson Tone Analyzer is used.

[0771] server:

[0772] The server is a central device that controls voice capture, text conversion, translation, emotion recognition, and sending and receiving the results. Typically, a cloud server (e.g., an AWS EC2 instance) is used.

[0773] Display means:

[0774] The display means is software or a device for displaying the translated text data and emotion data on the video call screen. A display built into the user terminal is used.

[0775] Specific examples

[0776] For example, consider the case where user A selects Japanese as their first language and the other user, user B, selects English as their second language. When user A says, "Hello, how are you?", the speech is captured on the device and sent to the server. The server uses a speech recognition engine to convert this speech into text, "Hello, how are you?" This text is sent to a translation AI, which translates it into English as "Hello, how are you?" This voice data and text data are then analyzed by an emotion engine, which recognizes the emotion "friendly." The translated text and emotion data are sent to the devices of user A and user B via the server and displayed on the video call screen.

[0777] Similarly, when User B says, "I am fine, thank you," the voice data is also sent to the server and converted into text by the speech recognition engine. This text is then translated into Japanese by the translation AI as "I am fine, thank you." In this process, the emotion engine recognizes the emotion "happy." The translated text and emotion data are then sent again via the server to the devices of User A and User B and displayed.

[0778] Prompt Sentence Examples

[0779] "Create a program that converts voice data into text in real time, translates that text into a specified language, recognizes emotions, and displays both translations and emotion data. The technologies used are a speech recognition engine, a translation AI, and an emotion engine."

[0780] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0781] Step 1: Select your language

[0782] The user selects their own language (first language) and the other person's language (second language) on the settings screen of a video calling app.

[0783] Input: The primary and secondary languages ​​selected by the user.

[0784] Output: The selected language information is sent to the server and stored.

[0785] Specific operation: User A selects Japanese as the first language and User B selects English as the second language. The device sends this information to the server, which stores it in a database.

[0786] Step 2: Start a video call

[0787] When a user presses the start button for a video call, the terminal sends a connection request to the server.

[0788] Input: Video call initiation request.

[0789] Output: The server sends a connection request to the other user's device and the video call is established.

[0790] Specific operation: When User A presses the "Start call" button, the device sends a connection request to the server. The server then sends a connection request to User B's device to establish a video call.

[0791] Step 3: Capture and send audio

[0792] When the user starts speaking, the device's microphone captures the voice data and transmits it to the server in real time.

[0793] Input: User speech.

[0794] Output: The captured audio data is sent to the server.

[0795] Specific operation: When user A says "Hello, how are you?", the device's microphone captures the voice and sends it to the server.

[0796] Step 4: Speech recognition and text conversion

[0797] The server inputs the received voice data into a voice recognition engine and converts it into text data.

[0798] Input: Captured audio data.

[0799] Output: The audio data is converted to text data.

[0800] Specific behavior: The server sends the audio data to Google Cloud Speech-to-Text, and the text "Hello, how are you?" is obtained.

[0801] Step 5: Translate the text

[0802] The server sends the text data obtained through voice recognition to a translation AI, which translates it into the specified second language.

[0803] Input: Text data obtained by speech recognition.

[0804] Output: The text data is translated into the second language.

[0805] Specific operation: The server sends text data to the Google Translate API, and "Hello, how are you?" is translated into "Hello, how are you?"

[0806] Step 6: Recognize emotions

[0807] The server inputs the voice data and text data into an emotion engine to recognize the user's emotion.

[0808] Input: Audio data and text data.

[0809] Output: Emotion data.

[0810] How it works: The server sends voice and text data to IBM Watson Tone Analyzer, which recognizes the emotion "friendly."

[0811] Step 7: Send and display translated text and emotion data

[0812] The server sends the translated text and recognized emotion data to both user terminals and displays them on the video call screen.

[0813] Input: translated text data and recognized emotion data.

[0814] Output: Display of translated text and emotion data on the device.

[0815] Specific operation: The server sends the translated text "Hello, how are you?" and the emotion data "friendly" to the devices of user A and user B, and displays them on their respective video call screens.

[0816] (Application example 2)

[0817] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0818] While systems already exist to enable smooth communication between users who speak different languages, recognizing emotions and responding quickly based on them is crucial, especially in security services where emergency response is required. However, existing systems lack the ability to recognize and display emotions, making communication in emergencies difficult. There is also a need for technology that can display emotions in real time on a video call screen while supporting multiple languages.

[0819] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0820] In this invention, the server includes means for setting a first language and a second language for selection by the user, means for capturing voice data from the first terminal, means for converting the captured voice data into text data, means for translating the converted text data into the selected second language, means for recognizing the user's emotion based on the voice data and text data, and means for displaying the recognized emotion data on the video call screen. This enables smooth communication between users who speak different languages, and makes it possible to quickly recognize the user's emotion and respond based on it, particularly in security service situations where emergency response is required.

[0821] The "means for setting a first language and a second language for the user to select" is a mechanism for the user making a video call to select the language to use and register that information in the system.

[0822] The "means for capturing voice data from the first terminal" is a function for collecting speech as voice data using a device (smartphone, tablet, computer, etc.) used by a user.

[0823] The "means for converting captured voice data into text data" is a technique for converting collected voice data into text data using voice recognition technology.

[0824] "Means for translating the converted text data into a selected second language" refers to a translation technique for converting the converted text data in a first language into another selected language.

[0825] The "means for recognizing a user's emotions based on voice data and text data" is a technology for analyzing collected voice and text data and determining the user's emotions at the time of speech.

[0826] The "means for displaying recognized emotion data on the video call screen" is a function for displaying the emotion determined by the emotion engine in real time on the terminal screen of the user making the video call.

[0827] The system based on this invention aims to improve emergency calls in security services, and is specifically designed to realize real-time translation between users who speak different languages ​​and emotion recognition.

[0828] System Components

[0829] 1. User Device

[0830] A device that allows users to make video calls. Typical examples include smartphones, tablets, and computers. These devices can capture audio and transmit it to a server in real time.

[0831] 2. Speech Recognition Engine

[0832] This is a technology that converts captured voice data into text data. A representative example of this software is "SpeechRecognition."

[0833] 3. Translation AI

[0834] It is a technology that translates text data from one language to another. By using "Google Trans," real-time translation between multiple languages ​​is possible.

[0835] 4. Emotion Engine

[0836] This is a technology for recognizing user emotions based on voice and text data. "EmotionRecognizer" is a representative example.

[0837] 5. Video Call System

[0838] This is a system for managing video calls between users. It includes OpenCV for video processing and a custom module for managing video calls.

[0839] 6. Server

[0840] It is the central device that controls voice data capture, text conversion, translation, emotion recognition, and sending and receiving the results.

[0841] System Operation Overview

[0842] 1. Language Selection

[0843] Users select their own language (first language) and the other person's language (second language) on the device settings screen of their smartphone or other device.

[0844] 2. Start a video call

[0845] The user presses a button to start a video call, which sends a connection request to the server, which then sends a request to the other device, and the video call is established.

[0846] 3. Audio Capture

[0847] When the user starts speaking, the device captures voice data through the microphone and transmits it to the server in real time.

[0848] 4. Speech Recognition and Text Conversion

[0849] The server inputs the received voice data into a voice recognition engine and converts it into text data.

[0850] 5. Text Translation

[0851] The converted text data is passed from the server to a translation AI and translated into a second language.

[0852] 6. Emotional Recognition

[0853] The emotion engine recognizes the user's emotion based on the voice data and text data.

[0854] 7. Displaying translated text and sentiment data

[0855] The server transmits the translated text data and the recognized emotion data to the user terminal and the other terminal, and displays them on the video call screen.

[0856] The specific hardware and software used

[0857] Hardware

[0858] Smartphones, tablets, and computers

[0859] software

[0860] OpenCV: Image processing for video calls

[0861] SpeechRecognition: Speech recognition

[0862] GoogleTrans: Real-time translation

[0863] EmotionRecognizer: Emotion Recognition from Audio Data

[0864] Custom module for video chat system: Call management and communication with the server

[0865] Specific examples

[0866] For example, if a security staff member on-site says, "We've spotted a suspicious person. What should we do?", this data is instantly captured and converted into text data by a speech recognition engine. The translation AI then translates this text and displays it in English. At the same time, the emotion engine recognizes this as "tension," and the control center immediately begins emergency response.

[0867] Prompt Sentence Examples

[0868] "Translate emergency calls in Japanese from security staff into English in real time, and recognize and display the emotions expressed. The emotion engine recognizes emotions from voice and text data, and displays those emotions on the video call screen."

[0869] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0870] Step 1:

[0871] The user selects a primary and secondary language on the device's settings screen. The user's selected language information is sent from the device to the server, allowing the server to identify the language pair for translation.

[0872] Step 2:

[0873] To start a video call, a user presses a video call start button on the terminal, and the terminal sends this request to the server, which then sends a call start request to the other terminal and establishes the call.

[0874] Step 3:

[0875] When a user starts speaking, voice data is captured through the device's microphone. This voice data is sent to the server in real time. The server then converts the input voice data into text data using a voice recognition engine.

[0876] Step 4:

[0877] The server passes the converted text data to a translation AI, which translates it into the selected second language. The input is the converted text data, and the output is the translated text data. This is done by the translation AI (Googletrans).

[0878] Step 5:

[0879] Using the emotion engine, the server recognizes the user's emotion based on the voice and text data. The input is the voice and text data, and the output is the recognized emotion data. This is done by EmotionRecognizer.

[0880] Step 6:

[0881] The translated text data and the recognized emotion data are sent from the server to the user's device and the other device. The device displays this data on the screen. As an output, the text translated into the other user's language and the emotion display are reflected on the video call screen in real time.

[0882] Step 7:

[0883] The user checks the translated text and emotion data displayed on the screen. This process allows users to communicate in real time with people who speak different languages ​​and progress through the dialogue while understanding the other person's emotions.

[0884] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0885] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0886] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0887] [Third embodiment]

[0888] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0889] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0890] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0891] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0892] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0893] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0894] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0895] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0896] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0897] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0898] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0899] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0900] The system based on this invention is a video calling service that allows users to smoothly communicate with people who speak different languages. Specifically, it has the function of translating languages ​​in real time using voice recognition technology and translation AI and displaying the translation content. The specific operation form of the system and its components are described below.

[0901] System Components

[0902] 1. User terminal: The device that a user uses to make a video call (e.g., smartphone, tablet, computer, etc.).

[0903] 2. Speech recognition engine: Software or hardware that converts a user's voice data into text data.

[0904] 3. Translation AI: Artificial intelligence technology for translating text data from one language to another.

[0905] 4. Server: The central device that controls the capture of voice data, text conversion, translation, and sending and receiving of the results.

[0906] 5. Display means: Technology or equipment for displaying translated text data on the video call screen.

[0907] System Operation Overview

[0908] 1. Language selection: First, the user selects their own language (first language) and the other person's language (second language) on the device's settings screen.

[0909] 2. Starting a video call: The user presses a button to start a video call, which sends a connection request to the server, which then sends this request to the other device and establishes the call.

[0910] 3. Voice capture: When the user starts speaking, the device captures voice data through the microphone and transmits it to the server in real time.

[0911] 4. Speech recognition and text conversion: The server passes the received voice data to a speech recognition engine and converts it into text data.

[0912] 5. Text translation: The converted text data is passed from the server to a translation AI, where it is translated into a second language.

[0913] 6. Displaying translated text: The translated text data is sent to the user's and the other party's devices via the server and displayed on the video call screen.

[0914] Specific examples

[0915] For example, consider the case where user A selects Japanese and the other user, user B, selects English. When user A says, "Hello, how are you?", the speech is captured on the device and sent to the server. The server uses a speech recognition engine to convert this speech into text, "Hello, how are you?" This text is sent to a translation AI, which translates it into English as "Hello, how are you?" The translated text is sent to the devices of user A and user B via the server and displayed on the video call screen.

[0916] Similarly, when User B says "I am fine, thank you," the voice data is sent to the server and converted into text by a speech recognition engine. This text is then translated into Japanese by a translation AI as "I am fine, thank you." The translated text is then sent via the server again to the devices of User A and User B and displayed.

[0917] By repeating this process, users and their contacts can communicate in real time, overcoming language barriers. This system is designed to improve usability and enable smooth two-way communication.

[0918] The processing flow will be explained below.

[0919] Program processing flow details

[0920] Step 1:

[0921] The user launches the application on their device and transitions to the main screen, at which point the account information is loaded.

[0922] Step 2:

[0923] The user accesses the language selection screen and sets their own language (first language) and the other party's language (second language). The terminal then sends the selected language information to the server.

[0924] Step 3:

[0925] To start a video call, the user presses a video call start button, and the terminal sends the request to the server.

[0926] Step 4:

[0927] The server receives the user's request and sends a connection request to the other party's terminal. If the other party accepts the request, the call is established.

[0928] Step 5:

[0929] When a user starts speaking, the device captures the voice data through the microphone, and the captured voice data is sent to the server in real time.

[0930] Step 6:

[0931] The server inputs the received voice data into a voice recognition engine and converts it into text data.

[0932] Step 7:

[0933] The server passes the converted text data to a translation AI, which translates it into a second language.

[0934] Step 8:

[0935] The server transmits the translated text data to the user terminal and the other terminal.

[0936] Step 9:

[0937] The terminal displays the received translated text on the video call screen.

[0938] Step 10:

[0939] When the other person begins to speak, the other person's device also captures the voice data through the microphone and sends it to the server.

[0940] Step 11:

[0941] The server inputs the other party's voice data into a voice recognition engine and converts it into text data.

[0942] Step 12:

[0943] The server passes the converted text data to a translation AI, which translates it into the first language.

[0944] Step 13:

[0945] The server transmits the translated text data to the user terminal and the other terminal.

[0946] Step 14:

[0947] The user terminal displays the received translated text on the video call screen.

[0948] This process flow enables users to communicate with others in real time, overcoming language barriers. Both users and others can receive information in a language they can instantly understand, ensuring smooth dialogue.

[0949] Example 1

[0950] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0951] It is difficult for users who speak different languages ​​to communicate smoothly. Real-time interpretation is particularly required during video calls, but achieving this is technically complex and expensive. Furthermore, existing translation systems have problems such as low translation accuracy and a lack of real-time capabilities, resulting in a poor user experience. The present invention aims to solve these problems by providing a system that enables users who speak different languages ​​to communicate smoothly in real time.

[0952] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0953] In this invention, the server includes means for setting a first language and a second language for selection by the user, means for capturing voice data from the first terminal, means for converting the captured voice data into text data, means for translating the converted text data into the selected second language, means for displaying the translated text data, means for sending a connection request and establishing a video call, means for transmitting and receiving voice data via the server, means for converting the voice data into text data using a voice recognition engine, means for translating the text data into a specified language in real time using a translation AI, and means for displaying the translated text data on the video call screen, thereby enabling users to smoothly communicate in real time with parties who speak different languages.

[0954] "User" means a person or end user who makes a video call using the System.

[0955] "Primary language" refers to the language that a user selects within the system to use.

[0956] "Second language" refers to the language used by the person selected by the user within the system.

[0957] A "terminal" is a device for making video calls, and includes a smartphone, tablet, computer, etc.

[0958] "Voice data" refers to a digital recording of a user's speech.

[0959] "Text data" is voice data converted into character information.

[0960] A "voice recognition engine" is software or hardware that converts voice data into text data.

[0961] "Translation AI" is an artificial intelligence technology that translates text data into other languages.

[0962] The "server" is a central device that manages the processing of the entire system, and transmits and receives voice data, converts text, translates, and transmits the results.

[0963] The "video call screen" is a part of the user interface for displaying translated text data during a video call.

[0964] A "connection request" is a request signal sent from a server to start a video call.

[0965] "Capture" refers to the act of acquiring audio data using an input device such as a terminal's microphone.

[0966] The system based on this invention is a video calling service that enables users who speak different languages ​​to communicate smoothly. This system uses speech recognition technology and translation AI to translate languages ​​in real time and display the translation results. The detailed configuration and operation of the system are described below.

[0967] System Components

[0968] 1. User Device

[0969] This is a device that allows users to make video calls, such as a smartphone, tablet, or computer. These devices must have a microphone for voice input and a screen to display the translation results.

[0970] 2. Speech Recognition Engine

[0971] Software or hardware that converts a user's voice data into text data. Examples include Google Cloud Speech-to-Text and Amazon Transcribe.

[0972] 3. Translation AI

[0973] AI technology that translates text data from one language to another. Applications include DeepL, Google Cloud Translation API, and Microsoft Translator.

[0974] 4. Server

[0975] A central device that controls the capture of voice data, text conversion, translation, and sending and receiving of the results. Examples include AWS EC2 and Google Cloud Compute Engine.

[0976] 5. Display means

[0977] This refers to the technology and equipment used to display translated text data on the video call screen, either on the screen of the user's device or on the user interface (UI) of a dedicated application.

[0978] System Operation Overview

[0979] Before starting a video call, users set their preferred language (primary language) and the other party's preferred language (secondary language). Once the call is started, the following process is performed each time the user speaks:

[0980] 1. Audio Capture

[0981] When a user starts speaking, the voice data is captured through the device's microphone and sent to the server in real time.

[0982] 2. Speech Recognition and Text Conversion

[0983] The server passes the received voice data to a voice recognition engine and converts it into text data. For example, if user A says "Hello, how are you?", this voice becomes the text data "Hello, how are you?"

[0984] 3. Text Translation

[0985] The converted text data is passed to a translation AI and translated into the specified second language. In the example above, the text "Hello, how are you?" is translated into English as "Hello, how are you?"

[0986] 4. Displaying translated text

[0987] The translated text data is sent to the user's device and the other party's device via the server and displayed on the video call screen.

[0988] Specific examples

[0989] For example:

[0990] Consider the case where user A selects Japanese and the other user, user B, selects English. When user A says, "Hello, how are you?", the speech is captured on the device and sent to the server. The server uses a speech recognition engine to convert this speech into text, "Hello, how are you?" This text is sent to a translation AI, which translates it into English as "Hello, how are you?" The translated text is sent to the devices of user A and user B via the server and displayed on the video call screen.

[0991] Similarly, when User B says, "I am fine, thank you," the voice data is sent to the server and converted into text by a speech recognition engine. This text is then translated into Japanese by a translation AI as "I am fine, thank you." The translated text is then sent via the server again to the devices of User A and User B and displayed. By repeating this process, users can communicate in real time across language barriers.

[0992] Prompt Sentence Examples

[0993] Here are some example prompts to input to a generative AI model:

[0994] "Let's develop a video chat system with highly accurate speech recognition and real-time translation capabilities. Please tell us the specific steps involved. Please explain the roles of the user device, speech recognition engine, translation AI, server, and display means, and provide a detailed description of the processing steps, including setup and connection."

[0995] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0996] Step 1: Select your language

[0997] User action: The user selects their own language (first language) and the other person's language (second language) on the system settings screen.

[0998] Input: User-driven language selection through the interface.

[0999] Output: The system saves the primary and secondary language settings.

[1000] Step 2: Start a video call

[1001] User action: The user presses the start video call button.

[1002] Server operation: The server sends a connection request to the other device and establishes a video call connection.

[1003] Input: A request to start a video call initiated by the user.

[1004] Output: The server sends a connection request to the other user's device, and a video call connection is established.

[1005] Step 3: Capture audio

[1006] Device operation: When the user starts speaking, the device captures voice data in real time through the microphone.

[1007] Input: Audio capture via audio input device (microphone).

[1008] Output: The captured audio data is saved as internal data.

[1009] Step 4: Sending audio data to the server

[1010] Device operation: Captured audio data is sent to the server in real time.

[1011] Input: Audio capture data.

[1012] Output: The audio data is sent to the server.

[1013] Step 5: Speech recognition and text conversion

[1014] Server operation: The server passes the received voice data to a speech recognition engine and converts it into text data, for example, using Google Cloud Speech-to-Text.

[1015] Input: The audio data received by the server.

[1016] Output: Data converted into text by the speech recognition engine.

[1017] Step 6: Translate the text

[1018] Server operation: The converted text data is passed to a translation AI, which translates it into the specified second language, for example, using DeepL or the Google Cloud Translation API.

[1019] Input: Text data converted by speech recognition.

[1020] Output: Text data translated into a second language by the translation AI.

[1021] Step 7: Send and view translation results

[1022] Server operation: Translated text data is sent to the user and the other party's device.

[1023] Device operation: The device displays the received translated text data on the video call screen.

[1024] Input: The translated text data sent by the server.

[1025] Output: The translated text data is displayed on the user's and the recipient's devices.

[1026] As a concrete example, when user A says "Hello, how are you?", the speech is captured on the device and sent to the server. The server uses a speech recognition engine to convert this speech into text "Hello, how are you?", which is then passed to a translation AI that translates it into English as "Hello, how are you?" The translated text is sent to the devices of user A and user B and displayed on the video call screen.

[1027] (Application example 1)

[1028] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1029] In stores, it is difficult for foreign tourists and store staff who speak different languages ​​to communicate smoothly. Current tools and translation functions struggle to translate accurately in real time, resulting in frequent interruptions in conversations. This leads to a decline in the quality of customer service and impacts customer satisfaction. Therefore, there is a need for an effective communication support system with real-time translation functionality that can be used in stores.

[1030] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1031] In this invention, the server includes means for setting a first language and a second language for selection by a user, means for capturing voice data from a first terminal, means for converting the captured voice data into text data, means for translating the converted text data into the selected second language, means for displaying the translated text data, means for being installed in smart glasses or a smartphone and displaying the translated text in real time, and means for enabling smooth communication with customers in a store using the smart glasses or the smartphone, thereby enabling store clerks and foreign tourists to communicate in real time without worrying about language differences.

[1032] "Primary language" refers to the original language spoken by the user.

[1033] "Second language" refers to the language the user wishes to understand or the language the other person speaks.

[1034] A "user terminal" is a device used by a user, such as smart glasses or a smartphone.

[1035] "Voice data" refers to digital data of the voice spoken by the user.

[1036] "Text data" refers to character data obtained by converting voice data.

[1037] "Translation tools" refers to artificial intelligence technologies and software used to convert text data into different languages.

[1038] "Server" refers to a central system that processes and manages data.

[1039] "Display means" refers to the functionality for displaying text or information on devices such as smart glasses or smartphones.

[1040] "Real-time translation" refers to the process of providing translation results almost instantly after speech is spoken.

[1041] "Smart glasses" refers to a glasses-type device that displays information using AR technology.

[1042] A "smartphone" refers to a mobile phone with advanced computing and connectivity capabilities.

[1043] "Customer" refers to a shopper who visits a store or a person who uses a service.

[1044] "Store clerk" refers to an employee who deals with customers in a store.

[1045] "Translated text" refers to the result of translating text written in a source language into another language.

[1046] The system for implementing this invention is a real-time translation system for smooth communication between store clerks and foreign tourists. This system includes a user terminal, a voice recognition engine, a translation AI, and a server.

[1047] Program processing overview

[1048] The server performs a series of processes: capturing voice data from the user's device, converting it into text data, translating it into the selected second language, and displaying the translated text data. Specifically, via an application installed on smart glasses or a smartphone, the voice data is converted into text data using a voice recognition engine (e.g., Google Speech-to-Text API), and the text data is translated using a translation AI (e.g., DeepL API). The server manages this entire process and displays the translation results on the user's device in real time.

[1049] Hardware and software used

[1050] Hardware

[1051] Smart glasses (e.g., glasses-type devices using AR technology)

[1052] Smartphone

[1053] software

[1054] Speech recognition engine (e.g. Google Speech-to-Text API)

[1055] Translation AI (e.g. DeepL API)

[1056] Cloud server (e.g. AWS or Google Cloud)

[1057] Data processing and calculation flow

[1058] 1. Language Selection

[1059] The store clerk selects a foreign language that corresponds to his or her own language on the user terminal.

[1060] 2. Audio Capture

[1061] When a customer speaks into the microphone on their smart glasses or smartphone, voice data is captured.

[1062] 3. Voice Recognition

[1063] The captured voice data is sent to a cloud server and converted into text data using the Google Speech-to-Text API.

[1064] 4. Text Translation

[1065] The obtained text data is translated into the store clerk's language using the DeepL API.

[1066] 5. Displaying the translation results

[1067] The translated text is displayed in real time on smart glasses or smartphones, allowing store clerks to instantly understand the customer's questions and what they are saying.

[1068] Specific examples

[1069] For example, consider a case where a customer speaks Japanese and a store clerk speaks English. When the customer asks, "Where is this product?", the voice is captured by the microphone on the smart glasses or smartphone. The captured voice data is sent to a server and converted into text data, "Where is this product located?", using the Google Speech-to-Text API. This text data is then translated into English, "Where is this product located?", using the DeepL API, and the result is instantly displayed on the display of the smart glasses or smartphone. The store clerk can then look at the English translation and provide the customer with appropriate guidance.

[1070] Prompt Sentence Examples

[1071] Test the performance of your application by inputting prompt sentences like the following into your generative AI model:

[1072] Generate the following HTML code:

[1073] 1. JavaScript code to capture customer utterances.

[1074] 2. Send the captured audio data to the Google Speech-to-Text API.

[1075] 3. The resulting text is passed to the DeepL API for translation.

[1076] 4. HTML and CSS for displaying the translation results on the screen in real time.

[1077] Below are the details of the APIs we will use:

[1078] Google Speech-to-Text API: https: / / cloud.google.com / speech-to-text

[1079] DeepL API: https: / / www.deepl.com / pro-api

[1080] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1081] Step 1:

[1082] Language selection

[1083] The user selects their own language (first language) and the target foreign language (second language) on the device (smart glasses or smartphone).

[1084] Input: The primary and secondary languages ​​selected by the user.

[1085] Data processing: The selected language information is sent from the terminal to the server, and the initial connection settings are made.

[1086] Output: Language setting information stored on the server.

[1087] Step 2:

[1088] Audio Capture

[1089] When a customer speaks into the microphone on their smart glasses or smartphone, the voice data is captured by the device.

[1090] Input: Customer utterance (audio data).

[1091] Data processing: Voice data is collected in real time by the device and sent to the server.

[1092] Output: The audio data sent to the server.

[1093] Step 3:

[1094] Voice Recognition

[1095] The server receives the transmitted voice data and converts it into text data using a speech recognition engine (e.g., Google Speech-to-Text API).

[1096] Input: Audio data sent to the server.

[1097] Data calculation: The process of converting voice to text using a voice recognition engine.

[1098] Output: Text data (first language).

[1099] Step 4:

[1100] Text Translation

[1101] The server passes the text data to a translation AI (e.g., DeepL API) to translate it into the selected second language.

[1102] Input: Text data (primary language) generated by the speech recognition engine.

[1103] Data calculation: Translation processing of text data using translation AI.

[1104] Output: Translated text data (second language).

[1105] Step 5:

[1106] Displaying translation results

[1107] The server sends the translated text data to the user's device, where it is displayed on smart glasses or a smartphone.

[1108] Input: Translated text data (second language).

[1109] Data processing: Formatting text data and sending it to the user's device.

[1110] Output: Translated text displayed on smart glasses or smartphone.

[1111] Specific operation example

[1112] If a user selects English as their first language and Japanese as their second language, that information is sent to the server. When a customer speaks a question in Japanese, the voice data is captured by the microphone in the smart glasses and sent to the server. The server uses the Google Speech-to-Text API to convert the voice data into text data such as "Where is this product located?" This text data is then translated into English as "Where is this product located?" using the DeepL API and finally displayed on the smart glasses. The store clerk can view this translation result and immediately respond to the customer.

[1113] Prompt Sentence Examples

[1114] Generate the following HTML code:

[1115] 1. JavaScript code to capture customer utterances.

[1116] 2. Send the captured audio data to the Google Speech-to-Text API.

[1117] 3. The resulting text is passed to the DeepL API for translation.

[1118] 4. HTML and CSS for displaying the translation results on the screen in real time.

[1119] Below are the details of the APIs we will use:

[1120] Google Speech-to-Text API: https: / / cloud.google.com / speech-to-text

[1121] DeepL API: https: / / www.deepl.com / pro-api

[1122] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1123] The system based on this invention not only provides a video call service that translates languages ​​in real time, but also has the ability to recognize and display the user's emotions. Specifically, in addition to speech recognition technology and translation AI, it uses an emotion engine to recognize the user's emotions when speaking and displays that information on the video call screen, thereby achieving richer communication. The specific operating form of the system and its components are described below.

[1124] System Components

[1125] 1. User terminal: The device that a user uses to make a video call (e.g., smartphone, tablet, computer, etc.).

[1126] 2. Speech recognition engine: Software or hardware that converts a user's voice data into text data.

[1127] 3. Translation AI: Artificial intelligence technology for translating text data from one language to another.

[1128] 4. Emotion Engine: Technology for recognizing user emotions based on voice and text data.

[1129] 5. Server: The central device that controls voice data capture, text conversion, translation, emotion recognition, and sending and receiving the results.

[1130] 6. Display means: Technology and devices for displaying translated text data and emotion data on the video call screen.

[1131] System Operation Overview

[1132] 1. Language selection: First, the user selects their own language (first language) and the other person's language (second language) on the device's settings screen. The selected language information is sent to the server.

[1133] 2. Starting a video call: The user presses a button to start a video call, which sends a connection request to the server. The server then sends a request to the other device to establish the call.

[1134] 3. Voice capture: When the user starts speaking, the device captures voice data through the microphone and transmits it to the server in real time.

[1135] 4. Speech recognition and text conversion: The server inputs the received voice data into a speech recognition engine and converts it into text data.

[1136] 5. Text translation: The converted text data is passed from the server to a translation AI, which translates it into a second language.

[1137] 6. Emotion Recognition: The emotion engine recognizes the user's emotions based on the voice and text data.

[1138] 7. Displaying translated text and emotion data: The server sends the translated text data and the recognized emotion data to the user terminal and the other terminal, and displays them on the video call screen.

[1139] Specific examples

[1140] For example, consider the case where user A selects Japanese and the other user, user B, selects English. When user A says, "Hello, how are you?", the speech is captured on the device and sent to the server. The server uses a speech recognition engine to convert this speech into text, "Hello, how are you?" This text is sent to the translation AI, which translates it into English as "Hello, how are you?" Furthermore, this speech data and text data undergo emotion recognition by an emotion engine, which recognizes the emotion "friendly." The translated text and emotion data are sent to the devices of user A and user B via the server and displayed on the video call screen.

[1141] Similarly, when User B says, "I am fine, thank you," the voice data is also sent to the server and converted into text by the speech recognition engine. This text is then translated into Japanese by the translation AI as "I am fine, thank you." In this process, the emotion engine recognizes the emotion "happy." The translated text and emotion data are then sent again via the server to the devices of User A and User B and displayed.

[1142] This process allows users to communicate in real time, with greater depth and depth of emotion. By conveying emotions as well as translation, this system can build deeper understanding and bonds.

[1143] The processing flow will be explained below.

[1144] Program processing flow details

[1145] Step 1:

[1146] The user launches the application on their device and transitions to the main screen, at which point the account information is loaded.

[1147] Step 2:

[1148] The user accesses the language selection screen and sets their own language (first language) and the other party's language (second language). The terminal then sends the selected language information to the server.

[1149] Step 3:

[1150] To start a video call, the user presses a video call start button, and the terminal sends the request to the server.

[1151] Step 4:

[1152] The server receives the user's request and sends a connection request to the other party's terminal. If the other party accepts the request, the call is established.

[1153] Step 5:

[1154] When a user starts speaking, the device captures the voice data through the microphone, and the captured voice data is sent to the server in real time.

[1155] Step 6:

[1156] The server inputs the received voice data into a voice recognition engine and converts it into text data.

[1157] Step 7:

[1158] The server passes the converted text data to a translation AI, which translates it into a second language.

[1159] Step 8:

[1160] The server inputs the translated text data into an emotion engine to recognize the user's emotion.

[1161] Step 9:

[1162] The server transmits the translated text data and the recognized emotion data to the user terminal and the other terminal.

[1163] Step 10:

[1164] The device displays the received translated text and emotion data on the video call screen.

[1165] Step 11:

[1166] When the other person begins to speak, the other person's device also captures the voice data through the microphone and sends it to the server.

[1167] Step 12:

[1168] The server inputs the other party's voice data into a voice recognition engine and converts it into text data.

[1169] Step 13:

[1170] The server passes the converted text data to a translation AI, which translates it into the first language.

[1171] Step 14:

[1172] The server inputs the translated text data into an emotion engine to recognize the other person's emotions.

[1173] Step 15:

[1174] The server transmits the translated text data and the recognized emotion data to the user terminal and the other terminal.

[1175] Step 16:

[1176] The user terminal displays the received translated text and emotion data on the video call screen.

[1177] This process flow allows users to communicate with others in real time, overcoming language barriers and including emotions. Both users and others can receive information in language and emotions that they can instantly understand, resulting in smooth dialogue and deep understanding.

[1178] Example 2

[1179] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1180] Conventional real-time translation systems can convert voice data into text data and translate it, but they lack a means to convey user emotions. This makes it difficult for users to understand each other's emotions, which can lead to a decline in the quality of communication. For example, not being able to read emotions can lead to misunderstandings and miscommunication.

[1181] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for setting a first language and a second language for the user to select, means for capturing voice data from the first terminal, means for converting the captured voice data into text data, means for translating the converted text data into the selected second language, means for displaying the translated text data, means for recognizing the user's emotion based on the voice data and text data, and means for displaying the recognized emotion data on the video call screen. This allows the user to translate and convey emotions in real time, thereby achieving richer communication.

[1182] "Users" are people who use this system to make video calls.

[1183] "Primary language" refers to the user's preferred language.

[1184] A "secondary language" is another language that is used in a video call with a user-selected party.

[1185] A "terminal" is a device (e.g., a smartphone, tablet, computer) that a user uses to make a video call.

[1186] A "server" is a central device that controls voice data capture, text conversion, translation, emotion recognition, and result transmission and reception.

[1187] "Voice data" refers to digital voice information obtained by capturing a user's speech through a microphone.

[1188] "Text data" refers to character information converted from voice data using voice recognition technology.

[1189] "Translation" is the process of converting text data written in one language into text data written in another language.

[1190] "Emotion recognition" refers to a technology that analyzes and identifies a user's emotions based on voice and text data.

[1191] A "video call screen" refers to a display screen that displays video, audio, and supplemental information between users during a video call.

[1192] The system based on this invention has a function to recognize and display the user's emotions in addition to a video call service that translates languages ​​in real time. The main components of this system include a user terminal, a voice recognition engine, a translation AI, an emotion engine, a server, and a display means.

[1193] Hardware and software used

[1194] User device:

[1195] A user terminal is a device used by a user to make video calls, and can be a smartphone, tablet, computer, etc. This terminal has a built-in microphone and speaker, and has a video call application installed.

[1196] Speech Recognition Engine:

[1197] The speech recognition engine is used to convert the user's voice data into text data. Specifically, speech recognition technology such as Google Cloud Speech-to-Text is used.

[1198] Translation AI:

[1199] Artificial intelligence technology is used to translate the converted text data from one language to another, such as the Google Translate API.

[1200] Emotion Engine:

[1201] This technology is used to recognize user emotions based on voice and text data. Specifically, IBM Watson Tone Analyzer is used.

[1202] server:

[1203] The server is a central device that controls voice capture, text conversion, translation, emotion recognition, and sending and receiving the results. Typically, a cloud server (e.g., an AWS EC2 instance) is used.

[1204] Display means:

[1205] The display means is software or a device for displaying the translated text data and emotion data on the video call screen. A display built into the user terminal is used.

[1206] Specific examples

[1207] For example, consider the case where user A selects Japanese as their first language and the other user, user B, selects English as their second language. When user A says, "Hello, how are you?", the speech is captured on the device and sent to the server. The server uses a speech recognition engine to convert this speech into text, "Hello, how are you?" This text is sent to a translation AI, which translates it into English as "Hello, how are you?" This voice data and text data are then analyzed by an emotion engine, which recognizes the emotion "friendly." The translated text and emotion data are sent to the devices of user A and user B via the server and displayed on the video call screen.

[1208] Similarly, when User B says, "I am fine, thank you," the voice data is also sent to the server and converted into text by the speech recognition engine. This text is then translated into Japanese by the translation AI as "I am fine, thank you." In this process, the emotion engine recognizes the emotion "happy." The translated text and emotion data are then sent again via the server to the devices of User A and User B and displayed.

[1209] Prompt Sentence Examples

[1210] "Create a program that converts voice data into text in real time, translates that text into a specified language, recognizes emotions, and displays both translations and emotion data. The technologies used are a speech recognition engine, a translation AI, and an emotion engine."

[1211] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1212] Step 1: Select your language

[1213] The user selects their own language (first language) and the other person's language (second language) on the settings screen of a video calling app.

[1214] Input: The primary and secondary languages ​​selected by the user.

[1215] Output: The selected language information is sent to the server and stored.

[1216] Specific operation: User A selects Japanese as the first language and User B selects English as the second language. The device sends this information to the server, which stores it in a database.

[1217] Step 2: Start a video call

[1218] When a user presses the start button for a video call, the terminal sends a connection request to the server.

[1219] Input: Video call initiation request.

[1220] Output: The server sends a connection request to the other user's device and the video call is established.

[1221] Specific operation: When User A presses the "Start call" button, the device sends a connection request to the server. The server then sends a connection request to User B's device to establish a video call.

[1222] Step 3: Capture and send audio

[1223] When the user starts speaking, the device's microphone captures the voice data and transmits it to the server in real time.

[1224] Input: User speech.

[1225] Output: The captured audio data is sent to the server.

[1226] Specific operation: When user A says "Hello, how are you?", the device's microphone captures the voice and sends it to the server.

[1227] Step 4: Speech recognition and text conversion

[1228] The server inputs the received voice data into a voice recognition engine and converts it into text data.

[1229] Input: Captured audio data.

[1230] Output: The audio data is converted to text data.

[1231] Specific behavior: The server sends the audio data to Google Cloud Speech-to-Text, and the text "Hello, how are you?" is obtained.

[1232] Step 5: Translate the text

[1233] The server sends the text data obtained through voice recognition to a translation AI, which translates it into the specified second language.

[1234] Input: Text data obtained by speech recognition.

[1235] Output: The text data is translated into the second language.

[1236] Specific operation: The server sends text data to the Google Translate API, and "Hello, how are you?" is translated into "Hello, how are you?"

[1237] Step 6: Recognize emotions

[1238] The server inputs the voice data and text data into an emotion engine to recognize the user's emotion.

[1239] Input: Audio data and text data.

[1240] Output: Emotion data.

[1241] How it works: The server sends voice and text data to IBM Watson Tone Analyzer, which recognizes the emotion "friendly."

[1242] Step 7: Send and display translated text and emotion data

[1243] The server sends the translated text and recognized emotion data to both user terminals and displays them on the video call screen.

[1244] Input: translated text data and recognized emotion data.

[1245] Output: Display of translated text and emotion data on the device.

[1246] Specific operation: The server sends the translated text "Hello, how are you?" and the emotion data "friendly" to the devices of user A and user B, and displays them on their respective video call screens.

[1247] (Application example 2)

[1248] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1249] While systems already exist to enable smooth communication between users who speak different languages, recognizing emotions and responding quickly based on them is crucial, especially in security services where emergency response is required. However, existing systems lack the ability to recognize and display emotions, making communication in emergencies difficult. There is also a need for technology that can display emotions in real time on a video call screen while supporting multiple languages.

[1250] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1251] In this invention, the server includes means for setting a first language and a second language for selection by the user, means for capturing voice data from the first terminal, means for converting the captured voice data into text data, means for translating the converted text data into the selected second language, means for recognizing the user's emotion based on the voice data and text data, and means for displaying the recognized emotion data on the video call screen. This enables smooth communication between users who speak different languages, and makes it possible to quickly recognize the user's emotion and respond based on it, particularly in security service situations where emergency response is required.

[1252] The "means for setting a first language and a second language for the user to select" is a mechanism for the user making a video call to select the language to use and register that information in the system.

[1253] The "means for capturing voice data from the first terminal" is a function for collecting speech as voice data using a device (smartphone, tablet, computer, etc.) used by a user.

[1254] The "means for converting captured voice data into text data" is a technique for converting collected voice data into text data using voice recognition technology.

[1255] "Means for translating the converted text data into a selected second language" refers to a translation technique for converting the converted text data in a first language into another selected language.

[1256] The "means for recognizing a user's emotions based on voice data and text data" is a technology for analyzing collected voice and text data and determining the user's emotions at the time of speech.

[1257] The "means for displaying recognized emotion data on the video call screen" is a function for displaying the emotion determined by the emotion engine in real time on the terminal screen of the user making the video call.

[1258] The system based on this invention aims to improve emergency calls in security services, and is specifically designed to realize real-time translation between users who speak different languages ​​and emotion recognition.

[1259] System Components

[1260] 1. User Device

[1261] A device that allows users to make video calls. Typical examples include smartphones, tablets, and computers. These devices can capture audio and transmit it to a server in real time.

[1262] 2. Speech Recognition Engine

[1263] This is a technology that converts captured voice data into text data. A representative example of this software is "SpeechRecognition."

[1264] 3. Translation AI

[1265] It is a technology that translates text data from one language to another. By using "Google Trans," real-time translation between multiple languages ​​is possible.

[1266] 4. Emotion Engine

[1267] This is a technology for recognizing user emotions based on voice and text data. "EmotionRecognizer" is a representative example.

[1268] 5. Video Call System

[1269] This is a system for managing video calls between users. It includes OpenCV for video processing and a custom module for managing video calls.

[1270] 6. Server

[1271] It is the central device that controls voice data capture, text conversion, translation, emotion recognition, and sending and receiving the results.

[1272] System Operation Overview

[1273] 1. Language Selection

[1274] Users select their own language (first language) and the other person's language (second language) on the device settings screen of their smartphone or other device.

[1275] 2. Start a video call

[1276] The user presses a button to start a video call, which sends a connection request to the server, which then sends a request to the other device, and the video call is established.

[1277] 3. Audio Capture

[1278] When the user starts speaking, the device captures voice data through the microphone and transmits it to the server in real time.

[1279] 4. Speech Recognition and Text Conversion

[1280] The server inputs the received voice data into a voice recognition engine and converts it into text data.

[1281] 5. Text Translation

[1282] The converted text data is passed from the server to a translation AI and translated into a second language.

[1283] 6. Emotional Recognition

[1284] The emotion engine recognizes the user's emotion based on the voice data and text data.

[1285] 7. Displaying translated text and sentiment data

[1286] The server transmits the translated text data and the recognized emotion data to the user terminal and the other terminal, and displays them on the video call screen.

[1287] The specific hardware and software used

[1288] Hardware

[1289] Smartphones, tablets, and computers

[1290] software

[1291] OpenCV: Image processing for video calls

[1292] SpeechRecognition: Speech recognition

[1293] GoogleTrans: Real-time translation

[1294] EmotionRecognizer: Emotion Recognition from Audio Data

[1295] Custom module for video chat system: Call management and communication with the server

[1296] Specific examples

[1297] For example, if a security staff member on-site says, "We've spotted a suspicious person. What should we do?", this data is instantly captured and converted into text data by a speech recognition engine. The translation AI then translates this text and displays it in English. At the same time, the emotion engine recognizes this as "tension," and the control center immediately begins emergency response.

[1298] Prompt Sentence Examples

[1299] "Translate emergency calls in Japanese from security staff into English in real time, and recognize and display the emotions expressed. The emotion engine recognizes emotions from voice and text data, and displays those emotions on the video call screen."

[1300] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1301] Step 1:

[1302] The user selects a primary and secondary language on the device's settings screen. The user's selected language information is sent from the device to the server, allowing the server to identify the language pair for translation.

[1303] Step 2:

[1304] To start a video call, a user presses a video call start button on the terminal, and the terminal sends this request to the server, which then sends a call start request to the other terminal and establishes the call.

[1305] Step 3:

[1306] When a user starts speaking, voice data is captured through the device's microphone. This voice data is sent to the server in real time. The server then converts the input voice data into text data using a voice recognition engine.

[1307] Step 4:

[1308] The server passes the converted text data to a translation AI, which translates it into the selected second language. The input is the converted text data, and the output is the translated text data. This is done by the translation AI (Googletrans).

[1309] Step 5:

[1310] Using the emotion engine, the server recognizes the user's emotion based on the voice and text data. The input is the voice and text data, and the output is the recognized emotion data. This is done by EmotionRecognizer.

[1311] Step 6:

[1312] The translated text data and the recognized emotion data are sent from the server to the user's device and the other device. The device displays this data on the screen. As an output, the text translated into the other user's language and the emotion display are reflected on the video call screen in real time.

[1313] Step 7:

[1314] The user checks the translated text and emotion data displayed on the screen. This process allows users to communicate in real time with people who speak different languages ​​and progress through the dialogue while understanding the other person's emotions.

[1315] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1316] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1317] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1318] [Fourth embodiment]

[1319] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1320] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1321] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1322] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1323] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1324] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1325] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1326] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1327] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1328] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1329] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1330] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1331] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1332] The system based on this invention is a video calling service that allows users to smoothly communicate with people who speak different languages. Specifically, it has the function of translating languages ​​in real time using voice recognition technology and translation AI and displaying the translation content. The specific operation form of the system and its components are described below.

[1333] System Components

[1334] 1. User terminal: The device that a user uses to make a video call (e.g., smartphone, tablet, computer, etc.).

[1335] 2. Speech recognition engine: Software or hardware that converts a user's voice data into text data.

[1336] 3. Translation AI: Artificial intelligence technology for translating text data from one language to another.

[1337] 4. Server: The central device that controls the capture of voice data, text conversion, translation, and sending and receiving of the results.

[1338] 5. Display means: Technology or equipment for displaying translated text data on the video call screen.

[1339] System Operation Overview

[1340] 1. Language selection: First, the user selects their own language (first language) and the other person's language (second language) on the device's settings screen.

[1341] 2. Starting a video call: The user presses a button to start a video call, which sends a connection request to the server, which then sends this request to the other device and establishes the call.

[1342] 3. Voice capture: When the user starts speaking, the device captures voice data through the microphone and transmits it to the server in real time.

[1343] 4. Speech recognition and text conversion: The server passes the received voice data to a speech recognition engine and converts it into text data.

[1344] 5. Text translation: The converted text data is passed from the server to a translation AI, where it is translated into a second language.

[1345] 6. Displaying translated text: The translated text data is sent to the user's and the other party's devices via the server and displayed on the video call screen.

[1346] Specific examples

[1347] For example, consider the case where user A selects Japanese and the other user, user B, selects English. When user A says, "Hello, how are you?", the speech is captured on the device and sent to the server. The server uses a speech recognition engine to convert this speech into text, "Hello, how are you?" This text is sent to a translation AI, which translates it into English as "Hello, how are you?" The translated text is sent to the devices of user A and user B via the server and displayed on the video call screen.

[1348] Similarly, when User B says "I am fine, thank you," the voice data is sent to the server and converted into text by a speech recognition engine. This text is then translated into Japanese by a translation AI as "I am fine, thank you." The translated text is then sent via the server again to the devices of User A and User B and displayed.

[1349] By repeating this process, users and their contacts can communicate in real time, overcoming language barriers. This system is designed to improve usability and enable smooth two-way communication.

[1350] The processing flow will be explained below.

[1351] Program processing flow details

[1352] Step 1:

[1353] The user launches the application on their device and transitions to the main screen, at which point the account information is loaded.

[1354] Step 2:

[1355] The user accesses the language selection screen and sets their own language (first language) and the other party's language (second language). The terminal then sends the selected language information to the server.

[1356] Step 3:

[1357] To start a video call, the user presses a video call start button, and the terminal sends the request to the server.

[1358] Step 4:

[1359] The server receives the user's request and sends a connection request to the other party's terminal. If the other party accepts the request, the call is established.

[1360] Step 5:

[1361] When a user starts speaking, the device captures the voice data through the microphone, and the captured voice data is sent to the server in real time.

[1362] Step 6:

[1363] The server inputs the received voice data into a voice recognition engine and converts it into text data.

[1364] Step 7:

[1365] The server passes the converted text data to a translation AI, which translates it into a second language.

[1366] Step 8:

[1367] The server transmits the translated text data to the user terminal and the other terminal.

[1368] Step 9:

[1369] The terminal displays the received translated text on the video call screen.

[1370] Step 10:

[1371] When the other person begins to speak, the other person's device also captures the voice data through the microphone and sends it to the server.

[1372] Step 11:

[1373] The server inputs the other party's voice data into a voice recognition engine and converts it into text data.

[1374] Step 12:

[1375] The server passes the converted text data to a translation AI, which translates it into the first language.

[1376] Step 13:

[1377] The server transmits the translated text data to the user terminal and the other terminal.

[1378] Step 14:

[1379] The user terminal displays the received translated text on the video call screen.

[1380] This process flow enables users to communicate with others in real time, overcoming language barriers. Both users and others can receive information in a language they can instantly understand, ensuring smooth dialogue.

[1381] Example 1

[1382] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1383] It is difficult for users who speak different languages ​​to communicate smoothly. Real-time interpretation is particularly required during video calls, but achieving this is technically complex and expensive. Furthermore, existing translation systems have problems such as low translation accuracy and a lack of real-time capabilities, resulting in a poor user experience. The present invention aims to solve these problems by providing a system that enables users who speak different languages ​​to communicate smoothly in real time.

[1384] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1385] In this invention, the server includes means for setting a first language and a second language for selection by the user, means for capturing voice data from the first terminal, means for converting the captured voice data into text data, means for translating the converted text data into the selected second language, means for displaying the translated text data, means for sending a connection request and establishing a video call, means for transmitting and receiving voice data via the server, means for converting the voice data into text data using a voice recognition engine, means for translating the text data into a specified language in real time using a translation AI, and means for displaying the translated text data on the video call screen, thereby enabling users to smoothly communicate in real time with parties who speak different languages.

[1386] "User" means a person or end user who makes a video call using the System.

[1387] "Primary language" refers to the language that a user selects within the system to use.

[1388] "Second language" refers to the language used by the person selected by the user within the system.

[1389] A "terminal" is a device for making video calls, and includes a smartphone, tablet, computer, etc.

[1390] "Voice data" refers to a digital recording of a user's speech.

[1391] "Text data" is voice data converted into character information.

[1392] A "voice recognition engine" is software or hardware that converts voice data into text data.

[1393] "Translation AI" is an artificial intelligence technology that translates text data into other languages.

[1394] The "server" is a central device that manages the processing of the entire system, and transmits and receives voice data, converts text, translates, and transmits the results.

[1395] The "video call screen" is a part of the user interface for displaying translated text data during a video call.

[1396] A "connection request" is a request signal sent from a server to start a video call.

[1397] "Capture" refers to the act of acquiring audio data using an input device such as a terminal's microphone.

[1398] The system based on this invention is a video calling service that enables users who speak different languages ​​to communicate smoothly. This system uses speech recognition technology and translation AI to translate languages ​​in real time and display the translation results. The detailed configuration and operation of the system are described below.

[1399] System Components

[1400] 1. User Device

[1401] This is a device that allows users to make video calls, such as a smartphone, tablet, or computer. These devices must have a microphone for voice input and a screen to display the translation results.

[1402] 2. Speech Recognition Engine

[1403] Software or hardware that converts a user's voice data into text data. Examples include Google Cloud Speech-to-Text and Amazon Transcribe.

[1404] 3. Translation AI

[1405] AI technology that translates text data from one language to another. Applications include DeepL, Google Cloud Translation API, and Microsoft Translator.

[1406] 4. Server

[1407] A central device that controls the capture of voice data, text conversion, translation, and sending and receiving of the results. Examples include AWS EC2 and Google Cloud Compute Engine.

[1408] 5. Display means

[1409] This refers to the technology and equipment used to display translated text data on the video call screen, either on the screen of the user's device or on the user interface (UI) of a dedicated application.

[1410] System Operation Overview

[1411] Before starting a video call, users set their preferred language (primary language) and the other party's preferred language (secondary language). Once the call is started, the following process is performed each time the user speaks:

[1412] 1. Audio Capture

[1413] When a user starts speaking, the voice data is captured through the device's microphone and sent to the server in real time.

[1414] 2. Speech Recognition and Text Conversion

[1415] The server passes the received voice data to a voice recognition engine and converts it into text data. For example, if user A says "Hello, how are you?", this voice becomes the text data "Hello, how are you?"

[1416] 3. Text Translation

[1417] The converted text data is passed to a translation AI and translated into the specified second language. In the example above, the text "Hello, how are you?" is translated into English as "Hello, how are you?"

[1418] 4. Displaying translated text

[1419] The translated text data is sent to the user's device and the other party's device via the server and displayed on the video call screen.

[1420] Specific examples

[1421] For example:

[1422] Consider the case where user A selects Japanese and the other user, user B, selects English. When user A says, "Hello, how are you?", the speech is captured on the device and sent to the server. The server uses a speech recognition engine to convert this speech into text, "Hello, how are you?" This text is sent to a translation AI, which translates it into English as "Hello, how are you?" The translated text is sent to the devices of user A and user B via the server and displayed on the video call screen.

[1423] Similarly, when User B says, "I am fine, thank you," the voice data is sent to the server and converted into text by a speech recognition engine. This text is then translated into Japanese by a translation AI as "I am fine, thank you." The translated text is then sent via the server again to the devices of User A and User B and displayed. By repeating this process, users can communicate in real time across language barriers.

[1424] Prompt Sentence Examples

[1425] Here are some example prompts to input to a generative AI model:

[1426] "Let's develop a video chat system with highly accurate speech recognition and real-time translation capabilities. Please tell us the specific steps involved. Please explain the roles of the user device, speech recognition engine, translation AI, server, and display means, and provide a detailed description of the processing steps, including setup and connection."

[1427] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1428] Step 1: Select your language

[1429] User action: The user selects their own language (first language) and the other person's language (second language) on the system settings screen.

[1430] Input: User-driven language selection through the interface.

[1431] Output: The system saves the primary and secondary language settings.

[1432] Step 2: Start a video call

[1433] User action: The user presses the start video call button.

[1434] Server operation: The server sends a connection request to the other device and establishes a video call connection.

[1435] Input: A request to start a video call initiated by the user.

[1436] Output: The server sends a connection request to the other user's device, and a video call connection is established.

[1437] Step 3: Capture audio

[1438] Device operation: When the user starts speaking, the device captures voice data in real time through the microphone.

[1439] Input: Audio capture via audio input device (microphone).

[1440] Output: The captured audio data is saved as internal data.

[1441] Step 4: Sending audio data to the server

[1442] Device operation: Captured audio data is sent to the server in real time.

[1443] Input: Audio capture data.

[1444] Output: The audio data is sent to the server.

[1445] Step 5: Speech recognition and text conversion

[1446] Server operation: The server passes the received voice data to a speech recognition engine and converts it into text data, for example, using Google Cloud Speech-to-Text.

[1447] Input: The audio data received by the server.

[1448] Output: Data converted into text by the speech recognition engine.

[1449] Step 6: Translate the text

[1450] Server operation: The converted text data is passed to a translation AI, which translates it into the specified second language, for example, using DeepL or the Google Cloud Translation API.

[1451] Input: Text data converted by speech recognition.

[1452] Output: Text data translated into a second language by the translation AI.

[1453] Step 7: Send and view translation results

[1454] Server operation: Translated text data is sent to the user and the other party's device.

[1455] Device operation: The device displays the received translated text data on the video call screen.

[1456] Input: The translated text data sent by the server.

[1457] Output: The translated text data is displayed on the user's and the recipient's devices.

[1458] As a concrete example, when user A says "Hello, how are you?", the speech is captured on the device and sent to the server. The server uses a speech recognition engine to convert this speech into text "Hello, how are you?", which is then passed to a translation AI that translates it into English as "Hello, how are you?" The translated text is sent to the devices of user A and user B and displayed on the video call screen.

[1459] (Application example 1)

[1460] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1461] In stores, it is difficult for foreign tourists and store staff who speak different languages ​​to communicate smoothly. Current tools and translation functions struggle to translate accurately in real time, resulting in frequent interruptions in conversations. This leads to a decline in the quality of customer service and impacts customer satisfaction. Therefore, there is a need for an effective communication support system with real-time translation functionality that can be used in stores.

[1462] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1463] In this invention, the server includes means for setting a first language and a second language for selection by a user, means for capturing voice data from a first terminal, means for converting the captured voice data into text data, means for translating the converted text data into the selected second language, means for displaying the translated text data, means for being installed in smart glasses or a smartphone and displaying the translated text in real time, and means for enabling smooth communication with customers in a store using the smart glasses or the smartphone, thereby enabling store clerks and foreign tourists to communicate in real time without worrying about language differences.

[1464] "Primary language" refers to the original language spoken by the user.

[1465] "Second language" refers to the language the user wishes to understand or the language the other person speaks.

[1466] A "user terminal" is a device used by a user, such as smart glasses or a smartphone.

[1467] "Voice data" refers to digital data of the voice spoken by the user.

[1468] "Text data" refers to character data obtained by converting voice data.

[1469] "Translation tools" refers to artificial intelligence technologies and software used to convert text data into different languages.

[1470] "Server" refers to a central system that processes and manages data.

[1471] "Display means" refers to the functionality for displaying text or information on devices such as smart glasses or smartphones.

[1472] "Real-time translation" refers to the process of providing translation results almost instantly after speech is spoken.

[1473] "Smart glasses" refers to a glasses-type device that displays information using AR technology.

[1474] A "smartphone" refers to a mobile phone with advanced computing and connectivity capabilities.

[1475] "Customer" refers to a shopper who visits a store or a person who uses a service.

[1476] "Store clerk" refers to an employee who deals with customers in a store.

[1477] "Translated text" refers to the result of translating text written in a source language into another language.

[1478] The system for implementing this invention is a real-time translation system for smooth communication between store clerks and foreign tourists. This system includes a user terminal, a voice recognition engine, a translation AI, and a server.

[1479] Program processing overview

[1480] The server performs a series of processes: capturing voice data from the user's device, converting it into text data, translating it into the selected second language, and displaying the translated text data. Specifically, via an application installed on smart glasses or a smartphone, the voice data is converted into text data using a voice recognition engine (e.g., Google Speech-to-Text API), and the text data is translated using a translation AI (e.g., DeepL API). The server manages this entire process and displays the translation results on the user's device in real time.

[1481] Hardware and software used

[1482] Hardware

[1483] Smart glasses (e.g., glasses-type devices using AR technology)

[1484] Smartphone

[1485] software

[1486] Speech recognition engine (e.g. Google Speech-to-Text API)

[1487] Translation AI (e.g. DeepL API)

[1488] Cloud server (e.g. AWS or Google Cloud)

[1489] Data processing and calculation flow

[1490] 1. Language Selection

[1491] The store clerk selects a foreign language that corresponds to his or her own language on the user terminal.

[1492] 2. Audio Capture

[1493] When a customer speaks into the microphone on their smart glasses or smartphone, voice data is captured.

[1494] 3. Voice Recognition

[1495] The captured voice data is sent to a cloud server and converted into text data using the Google Speech-to-Text API.

[1496] 4. Text Translation

[1497] The obtained text data is translated into the store clerk's language using the DeepL API.

[1498] 5. Displaying the translation results

[1499] The translated text is displayed in real time on smart glasses or smartphones, allowing store clerks to instantly understand the customer's questions and what they are saying.

[1500] Specific examples

[1501] For example, consider a case where a customer speaks Japanese and a store clerk speaks English. When the customer asks, "Where is this product?", the voice is captured by the microphone on the smart glasses or smartphone. The captured voice data is sent to a server and converted into text data, "Where is this product located?", using the Google Speech-to-Text API. This text data is then translated into English, "Where is this product located?", using the DeepL API, and the result is instantly displayed on the display of the smart glasses or smartphone. The store clerk can then look at the English translation and provide the customer with appropriate guidance.

[1502] Prompt Sentence Examples

[1503] Test the performance of your application by inputting prompt sentences like the following into your generative AI model:

[1504] Generate the following HTML code:

[1505] 1. JavaScript code to capture customer utterances.

[1506] 2. Send the captured audio data to the Google Speech-to-Text API.

[1507] 3. The resulting text is passed to the DeepL API for translation.

[1508] 4. HTML and CSS for displaying the translation results on the screen in real time.

[1509] Below are the details of the APIs we will use:

[1510] Google Speech-to-Text API: https: / / cloud.google.com / speech-to-text

[1511] DeepL API: https: / / www.deepl.com / pro-api

[1512] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1513] Step 1:

[1514] Language selection

[1515] The user selects their own language (first language) and the target foreign language (second language) on the device (smart glasses or smartphone).

[1516] Input: The primary and secondary languages ​​selected by the user.

[1517] Data processing: The selected language information is sent from the terminal to the server, and the initial connection settings are made.

[1518] Output: Language setting information stored on the server.

[1519] Step 2:

[1520] Audio Capture

[1521] When a customer speaks into the microphone on their smart glasses or smartphone, the voice data is captured by the device.

[1522] Input: Customer utterance (audio data).

[1523] Data processing: Voice data is collected in real time by the device and sent to the server.

[1524] Output: The audio data sent to the server.

[1525] Step 3:

[1526] Voice Recognition

[1527] The server receives the transmitted voice data and converts it into text data using a speech recognition engine (e.g., Google Speech-to-Text API).

[1528] Input: Audio data sent to the server.

[1529] Data calculation: The process of converting voice to text using a voice recognition engine.

[1530] Output: Text data (first language).

[1531] Step 4:

[1532] Text Translation

[1533] The server passes the text data to a translation AI (e.g., DeepL API) to translate it into the selected second language.

[1534] Input: Text data (primary language) generated by the speech recognition engine.

[1535] Data calculation: Translation processing of text data using translation AI.

[1536] Output: Translated text data (second language).

[1537] Step 5:

[1538] Displaying translation results

[1539] The server sends the translated text data to the user's device, where it is displayed on smart glasses or a smartphone.

[1540] Input: Translated text data (second language).

[1541] Data processing: Formatting text data and sending it to the user's device.

[1542] Output: Translated text displayed on smart glasses or smartphone.

[1543] Specific operation example

[1544] If a user selects English as their first language and Japanese as their second language, that information is sent to the server. When a customer speaks a question in Japanese, the voice data is captured by the microphone in the smart glasses and sent to the server. The server uses the Google Speech-to-Text API to convert the voice data into text data such as "Where is this product located?" This text data is then translated into English as "Where is this product located?" using the DeepL API and finally displayed on the smart glasses. The store clerk can view this translation result and immediately respond to the customer.

[1545] Prompt Sentence Examples

[1546] Generate the following HTML code:

[1547] 1. JavaScript code to capture customer utterances.

[1548] 2. Send the captured audio data to the Google Speech-to-Text API.

[1549] 3. The resulting text is passed to the DeepL API for translation.

[1550] 4. HTML and CSS for displaying the translation results on the screen in real time.

[1551] Below are the details of the APIs we will use:

[1552] Google Speech-to-Text API: https: / / cloud.google.com / speech-to-text

[1553] DeepL API: https: / / www.deepl.com / pro-api

[1554] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1555] The system based on this invention not only provides a video call service that translates languages ​​in real time, but also has the ability to recognize and display the user's emotions. Specifically, in addition to speech recognition technology and translation AI, it uses an emotion engine to recognize the user's emotions when speaking and displays that information on the video call screen, thereby achieving richer communication. The specific operating form of the system and its components are described below.

[1556] System Components

[1557] 1. User terminal: The device that a user uses to make a video call (e.g., smartphone, tablet, computer, etc.).

[1558] 2. Speech recognition engine: Software or hardware that converts a user's voice data into text data.

[1559] 3. Translation AI: Artificial intelligence technology for translating text data from one language to another.

[1560] 4. Emotion Engine: Technology for recognizing user emotions based on voice and text data.

[1561] 5. Server: The central device that controls voice data capture, text conversion, translation, emotion recognition, and sending and receiving the results.

[1562] 6. Display means: Technology and devices for displaying translated text data and emotion data on the video call screen.

[1563] System Operation Overview

[1564] 1. Language selection: First, the user selects their own language (first language) and the other person's language (second language) on the device's settings screen. The selected language information is sent to the server.

[1565] 2. Starting a video call: The user presses a button to start a video call, which sends a connection request to the server. The server then sends a request to the other device to establish the call.

[1566] 3. Voice capture: When the user starts speaking, the device captures voice data through the microphone and transmits it to the server in real time.

[1567] 4. Speech recognition and text conversion: The server inputs the received voice data into a speech recognition engine and converts it into text data.

[1568] 5. Text translation: The converted text data is passed from the server to a translation AI, which translates it into a second language.

[1569] 6. Emotion Recognition: The emotion engine recognizes the user's emotions based on the voice and text data.

[1570] 7. Displaying translated text and emotion data: The server sends the translated text data and the recognized emotion data to the user terminal and the other terminal, and displays them on the video call screen.

[1571] Specific examples

[1572] For example, consider the case where user A selects Japanese and the other user, user B, selects English. When user A says, "Hello, how are you?", the speech is captured on the device and sent to the server. The server uses a speech recognition engine to convert this speech into text, "Hello, how are you?" This text is sent to the translation AI, which translates it into English as "Hello, how are you?" Furthermore, this speech data and text data undergo emotion recognition by an emotion engine, which recognizes the emotion "friendly." The translated text and emotion data are sent to the devices of user A and user B via the server and displayed on the video call screen.

[1573] Similarly, when User B says, "I am fine, thank you," the voice data is also sent to the server and converted into text by the speech recognition engine. This text is then translated into Japanese by the translation AI as "I am fine, thank you." In this process, the emotion engine recognizes the emotion "happy." The translated text and emotion data are then sent again via the server to the devices of User A and User B and displayed.

[1574] This process allows users to communicate in real time, with greater depth and depth of emotion. By conveying emotions as well as translation, this system can build deeper understanding and bonds.

[1575] The processing flow will be explained below.

[1576] Program processing flow details

[1577] Step 1:

[1578] The user launches the application on their device and transitions to the main screen, at which point the account information is loaded.

[1579] Step 2:

[1580] The user accesses the language selection screen and sets their own language (first language) and the other party's language (second language). The terminal then sends the selected language information to the server.

[1581] Step 3:

[1582] To start a video call, the user presses a video call start button, and the terminal sends the request to the server.

[1583] Step 4:

[1584] The server receives the user's request and sends a connection request to the other party's terminal. If the other party accepts the request, the call is established.

[1585] Step 5:

[1586] When a user starts speaking, the device captures the voice data through the microphone, and the captured voice data is sent to the server in real time.

[1587] Step 6:

[1588] The server inputs the received voice data into a voice recognition engine and converts it into text data.

[1589] Step 7:

[1590] The server passes the converted text data to a translation AI, which translates it into a second language.

[1591] Step 8:

[1592] The server inputs the translated text data into an emotion engine to recognize the user's emotion.

[1593] Step 9:

[1594] The server transmits the translated text data and the recognized emotion data to the user terminal and the other terminal.

[1595] Step 10:

[1596] The device displays the received translated text and emotion data on the video call screen.

[1597] Step 11:

[1598] When the other person begins to speak, the other person's device also captures the voice data through the microphone and sends it to the server.

[1599] Step 12:

[1600] The server inputs the other party's voice data into a voice recognition engine and converts it into text data.

[1601] Step 13:

[1602] The server passes the converted text data to a translation AI, which translates it into the first language.

[1603] Step 14:

[1604] The server inputs the translated text data into an emotion engine to recognize the other person's emotions.

[1605] Step 15:

[1606] The server transmits the translated text data and the recognized emotion data to the user terminal and the other terminal.

[1607] Step 16:

[1608] The user terminal displays the received translated text and emotion data on the video call screen.

[1609] This process flow allows users to communicate with others in real time, overcoming language barriers and including emotions. Both users and others can receive information in language and emotions that they can instantly understand, resulting in smooth dialogue and deep understanding.

[1610] Example 2

[1611] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1612] Conventional real-time translation systems can convert voice data into text data and translate it, but they lack a means to convey user emotions. This makes it difficult for users to understand each other's emotions, which can lead to a decline in the quality of communication. For example, not being able to read emotions can lead to misunderstandings and miscommunication.

[1613] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for setting a first language and a second language for the user to select, means for capturing voice data from the first terminal, means for converting the captured voice data into text data, means for translating the converted text data into the selected second language, means for displaying the translated text data, means for recognizing the user's emotion based on the voice data and text data, and means for displaying the recognized emotion data on the video call screen. This allows the user to translate and convey emotions in real time, thereby achieving richer communication.

[1614] "Users" are people who use this system to make video calls.

[1615] "Primary language" refers to the user's preferred language.

[1616] A "secondary language" is another language that is used in a video call with a user-selected party.

[1617] A "terminal" is a device (e.g., a smartphone, tablet, computer) that a user uses to make a video call.

[1618] A "server" is a central device that controls voice data capture, text conversion, translation, emotion recognition, and result transmission and reception.

[1619] "Voice data" refers to digital voice information obtained by capturing a user's speech through a microphone.

[1620] "Text data" refers to character information converted from voice data using voice recognition technology.

[1621] "Translation" is the process of converting text data written in one language into text data written in another language.

[1622] "Emotion recognition" refers to a technology that analyzes and identifies a user's emotions based on voice and text data.

[1623] A "video call screen" refers to a display screen that displays video, audio, and supplemental information between users during a video call.

[1624] The system based on this invention has a function to recognize and display the user's emotions in addition to a video call service that translates languages ​​in real time. The main components of this system include a user terminal, a voice recognition engine, a translation AI, an emotion engine, a server, and a display means.

[1625] Hardware and software used

[1626] User device:

[1627] A user terminal is a device used by a user to make video calls, and can be a smartphone, tablet, computer, etc. This terminal has a built-in microphone and speaker, and has a video call application installed.

[1628] Speech Recognition Engine:

[1629] The speech recognition engine is used to convert the user's voice data into text data. Specifically, speech recognition technology such as Google Cloud Speech-to-Text is used.

[1630] Translation AI:

[1631] Artificial intelligence technology is used to translate the converted text data from one language to another, such as the Google Translate API.

[1632] Emotion Engine:

[1633] This technology is used to recognize user emotions based on voice and text data. Specifically, IBM Watson Tone Analyzer is used.

[1634] server:

[1635] The server is a central device that controls voice capture, text conversion, translation, emotion recognition, and sending and receiving the results. Typically, a cloud server (e.g., an AWS EC2 instance) is used.

[1636] Display means:

[1637] The display means is software or a device for displaying the translated text data and emotion data on the video call screen. A display built into the user terminal is used.

[1638] Specific examples

[1639] For example, consider the case where user A selects Japanese as their first language and the other user, user B, selects English as their second language. When user A says, "Hello, how are you?", the speech is captured on the device and sent to the server. The server uses a speech recognition engine to convert this speech into text, "Hello, how are you?" This text is sent to a translation AI, which translates it into English as "Hello, how are you?" This voice data and text data are then analyzed by an emotion engine, which recognizes the emotion "friendly." The translated text and emotion data are sent to the devices of user A and user B via the server and displayed on the video call screen.

[1640] Similarly, when User B says, "I am fine, thank you," the voice data is also sent to the server and converted into text by the speech recognition engine. This text is then translated into Japanese by the translation AI as "I am fine, thank you." In this process, the emotion engine recognizes the emotion "happy." The translated text and emotion data are then sent again via the server to the devices of User A and User B and displayed.

[1641] Prompt Sentence Examples

[1642] "Create a program that converts voice data into text in real time, translates that text into a specified language, recognizes emotions, and displays both translations and emotion data. The technologies used are a speech recognition engine, a translation AI, and an emotion engine."

[1643] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1644] Step 1: Select your language

[1645] The user selects their own language (first language) and the other person's language (second language) on the settings screen of a video calling app.

[1646] Input: The primary and secondary languages ​​selected by the user.

[1647] Output: The selected language information is sent to the server and stored.

[1648] Specific operation: User A selects Japanese as the first language and User B selects English as the second language. The device sends this information to the server, which stores it in a database.

[1649] Step 2: Start a video call

[1650] When a user presses the start button for a video call, the terminal sends a connection request to the server.

[1651] Input: Video call initiation request.

[1652] Output: The server sends a connection request to the other user's device and the video call is established.

[1653] Specific operation: When User A presses the "Start call" button, the device sends a connection request to the server. The server then sends a connection request to User B's device to establish a video call.

[1654] Step 3: Capture and send audio

[1655] When the user starts speaking, the device's microphone captures the voice data and transmits it to the server in real time.

[1656] Input: User speech.

[1657] Output: The captured audio data is sent to the server.

[1658] Specific operation: When user A says "Hello, how are you?", the device's microphone captures the voice and sends it to the server.

[1659] Step 4: Speech recognition and text conversion

[1660] The server inputs the received voice data into a voice recognition engine and converts it into text data.

[1661] Input: Captured audio data.

[1662] Output: The audio data is converted to text data.

[1663] Specific behavior: The server sends the audio data to Google Cloud Speech-to-Text, and the text "Hello, how are you?" is obtained.

[1664] Step 5: Translate the text

[1665] The server sends the text data obtained through voice recognition to a translation AI, which translates it into the specified second language.

[1666] Input: Text data obtained by speech recognition.

[1667] Output: The text data is translated into the second language.

[1668] Specific operation: The server sends text data to the Google Translate API, and "Hello, how are you?" is translated into "Hello, how are you?"

[1669] Step 6: Recognize emotions

[1670] The server inputs the voice data and text data into an emotion engine to recognize the user's emotion.

[1671] Input: Audio data and text data.

[1672] Output: Emotion data.

[1673] How it works: The server sends voice and text data to IBM Watson Tone Analyzer, which recognizes the emotion "friendly."

[1674] Step 7: Send and display translated text and emotion data

[1675] The server sends the translated text and recognized emotion data to both user terminals and displays them on the video call screen.

[1676] Input: translated text data and recognized emotion data.

[1677] Output: Display of translated text and emotion data on the device.

[1678] Specific operation: The server sends the translated text "Hello, how are you?" and the emotion data "friendly" to the devices of user A and user B, and displays them on their respective video call screens.

[1679] (Application example 2)

[1680] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1681] While systems already exist to enable smooth communication between users who speak different languages, recognizing emotions and responding quickly based on them is crucial, especially in security services where emergency response is required. However, existing systems lack the ability to recognize and display emotions, making communication in emergencies difficult. There is also a need for technology that can display emotions in real time on a video call screen while supporting multiple languages.

[1682] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1683] In this invention, the server includes means for setting a first language and a second language for selection by the user, means for capturing voice data from the first terminal, means for converting the captured voice data into text data, means for translating the converted text data into the selected second language, means for recognizing the user's emotion based on the voice data and text data, and means for displaying the recognized emotion data on the video call screen. This enables smooth communication between users who speak different languages, and makes it possible to quickly recognize the user's emotion and respond based on it, particularly in security service situations where emergency response is required.

[1684] The "means for setting a first language and a second language for the user to select" is a mechanism for the user making a video call to select the language to use and register that information in the system.

[1685] The "means for capturing voice data from the first terminal" is a function for collecting speech as voice data using a device (smartphone, tablet, computer, etc.) used by a user.

[1686] The "means for converting captured voice data into text data" is a technique for converting collected voice data into text data using voice recognition technology.

[1687] "Means for translating the converted text data into a selected second language" refers to a translation technique for converting the converted text data in a first language into another selected language.

[1688] The "means for recognizing a user's emotions based on voice data and text data" is a technology for analyzing collected voice and text data and determining the user's emotions at the time of speech.

[1689] The "means for displaying recognized emotion data on the video call screen" is a function for displaying the emotion determined by the emotion engine in real time on the terminal screen of the user making the video call.

[1690] The system based on this invention aims to improve emergency calls in security services, and is specifically designed to realize real-time translation between users who speak different languages ​​and emotion recognition.

[1691] System Components

[1692] 1. User Device

[1693] A device that allows users to make video calls. Typical examples include smartphones, tablets, and computers. These devices can capture audio and transmit it to a server in real time.

[1694] 2. Speech Recognition Engine

[1695] This is a technology that converts captured voice data into text data. A representative example of this software is "SpeechRecognition."

[1696] 3. Translation AI

[1697] It is a technology that translates text data from one language to another. By using "Google Trans," real-time translation between multiple languages ​​is possible.

[1698] 4. Emotion Engine

[1699] This is a technology for recognizing user emotions based on voice and text data. "EmotionRecognizer" is a representative example.

[1700] 5. Video Call System

[1701] This is a system for managing video calls between users. It includes OpenCV for video processing and a custom module for managing video calls.

[1702] 6. Server

[1703] It is the central device that controls voice data capture, text conversion, translation, emotion recognition, and sending and receiving the results.

[1704] System Operation Overview

[1705] 1. Language Selection

[1706] Users select their own language (first language) and the other person's language (second language) on the device settings screen of their smartphone or other device.

[1707] 2. Start a video call

[1708] The user presses a button to start a video call, which sends a connection request to the server, which then sends a request to the other device, and the video call is established.

[1709] 3. Audio Capture

[1710] When the user starts speaking, the device captures voice data through the microphone and transmits it to the server in real time.

[1711] 4. Speech Recognition and Text Conversion

[1712] The server inputs the received voice data into a voice recognition engine and converts it into text data.

[1713] 5. Text Translation

[1714] The converted text data is passed from the server to a translation AI and translated into a second language.

[1715] 6. Emotional Recognition

[1716] The emotion engine recognizes the user's emotion based on the voice data and text data.

[1717] 7. Displaying translated text and sentiment data

[1718] The server transmits the translated text data and the recognized emotion data to the user terminal and the other terminal, and displays them on the video call screen.

[1719] The specific hardware and software used

[1720] Hardware

[1721] Smartphones, tablets, and computers

[1722] software

[1723] OpenCV: Image processing for video calls

[1724] SpeechRecognition: Speech recognition

[1725] GoogleTrans: Real-time translation

[1726] EmotionRecognizer: Emotion Recognition from Audio Data

[1727] Custom module for video chat system: Call management and communication with the server

[1728] Specific examples

[1729] For example, if a security staff member on-site says, "We've spotted a suspicious person. What should we do?", this data is instantly captured and converted into text data by a speech recognition engine. The translation AI then translates this text and displays it in English. At the same time, the emotion engine recognizes this as "tension," and the control center immediately begins emergency response.

[1730] Prompt Sentence Examples

[1731] "Translate emergency calls in Japanese from security staff into English in real time, and recognize and display the emotions expressed. The emotion engine recognizes emotions from voice and text data, and displays those emotions on the video call screen."

[1732] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1733] Step 1:

[1734] The user selects a primary and secondary language on the device's settings screen. The user's selected language information is sent from the device to the server, allowing the server to identify the language pair for translation.

[1735] Step 2:

[1736] To start a video call, a user presses a video call start button on the terminal, and the terminal sends this request to the server, which then sends a call start request to the other terminal and establishes the call.

[1737] Step 3:

[1738] When a user starts speaking, voice data is captured through the device's microphone. This voice data is sent to the server in real time. The server then converts the input voice data into text data using a voice recognition engine.

[1739] Step 4:

[1740] The server passes the converted text data to a translation AI, which translates it into the selected second language. The input is the converted text data, and the output is the translated text data. This is done by the translation AI (Googletrans).

[1741] Step 5:

[1742] Using the emotion engine, the server recognizes the user's emotion based on the voice and text data. The input is the voice and text data, and the output is the recognized emotion data. This is done by EmotionRecognizer.

[1743] Step 6:

[1744] The translated text data and the recognized emotion data are sent from the server to the user's device and the other device. The device displays this data on the screen. As an output, the text translated into the other user's language and the emotion display are reflected on the video call screen in real time.

[1745] Step 7:

[1746] The user checks the translated text and emotion data displayed on the screen. This process allows users to communicate in real time with people who speak different languages ​​and progress through the dialogue while understanding the other person's emotions.

[1747] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1748] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1749] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1750] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1751] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1752] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1753] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1754] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1755] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1756] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1757] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1758] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1759] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1760] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1761] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1762] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1763] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1764] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1765] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1766] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1767] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1768] The following is further disclosed regarding the above embodiment.

[1769] (Claim 1)

[1770] means for setting a first language and a second language for selection by a user;

[1771] means for capturing audio data from the first terminal;

[1772] means for converting the captured audio data into text data;

[1773] means for translating the converted text data into a selected second language;

[1774] a means for displaying the translated text data;

[1775] A system including:

[1776] (Claim 2)

[1777] 10. The system of claim 1, which provides real-time translation of speech data from a first language to a second language.

[1778] (Claim 3)

[1779] 10. The system of claim 1, further comprising: capturing voice data from the second terminal; translating the captured voice data into the first language and displaying the translated voice data.

[1780] "Example 1"

[1781] (Claim 1)

[1782] means for setting a first language and a second language for selection by a user;

[1783] means for capturing audio data from the first terminal;

[1784] means for converting the captured audio data into text data;

[1785] means for translating the converted text data into a selected second language;

[1786] a means for displaying the translated text data;

[1787] means for sending a connection request and establishing a video call;

[1788] means for transmitting and receiving voice data via a server;

[1789] means for converting voice data into text data using a voice recognition engine;

[1790] A means to translate text data into a specified language using translation AI in real time;

[1791] a means for displaying the translated text data on the video call screen;

[1792] A system including:

[1793] (Claim 2)

[1794] 10. The system of claim 1, which provides real-time translation of speech data from a first language to a second language.

[1795] (Claim 3)

[1796] 10. The system of claim 1, further comprising: capturing voice data from the second terminal; translating the captured voice data into the first language and displaying the translated voice data.

[1797] "Application Example 1"

[1798] (Claim 1)

[1799] means for setting a first language and a second language for selection by a user;

[1800] means for capturing audio data from the first terminal;

[1801] means for converting the captured audio data into text data;

[1802] means for translating the converted text data into a selected second language;

[1803] a means for displaying the translated text data;

[1804] A means to display real-time translated text on smart glasses or smartphones;

[1805] A means for enabling smooth communication with customers in stores using smart glasses or smartphones;

[1806] A system including:

[1807] (Claim 2)

[1808] 10. The system of claim 1, which provides real-time translation of speech data from a first language to a second language.

[1809] (Claim 3)

[1810] 10. The system of claim 1, further comprising: capturing voice data from a second terminal; translating the captured voice data into a first language; and displaying the translated voice data using smart glasses or a smartphone.

[1811] "Example 2: Combining Emotion Engines"

[1812] (Claim 1)

[1813] means for setting a first language and a second language for selection by a user;

[1814] means for capturing audio data from the first terminal;

[1815] means for converting the captured audio data into text data;

[1816] means for translating the converted text data into a selected second language;

[1817] a means for displaying the translated text data;

[1818] means for recognizing a user's emotion based on voice data and text data;

[1819] a means for displaying the recognized emotion data on a video call screen;

[1820] A system including:

[1821] (Claim 2)

[1822] 10. The system of claim 1, which provides real-time translation of speech data from a first language to a second language.

[1823] (Claim 3)

[1824] 10. The system of claim 1, further comprising: capturing voice data from the second terminal; translating the captured voice data into the first language and displaying the translated voice data.

[1825] "Application example 2 when combining emotion engines"

[1826] (Claim 1)

[1827] means for setting a first language and a second language for selection by a user;

[1828] means for capturing audio data from the first terminal;

[1829] means for converting the captured audio data into text data;

[1830] means for translating the converted text data into a selected second language;

[1831] a means for displaying the translated text data;

[1832] means for recognizing a user's emotion based on voice data and text data;

[1833] a means for displaying the recognized emotion data on a video call screen;

[1834] A system including:

[1835] (Claim 2)

[1836] 10. The system of claim 1, which provides real-time translation of speech data from a first language to a second language.

[1837] (Claim 3)

[1838] 10. The system of claim 1, further comprising: capturing voice data from the second terminal; translating the captured voice data into the first language and displaying the translated voice data. [Explanation of symbols]

[1839] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for setting a first language and a second language for selection by a user; means for capturing audio data from the first terminal; means for converting the captured audio data into text data; means for translating the converted text data into a selected second language; a means for displaying the translated text data; A system including:

2. 10. The system of claim 1, wherein the system provides real-time translation of speech data from a first language to a second language.

3. 10. The system of claim 1, further comprising: capturing voice data from the second terminal; translating the captured voice data into the first language and displaying the translated voice data.

Images