System

Abstract

To provide a system capable of automatically conversing even when language is not common.SOLUTION: A system includes: means for acquiring voice data; means for converting the voice data into text data; means for translating the text data into two or more languages different from the language of the text data using generative AI; and means for projecting the translated text data on a predetermined surface.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] It is difficult for people who speak different languages ​​to converse naturally in the same space due to language barriers, and existing translation systems rely on individual devices, making it difficult to share them across spaces. [Means for solving the problem]

[0005] Using generative AI's translation and summarization functions, we provide a system that enables conversations to take place in a designated space (room) regardless of the language spoken by people in that space. Speech is translated into multiple languages ​​and projected onto walls, enabling natural conversation even when the speakers do not share the same language. [Brief explanation of the drawings]

[0006] [Figure 1]1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment Example 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, coded random access memory (RAM) is a memory in which information is temporarily stored and is used as a work memory by a processor. In the following embodiments, coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (Solid State Drive (SSD)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0011] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0012] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0013] [First embodiment]

[0014] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0015] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0016] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0017] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0018] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0019] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0020] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0021] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0022] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0023] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0024] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0025] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0026] "Example 1"

[0027] One embodiment of the present invention provides a system with generative AI translation and summarization capabilities. This system is installed in a designated space (e.g., a conference room or living room) and translates conversations taking place within that space into multiple languages ​​in real time. The translation results are projected onto any surface, such as a wall or screen. This allows participants who do not share the same language to converse naturally.

[0028] "Example 2"

[0029] As a specific example of its use, consider an international conference. Even if participants speak different languages, this system can translate each participant's remarks into other languages ​​in real time and project them onto the wall. This allows all participants to understand the content of the conference and exchange opinions freely.

[0030] "Example 3"

[0031] This system is also suitable for use at home. For example, if a family member who speaks different languages ​​lives together, installing this system in the living room will facilitate smooth communication between family members. Family conversations are translated in real time and projected onto the wall, allowing for natural conversation without any sense of language barriers.

[0032] The processing flow of each embodiment will be described below.

[0033] "Example 1"

[0034] Step 1: A system with generative AI translation and summarization capabilities is installed in a designated space (e.g., a conference room or living room).

[0035] Step 2: Conversations taking place in the space are translated into multiple languages ​​in real time by the system.

[0036] Step 3: The translation result is projected onto any surface, such as a wall or screen.

[0037] Step 4: Even if participants do not share the same language, they can converse naturally by looking at the translation displayed on the screen.

[0038] "Example 2"

[0039] Step 1: At an international conference where participants speak different languages, the system is installed in the conference room.

[0040] Step 2: The system translates each participant's speech into other languages ​​in real time.

[0041] Step 3: The translation result is projected onto the wall.

[0042] Step 4: All participants can see the translation on the screen, understand the content of the meeting, and freely exchange opinions.

[0043] "Example 3"

[0044] Step 1: If your family members speak different languages, install the system in the living room or other area.

[0045] Step 2: The system translates the family's conversation in real time.

[0046] Step 3: The translation result is projected onto the wall.

[0047] Step 4: By having all family members see the translation displayed on screen, they can have natural conversations without feeling any language barrier.

[0048] Example 1

[0049] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0050] It is difficult for participants who speak different languages ​​to converse naturally in the same space. In particular, language barriers can be an obstacle in meetings or at home, making it difficult to share information smoothly. To solve this problem, real-time multilingual translation and summarization are required.

[0051] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0052] In this invention, the server includes means for capturing speech in real time, means for converting speech data into text data, means for translating and summarizing the text data into multiple languages ​​using a generative AI model, and means for displaying the translation results, thereby enabling participants who speak different languages ​​to converse naturally in the same space.

[0053] "Real-time audio capture means" refers to devices or technologies that instantly collect audio occurring within a designated space and convert it into a digital signal.

[0054] A "means for converting voice data into text data" is software or algorithms that analyzes collected voice signals and converts them into a corresponding text format.

[0055] A "generative AI model" is an advanced artificial intelligence model for natural language processing, capable of translating and summarizing text data.

[0056] "Means for translating and summarizing text data into multiple languages" means techniques and processes that use generative AI models to translate and, where necessary, summarize input text data into multiple specified languages.

[0057] "Means for displaying the translation results" refers to devices or technologies for visually displaying the translated text data, including projectors and displays.

[0058] "Means that enable natural conversation even when the language is not shared" refers to technologies and systems that enable participants who speak different languages ​​to understand conversation content that is translated in real time and communicate smoothly.

[0059] MODE FOR CARRYING OUT THE INVENTION

[0060] This invention is a system that enables participants who speak different languages ​​to converse naturally in the same space. The system includes a series of processes that capture speech in real time, convert the speech data into text data, translate and summarize the text data into multiple languages ​​using a generative AI model, and display the translation results.

[0061] Hardware and software used

[0062] Hardware:

[0063] Microphone: Highly sensitive microphones are used to capture sounds occurring within a designated space in real time.

[0064] Server: Provides the computational resources to process audio data, run generative AI models, and manage translation results.

[0065] Projector or display: Used to project the translation results onto a wall or screen.

[0066] software:

[0067] Speech recognition software (e.g., Google® Speech-to-Text API): converts voice data into text data.

[0068] Generative AI models (e.g., GPT-4 (registered trademark)): Translate text data into multiple languages ​​and summarize it as needed.

[0069] Translation software (e.g., Google Translate API): Used to supplement generative AI models.

[0070] Specific operation of the system

[0071] The server captures conversations in real time through microphones installed in the designated space. The captured audio data is converted into text using speech recognition software. A generative AI model is then used to translate the text into multiple languages ​​and summarize it as needed. The translated and summarized text is then projected onto a wall or screen via a projector or display.

[0072] Specific examples

[0073] For example, if there are participants who speak Japanese and English in a conference room, the system will operate as follows:

[0074] 1. The server captures speech in Japanese through a microphone installed in the conference room.

[0075] 2. The captured voice data is converted into text data using voice recognition software.

[0076] 3. The server uses a generative AI model to translate the Japanese text data into English.

[0077] 4. If necessary, the generative AI model summarizes the text data.

[0078] 5. The translated and summarized text data is displayed on a screen via a projector.

[0079] 6. The English translation will be displayed on the screen so that English-speaking participants can understand the content.

[0080] Prompt Sentence Examples

[0081] An example of a prompt sentence to input to the generative AI model is as follows:

[0082] "Please translate the conversation in the conference room in real time. Please translate and summarize the following Japanese text into English."

[0083] Using this prompt, the generative AI model performs the specified task, translating and summarizing the conversation.

[0084] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0085] Step 1:

[0086] Audio Capture

[0087] The server captures conversations in real time through microphones installed in the designated space. The input is the sound in the space, and the output is audio data converted into a digital signal. Specifically, the microphone collects the sound, converts it into a digital signal, and sends it to the server.

[0088] Step 2:

[0089] Voice Recognition

[0090] The server converts the received voice data into text data using speech recognition software (e.g., Google Speech-to-Text API). The input is voice data converted into digital signals, and the output is the corresponding text data. Specifically, the server inputs the voice data into the speech recognition software, which then analyzes the voice and generates text data.

[0091] Step 3:

[0092] Translation and summary

[0093] The server uses a generative AI model (e.g., GPT-4) to translate text data into multiple languages ​​and summarize it as needed. The input is text data obtained from speech recognition software, and the output is translated and summarized text data. Specifically, the server inputs text data into the generative AI model, and the model analyzes the text, translates it into the specified language, and summarizes it as needed.

[0094] Step 4:

[0095] display

[0096] The terminal (projector or display) projects the translation results sent from the server onto a wall or screen. The input is translated and summarized text data, and the output is a visually displayed translation result. Specifically, the server sends the translation result to the terminal, and the terminal displays the received data on the projector or display. The displayed translation result is projected in a location where it can be seen by participants.

[0097] (Application example 1)

[0098] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0099] There is a need to provide an environment where people who speak different languages ​​can converse naturally. In particular, in physical stores, it is important that communication between customers who speak a foreign language and store staff is smooth. However, currently, language barriers exist, making it difficult to provide smooth customer service. To solve this problem, a system with real-time translation functionality is needed.

[0100] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0101] In this invention, the server includes means for translating and summarizing using generative AI, means for displaying conversations translated into multiple languages ​​in a specified space (room), means for enabling natural conversation even when the language is not the same, and means for displaying the translation results on a smart device. This allows people who speak different languages ​​to view the translated conversation in real time through their smart devices and converse naturally.

[0102] "Generative AI" refers to artificial intelligence that generates new information based on data.

[0103] The "translation and summarization function" refers to the function that converts input language data into another language or summarizes its contents concisely.

[0104] A "designated space" refers to a specific location or area, such as a room or conference room.

[0105] "Multilingual" means two or more different languages.

[0106] "Means for displaying dialogue" refers to a device or method for visually displaying the translated dialogue.

[0107] "Means that enable natural conversation even when no common language is used" refers to techniques and methods that enable people who speak different languages ​​to communicate smoothly.

[0108] A "smart device" is an electronic device with internet connectivity and advanced computing power, including smartphones and smart glasses.

[0109] "Means for displaying the translation result" refers to a device or method for visually presenting the translated content to the user.

[0110] To implement the present invention, the following system configuration is required.

[0111] System Program

[0112] The system includes a program with generative AI translation and summarization functions. Specifically, it requires a program that integrates speech recognition, translation, and display functions.

[0113] Hardware and Software

[0114] This system uses the following hardware and software:

[0115] Hardware: Smart glasses (e.g., Google Glass®), microphone

[0116] Software: Python, Google Speech Recognition API, Google Translate API, OpenCV

[0117] Processing flow

[0118] 1. Speech recognition: The server converts the voice data acquired from the microphone into text data using the Google Speech Recognition API. Through this process, what the user said is captured as text.

[0119] 2. Translation: The server translates the acquired text data into the specified language using the Google Translate API. This process enables communication between users who speak different languages.

[0120] 3. Display: The server uses OpenCV to display the translated text on the smart glasses display, allowing the user to check the translation results in real time.

[0121] Specific examples

[0122] For example, if an English-speaking customer asks, "Where can I find the electronics section?", the server will recognize the speech and translate it into Japanese, displaying it on the smart glasses' display. This allows store staff to provide smooth service without having to worry about language barriers.

[0123] Prompt Sentence Examples

[0124] An example of a prompt sentence to input to the generative AI model is as follows:

[0125] Create a program that translates English speech into Japanese and displays it on the smart glasses display. Use the Google Speech Recognition API for speech recognition and the Google Translate API for translation. Use OpenCV for display.

[0126] In this way, the present invention provides an environment in which people who speak different languages ​​can converse naturally.

[0127] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0128] Step 1:

[0129] The server acquires voice data from the microphone. The user's spoken voice is input. This voice data is sent to the server and becomes input data for voice recognition processing.

[0130] Step 2:

[0131] The server converts the acquired voice data into text data using the Google Speech Recognition API. Voice data is input, and text data is output through a voice recognition algorithm. Through this process, what the user said is captured as text.

[0132] Step 3:

[0133] The server translates the converted text data into the specified language using the Google Translate API. Text data is input, and the translation algorithm outputs text data in the specified language. This process enables communication between users who speak different languages.

[0134] Step 4:

[0135] The server uses OpenCV to display the translated text data on the smart glasses display. The translated text data is input and displayed on the smart glasses display through a display algorithm. This process allows the user to check the translation results in real time.

[0136] Step 5:

[0137] The user checks the translation results displayed on the smart glasses display and continues the conversation. In this step, the user visually checks the translation results and uses them as input data to decide the next action.

[0138] Example 2

[0139] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0140] In situations such as international conferences, when participants speak different languages, language barriers can become an obstacle to communication. To solve this problem, real-time multilingual translation and display is required, but conventional technology has had difficulty in doing so efficiently and accurately.

[0141] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0142] In this invention, the server includes means for acquiring voice data, means for converting the voice data into text data, means for translating the text data into another language, and means for displaying the translated text data, thereby enabling participants who speak different languages ​​to communicate in real time.

[0143] "Voice data" refers to data in which a user's speech or voice is recorded in digital format.

[0144] "Text data" refers to data obtained by converting voice data into character information.

[0145] "Translation" is the process of converting text data written in one language into another language.

[0146] "Display" refers to the visual presentation of the translated text data.

[0147] "Speech recognition software" is software for converting voice data into text data.

[0148] A "generative AI model" is an artificial intelligence model for translating text data into other languages.

[0149] A "projector" is a device for projecting text data onto any surface.

[0150] This invention is a system that enables participants who speak different languages ​​to communicate in real time. The system performs a series of processes: acquiring voice data, converting it into text data, and then translating and displaying it in other languages.

[0151] First, the user speaks using a device (microphone or smartphone). The device captures this speech as voice data and sends it to a server. The server then uses voice recognition software (e.g., Google Cloud Speech-to-Text) to convert the received voice data into text data. Next, the server uses a generative AI model (e.g., OpenAI's GPT-4) to translate the text data into another language. Finally, the server projects the translated text data onto a wall using a projector or display.

[0152] As a concrete example, consider an international conference with participant A who speaks English and participant B who speaks Japanese. If participant A says "Hello, everyone. Welcome to the conference," this utterance is processed as follows:

[0153] 1. Participant A's device sends voice data to the server.

[0154] 2. The server uses Google Cloud Speech-to-Text to convert the speech into text data: "Hello, everyone. Welcome to the conference."

[0155] 3. The server uses a generative AI model (GPT-4) to translate this text data into Japanese: "Hello, everyone. Welcome to the meeting."

[0156] 4. The translated text data is projected onto the wall using a projector.

[0157] Example prompt sentence:

[0158] "Please translate the English text into Japanese: Hello, everyone. Welcome to the conference."

[0159] In this way, participant A's remarks are translated into Japanese in real time, allowing participant B to understand the content of the meeting. This system enables participants who speak different languages ​​to communicate smoothly.

[0160] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0161] Step 1:

[0162] A user speaks using a device (microphone or smartphone), which captures the speech as voice data and sends it to the server.

[0163] Input: User's speech (audio)

[0164] Output: Digital audio data

[0165] Specific operation: The user says, "Hello, everyone. Welcome to the conference." The device records this speech and sends it to the server as digital audio data.

[0166] Step 2:

[0167] The server converts the received voice data into text data using voice recognition software (e.g., Google Cloud Speech-to-Text).

[0168] Input: Digital audio data

[0169] Output: Text data

[0170] Specific operation: The server inputs the voice data into Google Cloud Speech-to-Text and obtains the text data "Hello, everyone. Welcome to the conference."

[0171] Step 3:

[0172] The server uses a generative AI model (e.g., OpenAI's GPT-4) to translate the text data into other languages.

[0173] Input: Text data

[0174] Output: Translated text data

[0175] Specific operation: The server inputs the text data "Hello, everyone. Welcome to the conference." into GPT-4 and obtains the Japanese translation result "Hello, everyone. Welcome to the conference."

[0176] Step 4:

[0177] The server then projects the translated text data onto a wall using a projector or display.

[0178] Input: Translated text data

[0179] Output: Text displayed on the wall

[0180] Specific operation: The server sends the text data "Hello everyone. Welcome to the meeting." to the projector and displays it on the wall.

[0181] In this way, the user's speech is translated into other languages ​​in real time, allowing other participants to understand the content of the conference.

[0182] (Application example 2)

[0183] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0184] The goal is to solve the problem of passengers who speak different languages ​​having difficulty communicating smoothly in self-driving vehicles, and to ensure that passengers can understand destination directions, emergency instructions, and tourist information in real time.

[0185] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means having a generative AI translation and summarization function, means for displaying conversations translated into multiple languages ​​in a specified space (room), means for enabling natural conversation even when the languages ​​are not the same, means for providing translated information in real time via an in-vehicle display device or mobile device when passengers in an autonomous vehicle speak different languages, and means for playing back the translated information by voice. This enables passengers who speak different languages ​​to communicate smoothly in an autonomous vehicle.

[0186] "Generative AI" is a technology that uses artificial intelligence to generate, translate, and summarize data.

[0187] The "translation and summary function" converts input language data into another language, extracts important information, and summarizes it concisely.

[0188] A "designated space" refers to a specific location or area, which in this case means a physical space such as a room or a car.

[0189] "Multiple languages" refers to two or more different languages, including, for example, Japanese, English, and French.

[0190] "Means for displaying the conversation" refers to a technology that visually displays the translated conversation content using a display device such as a monitor or projector.

[0191] "Means that enable natural conversation" are technologies that allow people who speak different languages ​​to communicate smoothly without feeling a language barrier.

[0192] An "autonomous vehicle" is a vehicle that drives autonomously using artificial intelligence and sensor technology.

[0193] A "display device" is a device for visually displaying information, and includes displays, monitors, and the like.

[0194] "Mobile device" refers to a portable electronic device such as a smartphone or tablet.

[0195] "Real-time translated information" means information in which speech or text is instantly converted into another language and provided without delay.

[0196] "Audio playback" refers to a technique that uses speech synthesis technology to output the translated text as audio.

[0197] The following system configuration will be described as an embodiment of the present invention.

[0198] System Configuration

[0199] The system includes a server with generative AI translation and summarization functions, a display device that projects conversations translated into multiple languages ​​in a designated space (room), an audio playback device that enables natural conversation even when the language is not the same, and a mobile device that provides real-time translated information when passengers in an autonomous vehicle speak different languages.

[0200] Program processing

[0201] The server runs a program that integrates speech recognition, translation, and speech synthesis functions. Specifically, it uses the following hardware and software:

[0202] Hardware: microphones, speakers, displays (in-car displays and smartphones)

[0203] Software: Python, SpeechRecognition library, Google Translate API, pyttsx3 library

[0204] Voice Recognition

[0205] The server converts the audio data acquired from the microphone into text data using the SpeechRecognition library. This text data represents what the passenger said.

[0206] translation

[0207] The server uses the Google Translate API to translate the acquired text data into the target language, for example, translating Japanese text into English.

[0208] Speech synthesis

[0209] The server uses the pyttsx3 library to convert the translated text data into audio data and play it through the speaker.

[0210] Specific examples

[0211] For example, if a Japanese-speaking passenger asks, "Where is the next stop?", the system will translate the question into English and display "Where is the next stop?" on the in-car display while simultaneously playing it aloud.

[0212] Prompt Sentence Examples

[0213] Translate the Japanese audio into English, display it on the screen, and play it back aloud.

[0214] In this way, passengers who speak different languages ​​will be able to communicate seamlessly inside an autonomous vehicle.

[0215] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0216] Step 1:

[0217] The user speaks into the microphone. The input is the user's voice data. The server receives this voice data and converts it into text data using the SpeechRecognition library. The output is text data representing the user's speech.

[0218] Step 2:

[0219] The server sends the text data obtained in step 1 to the Google Translate API and translates it into the specified target language. The input is the text data and the target language information. The output is the translated text data.

[0220] Step 3:

[0221] The server converts the translated text data into audio data using the pyttsx3 library. The input is the translated text data. The output is the audio data.

[0222] Step 4:

[0223] The server plays the audio data generated in step 3 through the speaker. The input is the audio data. The output is the audio that the user can hear.

[0224] Step 5:

[0225] The server sends the translated text data to an in-car display or mobile device for display on the display device. The input is the translated text data. The output is the translated text displayed on the display.

[0226] In this way, what the user says is translated in real time and provided as both audio and text.

[0227] Example 3

[0228] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0229] It is difficult for people who speak different languages ​​to converse naturally in the same space. Language barriers often become an obstacle to communication, especially at home or in small gatherings. To solve this problem, real-time speech recognition and translation, as well as a visual display of the translation results, are required.

[0230] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0231] In this invention, the server includes means for capturing voice input, means for transmitting the captured voice data to the server, means for converting the voice data into text, means for translating the text into a specified target language, means for transmitting the translated text to the terminal, and means for displaying the translated text, thereby enabling people who speak different languages ​​to communicate naturally while visually checking translated conversations in real time.

[0232] A "means for capturing voice input" is a device or function for detecting a user's speech and collecting the voice data.

[0233] The "means for transmitting captured voice data to a server" refers to a device or function for transmitting collected voice data to a server via a communication means such as the Internet.

[0234] "Means for converting voice data into text" refers to a device or function for analyzing voice data and converting the content into text format.

[0235] A "means for translating text into a specified target language" is a device or function for converting text data into a different language.

[0236] The "means for transmitting the translated text to the terminal" refers to a device or function for transmitting the translated text data to the terminal via a communication means such as the Internet.

[0237] The "means for displaying translated text" refers to a device or function for visually displaying translated text data.

[0238] This invention is a system that allows people who speak different languages ​​to converse naturally in the same space. The system includes a set of means for capturing voice input, translating it in real time, and visually displaying the results.

[0239] Hardware and software used

[0240] Hardware:

[0241] Device: Smart speaker or smart device with microphone

[0242] Display devices: projectors and smart displays

[0243] software:

[0244] Speech Recognition API: Google Cloud Speech-to-Text API

[0245] Translation API: Google Cloud Translation API

[0246] Data processing and calculation

[0247] Capturing audio input:

[0248] When a user speaks, the device captures the audio. For example, a smart speaker installed in the living room detects the user's speech.

[0249] Sending audio data:

[0250] The captured audio data is transmitted to a server over the Internet.

[0251] Voice Recognition:

[0252] The server converts the audio data into text using the Google Cloud Speech-to-Text API. For example, if a user says, "How was your day?", the API converts this to the text "How was your day?"

[0253] translation:

[0254] The server translates the converted text into the specified target language using the Google Cloud Translation API. For example, if a translation from English to Japanese is required, the API will translate "How was your day?" into "How was your day today?"

[0255] Sending translation results:

[0256] The translated text is sent to the device via the internet.

[0257] Displaying translation results:

[0258] The device then projects the translation results onto a wall or screen using a projector or smart display. For example, the text "How was your day?" could be displayed on a living room wall.

[0259] Specific examples

[0260] As a concrete example, consider a conversation between an English-speaking father and a Japanese-speaking mother in the living room.

[0261] 1. Capture audio input:

[0262] The father says, "How was your day?"

[0263] The device captures this audio and sends it to the server.

[0264] 2. Speech Recognition:

[0265] The server uses the Google Cloud Speech-to-Text API to convert the audio data into text: "How was your day?"

[0266] 3. Translation:

[0267] The server uses the Google Cloud Translation API to translate "How was your day?" to "How was your day?"

[0268] 4. Sending translation results:

[0269] The server sends the translated text "How was your day?" to the device.

[0270] 5. Displaying the translation results:

[0271] The device projects the text "How was your day?" onto the living room wall.

[0272] Example prompt sentence:

[0273] User: "How was your day?"

[0274] Server: "How was your day?"

[0275] This system allows smooth communication between family members who speak different languages ​​through real-time translation. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0276] Step 1:

[0277] Capture audio input

[0278] When a user speaks, the device captures the voice. For example, a smart speaker installed in a living room detects the user's utterance, "How was your day?" The input is the user's voice, and the output is the captured voice data.

[0279] Step 2:

[0280] Sending audio data

[0281] The terminal transmits the captured audio data to a server via the Internet. The input is the captured audio data, and the output is the audio data transmitted to the server.

[0282] Step 3:

[0283] Voice Recognition

[0284] The server converts the voice data into text using the Google Cloud Speech-to-Text API. For example, the voice data "How was your day?" is converted into text "How was your day?". The input is the voice data sent to the server, and the output is the converted text data.

[0285] Step 4:

[0286] translation

[0287] The server translates the converted text into the specified target language using the Google Cloud Translation API. For example, the text "How was your day?" is translated into Japanese as "How was today?". The input is the converted text data, and the output is the translated text data.

[0288] Step 5:

[0289] Sending translation results

[0290] The server transmits the translated text to the terminal via the Internet. The input is the translated text data, and the output is the translated text data transmitted to the terminal.

[0291] Step 6:

[0292] Displaying translation results

[0293] The device then projects the translation results onto a wall or screen via a projector or smart display. For example, the text "How was your day?" is displayed on the wall of your living room. The input is the translated text data sent to the device, and the output is the translated text displayed on the wall or screen.

[0294] (Application example 3)

[0295] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0296] In environments where people speaking different languages ​​coexist, it is important to achieve natural communication without feeling the language barrier. In particular, smooth conversations between multinational customers and store staff are required in brick-and-mortar stores. Conventional translation systems often lack practicality due to limitations in real-time performance and display methods.

[0297] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0298] In this invention, the server includes a means having a generative AI translation and summarization function, a means for projecting conversations translated into multiple languages ​​in a specified space (room), a means for enabling natural conversation even when the language is not the same, and a means for translating conversations with customers who speak different languages ​​in real time using smart glasses and displaying the translation on a display, thereby enabling smooth real-time communication between customers who speak different languages ​​and store clerks.

[0299] "Generative AI" is a system that uses artificial intelligence to generate, translate, and summarize data.

[0300] The "translation and summary function" is a function that converts input language data into another language or summarizes its contents concisely.

[0301] "Designated space" refers to the specific room or area in which the system operates.

[0302] "Multiple languages" refers to two or more different languages ​​and indicates the types of languages ​​that the system supports.

[0303] "Means for displaying dialogue" refers to devices or technologies that visually display the translated dialogue.

[0304] "Means that enable natural conversation" refers to methods and techniques that enable smooth communication without making language differences noticeable.

[0305] "Smart glasses" are glasses-type devices that have built-in displays and sensors and have the ability to visually display information.

[0306] "Means for translating in real time and displaying the results on a display" refers to technology or devices for instantly translating conversations and displaying the results on a display.

[0307] A system for implementing this invention includes means for having generative AI translation and summarization functions, means for displaying conversations translated into multiple languages ​​in a specified space, means for enabling natural conversation even when the languages ​​are not the same, and means for translating conversations with customers who speak different languages ​​in real time using smart glasses and displaying the translation on a display.

[0308] The server uses generative AI to translate and summarize the voice data in real time. Specifically, it converts the voice into text using speech recognition software (e.g., Google Speech Recognition API), and then translates the text into the specified language using translation software (e.g., Google Translate API). The translated text is then displayed on the smart glasses' display.

[0309] The device (smart glasses) receives the translated text sent from the server and displays it on its screen. This allows the store clerk to understand the customer's language and respond appropriately. The smart glasses have a built-in display and sensors, and can display information visually.

[0310] The user (store clerk) wears the smart glasses and converses with customers. What the customer says is translated in real time and displayed on the smart glasses' display, enabling smooth communication without any language barriers.

[0311] As a concrete example, consider a scenario in which an English-speaking tourist visits a brick-and-mortar store in Japan and asks a store clerk for product details. The store clerk is wearing smart glasses, and the tourist's English question is translated into Japanese in real time and displayed on the glasses' display. The clerk then looks at the translation and provides an appropriate response in Japanese.

[0312] Examples of prompts for generative AI models include:

[0313] "An English-speaking tourist visits a brick-and-mortar store in Japan and asks a store clerk for product details. The clerk is wearing smart glasses, and the tourist's English question is translated into Japanese in real time and displayed on the glasses' display. The clerk then looks at the translation and provides an appropriate response in Japanese. Please generate a Python program to realize this scenario."

[0314] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0315] Step 1:

[0316] The server captures the user's (store clerk's) voice through a microphone. The input is voice data, which is converted into text data using speech recognition software (e.g., Google Speech Recognition API). The output is text data.

[0317] Step 2:

[0318] The server inputs the text data obtained in step 1 into translation software (e.g., Google Translate API) and translates it into the specified language. The input is text data, which is converted into another language using translation software. The output is the translated text data.

[0319] Step 3:

[0320] The server sends the translated text data to the smart glasses. The input is the translated text data, which is sent to the smart glasses via the network. The output is the text data displayed on the smart glasses.

[0321] Step 4:

[0322] The terminal (smart glasses) receives the translated text sent from the server and displays it on the display. The input is the translated text data sent from the server, which is displayed on the display. The output is the translated text displayed on the display of the smart glasses.

[0323] Step 5:

[0324] The user (store clerk) reads the translated text displayed on the smart glasses' display and responds appropriately to the customer. The input is the translated text displayed on the smart glasses' display, and the clerk responds to the customer based on this. The output is the appropriate response to the customer.

[0325] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0326] "Example 1"

[0327] One embodiment of the present invention is a system that combines the translation and summarization functions of generative AI with an emotion engine. This system translates and summarizes a user's speech in real time and projects the results on the wall of a specified space (room). Furthermore, the emotion engine recognizes emotions from the user's tone of voice, facial expression, and choice of words, and adjusts the translation and summarization results accordingly. For example, if it recognizes that the user is angry, it adjusts the translation results to appropriately express that anger. It also recognizes emotions such as joy and surprise and adjusts the translation results accordingly. This enables natural conversations that include emotions, even between users who do not share the same language.

[0328] "Example 2"

[0329] The emotion engine can also adjust the content and format of the display. For example, if it recognizes that the user is happy, it can change the color of the displayed text to a brighter color. Or, if it recognizes that the user is surprised, it can make the displayed text larger, providing visual feedback according to the emotion. This allows the user to confirm that their emotions are being recognized by the system and reflected appropriately.

[0330] "Example 3"

[0331] Furthermore, this system not only adjusts the translation and summarization results according to the user's emotions, but also suggests dialogues that correspond to those emotions. For example, if it recognizes that the user is confused, it will identify the parts of the conversation that may be causing that confusion and provide an explanation or commentary. Similarly, if it recognizes that the user is happy, it will suggest replies that share that joy. This allows users to enjoy natural dialogue based on an understanding of their own emotions.

[0332] The processing flow of each embodiment will be described below.

[0333] "Example 1"

[0334] Step 1: The user starts a conversation, which is input to the system via an input device such as a microphone.

[0335] Step 2: Generative AI translates and summarizes the user's speech in real time. This translation and summary translates the user's speech into another language and extracts the main points.

[0336] Step 3: The emotion engine recognizes the user's emotions from their tone of voice, facial expressions, and choice of words.

[0337] Step 4: The generative AI adjusts the translation and summarization results based on the emotions recognized by the emotion engine.

[0338] Step 5: The adjusted translation and summary results are projected onto the wall or other surface of the designated space (room).

[0339] "Example 2"

[0340] Step 1: The user starts a conversation, which is input to the system via an input device such as a microphone.

[0341] Step 2: Generative AI translates and summarizes the user's speech in real time. This translation and summary translates the user's speech into another language and extracts the main points.

[0342] Step 3: The emotion engine recognizes the user's emotions from their tone of voice, facial expressions, and choice of words.

[0343] Step 4: The emotion engine adjusts the content and format of the display depending on the emotion it recognizes. For example, if it recognizes that the user is happy, it changes the color of the displayed text to a brighter color.

[0344] Step 5: The translation and summary results are projected onto the wall of a designated space (room) in the adjusted content and format.

[0345] "Example 3"

[0346] Step 1: The user starts a conversation, which is input to the system via an input device such as a microphone.

[0347] Step 2: Generative AI translates and summarizes the user's speech in real time. This translation and summary translates the user's speech into another language and extracts the main points.

[0348] Step 3: The emotion engine recognizes the user's emotions from their tone of voice, facial expressions, and choice of words.

[0349] Step 4: The generative AI suggests dialogue based on the emotion recognized by the emotion engine. For example, if it recognizes that the user is confused, it will identify parts of the conversation that may be causing that confusion and provide an explanation or commentary.

[0350] Step 5: Project the proposed dialogue onto a wall or other surface in a designated space (room).

[0351] Example 1

[0352] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0353] Conventional translation systems have had problems with natural conversations between users who speak different languages, especially with the difficulty of conveying emotional nuances. Real-time translation and summarization are also inadequate, often disrupting the flow of conversation. Furthermore, there are limited ways to properly display translation results, making them less convenient for users.

[0354] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means having a generative AI translation and summarization function, a means for displaying conversations translated into multiple languages ​​in a specified space, a means for enabling natural conversation even when the languages ​​are not the same, and a means for recognizing user emotions and adjusting the results of translation and summarization according to those emotions. This enables natural conversation, including emotions, even between users who speak different languages, realizes real-time translation and summarization, and allows the translation results to be displayed appropriately on any surface.

[0355] "Generative AI" is a system that uses artificial intelligence technology to generate, translate, and summarize data.

[0356] The "translation and summarization function" converts input text and audio data into other languages ​​and, if necessary, summarizes the content concisely.

[0357] A "designated space" refers to a specific location or area, such as a conference room or living room.

[0358] "Multiple languages" refers to two or more different languages, such as Japanese, English, and French.

[0359] "Conversation projection" refers to a method of displaying translated text on a surface such as a wall or screen using a projector or display.

[0360] "Means that enable natural conversation" refers to methods that facilitate communication between users who speak different languages ​​and enable natural conversation that includes emotions and nuances.

[0361] "Means for recognizing emotions" refers to a method of analyzing emotions from the user's tone of voice, facial expressions, and choice of words, and then providing appropriate translations and summaries based on that information.

[0362] "Adjustment measures" are methods for modifying translation and summarization results based on recognized emotions to make them more appropriate.

[0363] This invention is a system that combines the translation and summarization functions of generative AI with an emotion recognition engine, translating conversations taking place within a specified space into multiple languages ​​in real time and adjusting the translation results by recognizing emotions. Specific embodiments of this system are described below.

[0364] Hardware and software used

[0365] Hardware: high-performance microphones, cameras, projectors, servers

[0366] Software: Generative AI models (e.g., OpenAI's GPT-4), emotion recognition engines (e.g., Microsoft® Azure®'s Emotion API)

[0367] System configuration

[0368] 1. A high-performance microphone installed on the device captures conversations taking place within a designated space in real time.

[0369] 2. A camera installed on the device captures the user's facial expressions.

[0370] 3. The device sends the captured voice and facial expression data to the server via the network.

[0371] 4. The server inputs the received voice data into the generative AI model and converts it into text data.

[0372] 5. At the same time, the server uses an emotion recognition engine to analyze the user's emotions from the facial expression data and voice data.

[0373] 6. The server uses the generative AI model to translate the text data into multiple languages.

[0374] 7. The server adjusts the translation results based on the emotional data analyzed by the emotion recognition engine.

[0375] 8. The server sends the adjusted translation results to the device.

[0376] 9. The device uses a projector to project the translation results onto any surface, such as a wall or screen.

[0377] Specific examples

[0378] For example, suppose there is a Japanese-speaking user A and an English-speaking user B in a conference room. If user A says, "This project is very important," the system processes it as follows:

[0379] 1. The device's microphone captures what User A says.

[0380] 2. The device's camera captures User A's facial expression.

[0381] 3. The device sends the voice data and facial expression data to the server.

[0382] 4. The server converts the voice data into text, and the emotion recognition engine analyzes User A's emotions (e.g., serious expression).

[0383] 5. The server translates the text data into English using a generative AI model.

[0384] 6. Based on the results of the emotion recognition engine, the server adjusts the translation result to "This project is extremely important."

[0385] 7. The server sends the adjusted translation results to the device.

[0386] 8. The terminal uses a projector to project the translation results onto the wall of the conference room.

[0387] Prompt Sentence Examples

[0388] Examples of prompts to be input to a generative AI model include:

[0389] User says: "This project is very important"

[0390] User Sentiment: "Serious"

[0391] Target language: "English"

[0392] By inputting this prompt sentence into a generative AI model, appropriate translation results can be obtained.

[0393] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0394] Step 1:

[0395] A high-performance microphone installed on the device captures conversations taking place within a designated space in real time. The input is a voice signal, and the output is digital voice data. Specifically, the microphone converts the voice signal into digital data.

[0396] Step 2:

[0397] A camera installed on the device captures the user's facial expression. The input is the user's facial expression, and the output is image data. Specifically, the camera generates image data.

[0398] Step 3:

[0399] The device transmits the captured voice and facial expression data to a server via a network. The input is digital voice data and image data, and the output is data transmission to the server. Specifically, the device divides the data into packets and transmits them to the server using the TCP / IP protocol.

[0400] Step 4:

[0401] The server inputs the received voice data into a generative AI model and converts it into text data. The input is digital voice data, and the output is text data. Specifically, the server calls a voice recognition API and converts the voice data into text.

[0402] Step 5:

[0403] At the same time, the server uses an emotion recognition engine to analyze the user's emotions from the facial expression data and voice data. The input is image data and digital voice data, and the output is emotion data. Specifically, the server calls the emotion recognition API and analyzes the facial expression data and voice data.

[0404] Step 6:

[0405] The server uses a generative AI model to translate text data into multiple languages. The input is text data, and the output is translated text data. Specifically, the server calls a translation API to translate the text data into the specified language.

[0406] Step 7:

[0407] The server adjusts the translation results based on the emotional data analyzed by the emotion recognition engine. The input is the translated text data and the emotional data, and the output is the adjusted translated text data. Specifically, the server analyzes the emotional data and applies rules to adjust the tone and expression of the translation results.

[0408] Step 8:

[0409] The server sends the adjusted translation results to the terminal. The input is the adjusted translated text data, and the output is the data sent to the terminal. Specifically, the server divides the data into packets and sends them to the terminal using the TCP / IP protocol.

[0410] Step 9:

[0411] The terminal uses a projector to project the translation results onto any surface, such as a wall or screen. The input is the adjusted translation text data, and the output is the displayed translation result. Specifically, the terminal sends the received translation result to the projector and displays it at the specified location.

[0412] (Application example 1)

[0413] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0414] In a multilingual environment, it is difficult to achieve natural conversations between users who speak different languages. Furthermore, simple translation alone may not convey the nuances of emotions, which can lead to misunderstandings. In brick-and-mortar stores in particular, smooth communication between customers and store clerks is required, so real-time translation and emotional recognition and adjustment are necessary.

[0415] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0416] In this invention, the server includes means for translating and summarizing using generative AI, means for displaying conversations translated into multiple languages ​​in a specified space (room), means for enabling natural conversation even when the languages ​​are not the same, means for recognizing the user's emotions using an emotion recognition engine and adjusting the translation results, and means for displaying the translation results on a smart device, thereby enabling natural conversations that include emotions even between users who speak different languages.

[0417] "Generative AI" refers to artificial intelligence that uses natural language processing to generate, translate, summarize, and otherwise process text.

[0418] The "translation and summary function" is a function that translates input text into another language or summarizes its contents concisely.

[0419] A "designated space" is a specific location or room where the conversation takes place.

[0420] "Multilingual" means two or more different languages.

[0421] "Means for projecting the dialogue" refers to a method or device for projecting the translated text onto a display device such as a wall or screen.

[0422] An "emotion recognition engine" is software or hardware that analyzes and recognizes emotions from a user's tone of voice, facial expressions, and choice of words.

[0423] A "smart device" is a portable electronic device with internet connectivity and advanced computing capabilities.

[0424] "Means for adjusting the translation result" refers to a method or device that appropriately changes the content or expression of the translated text based on the recognized sentiment.

[0425] The system for implementing this invention is configured by combining the translation and summarization functions of generative AI, an emotion recognition engine, and a smart device. A specific embodiment of this system is shown below.

[0426] System configuration

[0427] 1. Generative AI translation and summarization functions

[0428] The server translates and summarizes the user's speech in real time using a generative AI model, such as GPT-4.

[0429] 2. Emotion Recognition Engine

[0430] The server uses an emotion recognition engine, such as Affectiva, to recognize the user's emotions from their tone of voice, facial expressions, and language.

[0431] 3. Smart Devices

[0432] The terminal uses smart devices such as smart glasses or head-mounted displays to display the translation results, allowing store clerks and users to check the translation results in real time. Specific devices that can be used include Google Glass and Microsoft HoloLens (registered trademark).

[0433] Data processing and calculation

[0434] 1. Acquiring voice input

[0435] When the user speaks, the voice data is acquired by the terminal.

[0436] 2. Converting voice data to text

[0437] The acquired voice data is sent to a server and converted into text data by voice recognition software.

[0438] 3. Emotion recognition

[0439] The text data is analyzed by an emotion recognition engine to recognize the user's emotions.

[0440] 4. Translation and Summary

[0441] The generative AI model translates text data into a specified language and summarizes it as needed.

[0442] 5. Adjustment of translation results

[0443] Based on the recognized emotion, the translation result is adjusted appropriately: for example, if the user is angry, the translation result will reflect that emotion.

[0444] 6. Displaying the translation results

[0445] The adjusted translation results are displayed on the smart device, allowing store clerks and users to check the translation results in real time.

[0446] Specific examples

[0447] For example, if a customer says, "I am very happy with the service!", it will be processed as follows:

[0448] 1. Audio data is acquired by the device.

[0449] 2. The audio data is converted into text data.

[0450] 3. The emotion recognition engine recognizes the emotion "happy."

[0451] 4. The generative AI model translates this as "I'm very satisfied with your service!"

[0452] 5. Based on the perceived emotion, it is adjusted to "(with joy) I am very satisfied with the service!"

[0453] 6. The adjusted translation results will be displayed on your smart device.

[0454] Prompt Sentence Examples

[0455] "You will create a program that translates customer utterances in real time, recognizes emotions, and adjusts the translation results accordingly. The emotion recognition engine used is Affectiva, and the translation API is Google Translate API. The translation results will be displayed on the smart glasses. Use the following as an example of a customer utterance: 'I am very happy with the service!'"

[0456] In this way, the present invention can realize natural conversations that include emotions even between users who speak different languages.

[0457] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0458] Step 1:

[0459] The user speaks.

[0460] The user utters, "I am very happy with the service!" This voice data is acquired by the terminal. The input is the user's voice data, and the output is the voice data sent to the terminal.

[0461] Step 2:

[0462] The terminal transmits the voice data to the server.

[0463] The terminal transmits the acquired voice data to the server. The input is the voice data acquired by the terminal, and the output is the voice data transmitted to the server.

[0464] Step 3:

[0465] The server converts the voice data into text data.

[0466] The server uses speech recognition software to convert the voice data into text data: the input is the voice data sent to the server, and the output is text data.

[0467] Step 4:

[0468] The server sends the text data to the emotion recognition engine.

[0469] The server sends the text data to the emotion recognition engine to recognize the user's emotion. The input is the text data, and the output is the recognized emotion data.

[0470] Step 5:

[0471] The server translates the text data using a generative AI model.

[0472] The server uses a generative AI model to translate text data into a specified language. The input is text data, and the output is the translated text data.

[0473] Step 6:

[0474] The server adjusts the translation results based on the emotion data.

[0475] The server adjusts the translation result appropriately based on the recognized emotion data. For example, if the emotion "happy" is recognized, it adds the phrase "(with joy)" to the translation result. The input is the translated text data and emotion data, and the output is the adjusted translation result.

[0476] Step 7:

[0477] The server sends the adjusted translation results to the device.

[0478] The server sends the adjusted translation result to the terminal. The input is the adjusted translation result, and the output is the translation result sent to the terminal.

[0479] Step 8:

[0480] The terminal displays the translation results on the smart device.

[0481] The terminal displays the received translation results on the smart device. The input is the translation result sent to the terminal, and the output is the translation result displayed on the smart device.

[0482] In this way, the user's speech is translated in real time, enabling natural conversation that includes emotions.

[0483] Example 2

[0484] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0485] In situations such as international conferences, when participants speak different languages, language barriers often become an obstacle to communication. It is also difficult to understand the content of the conference and exchange opinions in real time. Furthermore, the quality of communication can be reduced because visual feedback that properly reflects participants' emotions is not provided.

[0486] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means having a translation and summarization function of generative artificial intelligence, means for displaying conversations translated into multiple languages ​​in a specified space, means for enabling natural conversation even when the languages ​​are not the same, means for collecting voice data, means for converting the collected voice data into text data, means for recognizing emotions in the text data, and means for adjusting the content and format of the display based on the emotion recognition results. This enables participants who speak different languages ​​to understand the conversation in real time and receive visual feedback according to their emotions.

[0487] "Generative AI" is an AI that has the ability to analyze voice and text data and translate and summarize.

[0488] The "translation and summary function" is a function that converts input language data into another language or summarizes its contents concisely.

[0489] A "designated space" refers to a specific location such as a conference room or hall.

[0490] "Multiple languages" refers to different languages, including, for example, Japanese, English, and French.

[0491] "Means for projecting the dialogue" refers to a device or method for visually displaying the translated text using a projector, display, or the like.

[0492] "Means for collecting voice data" refers to a method for acquiring speech as digital voice data using a microphone or voice input device.

[0493] "Means for converting into text data" refers to a method of converting voice data into text information using voice recognition software.

[0494] "Emotion recognition means" refers to software or algorithms that analyze and identify a user's emotions from text data or voice data.

[0495] "Means for adjusting the content and format of the display" refers to a method for changing visual attributes such as the color and size of the displayed text based on the results of emotion recognition.

[0496] This invention is a system that enables participants who speak different languages ​​to understand conversations in real time and receive visual feedback according to their emotions in situations such as international conferences. This system includes means for having a generative artificial intelligence translation and summarization function, means for displaying conversations translated into multiple languages ​​in a specified space, means for enabling natural conversation even when the languages ​​are not the same, means for collecting voice data, means for converting the collected voice data into text data, means for recognizing emotions in the text data, and means for adjusting the content and format of the display based on the results of the emotion recognition.

[0497] Hardware and software used

[0498] Hardware:

[0499] High-performance servers (e.g., high-performance servers)

[0500] Projectors (e.g. high-resolution projectors)

[0501] Microphone (e.g. high-sensitivity microphone)

[0502] software:

[0503] Speech recognition software (e.g., speech recognition API)

[0504] Translation software (e.g. translation API)

[0505] Emotion recognition software (e.g., emotion recognition API)

[0506] Program processing explanation

[0507] The server translates the speech of participants into other languages ​​in real time and projects it onto the wall. It also has an emotion engine that adjusts the content and format of the display according to the user's emotions.

[0508] The device uses a microphone to collect speech from meeting participants and sends the audio data to a server. The server uses speech recognition software to convert the audio data into text data. The converted text data is then input into translation software and translated into the specified language. The translated text data is then input into emotion recognition software to analyze the user's emotions. The color and size of the displayed text are adjusted based on the results of the emotion recognition. The adjusted text data is then sent to a projector and projected onto the wall.

[0509] Specific examples

[0510] As a specific use case, consider the following scenario:

[0511] Scenario: At an international conference, there is participant A who speaks Japanese and participant B who speaks English.

[0512] Participant A says, "Hello everyone."

[0513] The device collects this speech with a microphone and transmits the audio data to the server.

[0514] The server uses voice recognition software to convert "Hello everyone" into text.

[0515] The server translates this to "Hello, everyone" using its translation software.

[0516] The server uses emotion recognition software to analyze the emotion in "Hello everyone" and detect joy.

[0517] The server changes the color of the text to a brighter color and sends it to the projector.

[0518] The terminal projects "Hello, everyone" onto the wall via a projector.

[0519] Prompt Sentence Examples

[0520] Write a program that translates the Japanese phrase "Hello everyone" into English, performs emotion recognition, and if joy is detected, changes the color of the text to a brighter color.

[0521] In this way, users can be sure that their emotions are recognized and reflected appropriately by the system.

[0522] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0523] Step 1: Collecting audio data

[0524] The device collects speech from participants using a microphone, and the collected voice data is sent to the server in real time.

[0525] Input: Meeting participant's remarks

[0526] Output: Digital audio data

[0527] How it works: A high-sensitivity microphone connected to the device captures what participants say and sends the audio data digitally to a server.

[0528] Step 2: Voice Recognition

[0529] The server inputs the received voice data into voice recognition software and converts the voice data into text data.

[0530] Input: Digital audio data

[0531] Output: Text data

[0532] Specific operation: The server sends the voice data to a speech recognition API and converts the voice "Hello everyone" into text "Hello everyone."

[0533] Step 3: Translation

[0534] The server inputs the converted text data into translation software and translates it into the specified language.

[0535] Input: Text data

[0536] Output: Translated text data

[0537] Specific operation: The server sends the text data "Hello, everyone" to the translation API, which translates it to "Hello, everyone."

[0538] Step 4: Emotion Recognition

[0539] The server inputs the translated text data into emotion recognition software to analyze the user's emotions.

[0540] Input: Translated text data

[0541] Output: Emotion recognition result

[0542] Specific operation: The server sends the text data "Hello everyone" to the emotion recognition API and detects the emotion of joy.

[0543] Step 5: Adjust the visual feedback

[0544] The server adjusts the color and size of the displayed text based on the results of emotion recognition.

[0545] Input: Emotion recognition results

[0546] Output: Adjusted text data

[0547] Specific operation: Based on the emotion recognition result, the server generates an instruction to change the color of the text to yellow.

[0548] Step 6: Projection

[0549] The terminal sends the adjusted text data to the projector, which projects it onto the wall.

[0550] Input: Adjusted text data

[0551] Output: Text projected on the wall

[0552] Specific operation: The terminal sends the adjusted text data "Hello, everyone" received from the server to the projector, projecting it in yellow letters on the wall.

[0553] In this way, users can be sure that their emotions are recognized and reflected appropriately by the system.

[0554] (Application example 2)

[0555] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0556] In multilingual environments such as international conferences and virtual stores, there is a need to achieve smooth communication across language barriers. Another challenge is to communicate information more intuitively and effectively by providing visual feedback that reflects the user's emotions. In particular, in virtual stores, it is important to provide appropriate information about products that users are interested in, thereby motivating them to purchase.

[0557] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0558] In this invention, the server includes a means for translating and summarizing using generative AI, a means for displaying conversations translated into multiple languages ​​in a specified space, a means for enabling natural conversation even when the language is not the same, a means for recognizing a user's emotions and adjusting the content and format of the display, and a means for displaying product information in a virtual store using smart glasses. This enables real-time translation and visual feedback according to emotions in a multilingual environment, making it easier for users to intuitively understand information. Furthermore, in the virtual store, providing appropriate information for products in which the user is interested can increase purchasing motivation.

[0559] "Generative AI" is a system that uses artificial intelligence to generate and convert data.

[0560] "Translation and summary function" refers to a function that translates input text or audio into another language or summarizes the content concisely.

[0561] A "designated space" refers to a specific location or area, including conference rooms and virtual stores.

[0562] A "conversation translated into multiple languages" is a conversation between multiple users who speak different languages, translated into each of the languages.

[0563] "Displaying means" refers to devices or techniques used to visually display translated text or information.

[0564] "Means that enable natural conversation" are techniques and methods that enable smooth communication beyond language differences.

[0565] "Means for recognizing the user's emotions and adjusting the content and format displayed" refers to technology that detects the user's emotions and changes the color and size of the displayed information accordingly.

[0566] "Smart glasses" are glasses-type devices that have the function of displaying information.

[0567] A "virtual store" is a virtual store that operates on the Internet, where users can browse and purchase products online.

[0568] "Product information" refers to information such as product details, features, and prices.

[0569] A system for implementing this invention includes means for having generative AI translation and summarization functions, means for displaying conversations translated into multiple languages ​​in a specified space, means for enabling natural conversation even when the languages ​​are not the same, means for recognizing the user's emotions and adjusting the content and format of the display, and means for displaying product information in a virtual store using smart glasses.

[0570] The server runs programs for speech recognition, translation, emotion recognition, and visual feedback. Specifically, speech recognition uses a microphone and Google's speech recognition API. Translation uses Google Translate API and emotion recognition uses Emotion Recognition API. Visual feedback uses OpenCV and displays information on smart glasses.

[0571] As a user wears the smart glasses and walks around the virtual store, descriptions of each product are translated into the user's language in real time. The display also changes depending on the user's emotions, providing feedback such as enlarging detailed information about products that interest the user.

[0572] For example, if a user says "I'd like to know more about this product" in a virtual store, this statement is translated into English in real time, and if the user is recognized as interested, the translated text is displayed in a brighter color and larger.

[0573] Examples of prompts to input to a generative AI model include:

[0574] When a user says "I want to know more about this product," create a program that translates this statement into English in real time and, if it recognizes that the user is interested, displays the translated text in a larger, brighter color.

[0575] In this way, the user can easily understand the information intuitively, and in the virtual store, appropriate information can be provided for the product in which the user is interested, thereby increasing the user's desire to purchase.

[0576] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0577] Step 1:

[0578] The server captures the user's speech through a microphone. The input is the user's voice, and the output is voice data. This voice data is converted into text using Google's speech recognition API. Specifically, the server acquires voice data from the microphone, sends it to the speech recognition API, and receives text data.

[0579] Step 2:

[0580] The server sends the text data obtained from the speech recognition API to the Google Translate API and translates it into the specified language. The input is the text data from the speech recognition API, and the output is the translated text data. Specifically, the server sends the text data to the translation API and receives the translation results.

[0581] Step 3:

[0582] The server captures the user's face through the camera and recognizes the user's emotions using the Emotion Recognition API. The input is image data from the camera, and the output is recognized emotion data. Specifically, the server acquires image data from the camera, sends it to the emotion recognition API, and receives the emotion data.

[0583] Step 4:

[0584] The server adjusts the display content and format based on the translated text data and the recognized emotion data. The input is the translated text data and emotion data, and the output is the adjusted display data. Specifically, the server changes the color and size of the text according to the emotion data.

[0585] Step 5:

[0586] The server sends the adjusted display data to the smart glasses and displays it to the user. The input is the adjusted display data, and the output is the information displayed on the smart glasses. In particular, the server sends the display data to the smart glasses, and the user visually confirms it.

[0587] Step 6:

[0588] If the user walks around the virtual store and requests more information about a product they are interested in, the server again performs voice recognition, translation, and emotion recognition and displays the appropriate information. The input is the user's new voice data, and the output is updated display data. Specifically, the server again captures the voice data and repeats steps 1 to 5 described above.

[0589] Example 3

[0590] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0591] When a family member who speaks different languages ​​lives together, the language barrier can become an obstacle to communication. Furthermore, communication can be hindered because appropriate dialogue is not conducted in response to the user's emotions. There is a need to solve these problems and realize natural conversations within the home.

[0592] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0593] In this invention, the server includes means for translating and summarizing using generative AI, means for displaying conversations translated into multiple languages ​​in a specified space, means for enabling natural conversation even when the languages ​​are not the same, means for recognizing the user's emotions and adjusting the translation and summarization results according to those emotions, and means for suggesting dialogue according to the user's emotions.This allows family members who speak different languages ​​to enjoy natural conversation and also allows them to receive dialogue suggestions according to their emotions.

[0594] "Generative AI" refers to technology that uses artificial intelligence to generate, translate, and summarize data.

[0595] "Translation and summary function" refers to the function of converting input text or audio into another language or summarizing the content concisely.

[0596] "Designated space" refers to the specific location or room where the system is installed.

[0597] "Multilingually translated conversation" refers to conversation content that has been translated into different languages.

[0598] "Means of projecting" refers to means of displaying information using a projector, display, etc.

[0599] "Means that enable natural conversation" refers to techniques and methods that enable smooth communication beyond language differences.

[0600] "Means for recognizing user emotions" refers to technology that uses cameras and sensors to analyze the user's facial expressions and tone of voice to determine their emotions.

[0601] "Means for adjusting translation and summary results according to emotions" refers to technology that appropriately changes the content of translations and summaries based on the user's emotional state.

[0602] "Means for suggesting dialogue" refers to technology that generates appropriate dialogue content and responses based on the user's emotions.

[0603] The present invention is a system that allows family members who speak different languages ​​to enjoy natural conversation, and also suggests dialogues according to the user's emotions. A specific embodiment of this system is described below.

[0604] Hardware and software used

[0605] Hardware: high-performance microphones, speakers, projectors, cameras, and servers

[0606] Software: speech recognition software (e.g., Google Speech-to-Text), translation software (e.g., Google Translate API), emotion recognition software (e.g., Microsoft Azure Emotion API), generative AI models (e.g., OpenAI GPT-4)

[0607] System Operation

[0608] 1. Acquiring voice input:

[0609] The server captures the user's voice in real time through a high-performance microphone.

[0610] Example: When family members start talking in the living room, a microphone picks up their voice and sends it to a server.

[0611] 2. Speech Recognition:

[0612] The server uses speech recognition software to convert the captured speech into text.

[0613] Example: Speech "Hello, how was your day?" is converted to text, producing the string "Hello, how was your day?"

[0614] 3. Translation:

[0615] The server uses translation software to translate the text into different languages.

[0616] Example: The Japanese text "Hello, how was your day?" is translated into English to produce the text "Hello, how was your day?"

[0617] 4. Emotion recognition:

[0618] The server captures the user's facial expressions with a camera and analyzes the user's emotions using emotion recognition software.

[0619] Example: A camera takes a picture of a user's face and sends the image data to a server, which analyzes the image and determines that the user is confused.

[0620] 5. Translation and summary adjustment:

[0621] The server adjusts the translation and summarization results according to the user's emotions.

[0622] Example: Changing translations to more concise and understandable language for confused users.

[0623] 6. Dialogue Proposal:

[0624] The server uses the generated AI model to suggest dialogues based on the user's emotions.

[0625] Example: For a confused user, identifying the areas that are causing the conversation and generating suggestions like, "I think this part is confusing. Would you like me to explain it more?"

[0626] 7. Displaying the results:

[0627] The terminal (projector) projects the translation results and dialogue suggestions onto the wall.

[0628] Example: The translation "Hello, how was your day?" appears on the wall, along with a suggestion: "This part may be difficult to understand. Would you like me to explain further?"

[0629] Prompt Sentence Examples

[0630] "If the user is confused, identify the part of the conversation that is causing the confusion and provide an explanation or commentary."

[0631] "If the user is happy, suggest a response that shares that joy."

[0632] This system allows family members who speak different languages ​​to enjoy natural conversation and also allows them to receive suggestions for dialogue according to their emotions.The flow of the identification process in the third embodiment will be described with reference to FIG.

[0633] Step 1:

[0634] Acquiring voice input

[0635] The server captures the user's voice in real time through a high-performance microphone.

[0636] Input: User's voice

[0637] Output: Audio data

[0638] How it works: When family members start talking in the living room, the microphone picks up their voice and sends it to the server.

[0639] Step 2:

[0640] Voice Recognition

[0641] The server uses speech recognition software to convert the captured speech into text.

[0642] Input: Audio data

[0643] Output: Text data

[0644] What it does: The speech "Hello, how was your day?" is converted to text, resulting in the string "Hello, how was your day?"

[0645] Step 3:

[0646] translation

[0647] The server uses translation software to translate the text into different languages.

[0648] Input: Text data

[0649] Output: Translated text data

[0650] Specific behavior: The Japanese text "Hello, how was your day?" is translated into English to produce the text "Hello, how was your day?"

[0651] Step 4:

[0652] emotion recognition

[0653] The server captures the user's facial expressions with a camera and analyzes the user's emotions using emotion recognition software.

[0654] Input: User facial expression image data

[0655] Output: Emotion data (e.g., confusion, joy, etc.)

[0656] What it does: The camera takes a picture of the user's face and sends the image data to the server, which analyzes the image and determines that the user is confused.

[0657] Step 5:

[0658] Coordination of translation and summarization

[0659] The server adjusts the translation and summarization results according to the user's emotions.

[0660] Input: Translated text data, emotion data

[0661] Output: Adjusted translation and summary text data

[0662] What it does: It makes the translation more concise and easier to understand for confused users.

[0663] Step 6:

[0664] Dialogue proposal

[0665] The server uses the generated AI model to suggest dialogues based on the user's emotions.

[0666] Input: Emotion data, conversation text data

[0667] Output: Text data of dialogue suggestions

[0668] Specific Behavior: Identify areas of conversation that are causing confusion for confused users and generate suggestions such as, "This part seems confusing. Would you like me to explain it further?"

[0669] Step 7:

[0670] Displaying the results

[0671] The terminal (projector) projects the translation results and dialogue suggestions onto the wall.

[0672] Input: Adjusted translation and summary text data, dialogue suggestion text data

[0673] Output: Text projected on the wall

[0674] What it does: The translated result "Hello, how was your day?" appears on the wall, along with a suggestion: "This part might be difficult to understand. Would you like me to explain further?"

[0675] (Application example 3)

[0676] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0677] In an environment where people who speak different languages ​​coexist, it is necessary to realize natural communication without feeling a language barrier. It is also important to understand the emotions of the other person when communicating and respond appropriately accordingly. Conventional systems have problems with poor real-time translation and emotion recognition, which reduces the quality of communication.

[0678] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes means having a generative AI translation and summarization function, means for displaying conversations translated into multiple languages ​​in a specified space, means for enabling natural conversation even when the languages ​​are not the same, means for recognizing emotions, and means for suggesting dialogue based on the recognized emotions. This enables people who speak different languages ​​to communicate naturally through conversations translated in real time and to respond appropriately according to the other person's emotions.

[0679] "Generative AI" is a system that uses artificial intelligence technology to generate, translate, and summarize data.

[0680] The "translation and summary function" is a function that converts input language data into another language or summarizes its contents concisely.

[0681] "Designated space" refers to the specific location or room where the system is installed.

[0682] "Multilingually translated conversation" refers to conversation content that is translated into different languages ​​in real time.

[0683] "Projection means" refers to any device or technology that allows the translated dialogue to be displayed on a wall, screen, or any other surface.

[0684] "Means that enable natural conversation" refers to techniques and methods that enable smooth communication without making language differences noticeable.

[0685] "Means for recognizing emotions" refers to technology or devices that analyze a user's facial expressions, tone of voice, etc. to identify emotions.

[0686] "Means for making dialogue suggestions" refers to techniques and methods for suggesting appropriate dialogue content and response methods based on recognized emotions.

[0687] A system for implementing this invention includes a generative AI translation and summarization function, a means for displaying conversations translated into multiple languages ​​in a specified space, a means for enabling natural conversation even when the languages ​​are not the same, a means for recognizing emotions, and a means for suggesting dialogue based on the recognized emotions.

[0688] The server provides generative AI translation and summarization functions. Specifically, it uses the Google Translate API to translate input language data into other languages ​​in real time and summarizes the content concisely. This enables smooth communication between users who speak different languages.

[0689] The device provides a means to project translated conversations into multiple languages ​​in a designated space. For example, smart glasses or a head-mounted display can be used to display the translated conversations in the user's field of vision. This allows the user to continue the conversation naturally without being aware of language differences.

[0690] The Emotion Recognition API is used to recognize emotions. The device's camera captures the user's facial expressions and tone of voice, which are then analyzed to identify their emotions. The recognized emotions are sent to the server and used as data to make dialogue suggestions.

[0691] The server provides a means to suggest dialogue based on the perceived emotions. For example, if the user is perceived as confused, the server can identify parts of the conversation that may be causing this confusion and provide explanations or commentary. If the user is perceived as happy, the server can suggest replies that share that joy.

[0692] As a concrete example, consider the following scenario: When a foreign tourist asks a question about a product in a brick-and-mortar store in Japan, the store clerk will translate and display the customer's language in real time through the smart glasses. Furthermore, the clerk will recognize the customer's emotions by analyzing their facial expressions and tone of voice, and suggest appropriate dialogue. For example, a prompt message will be displayed saying, "If the customer is confused, please provide a detailed explanation."

[0693] Using this system, people who speak different languages ​​can communicate naturally through real-time translated conversations and respond appropriately based on the other person's emotions.

[0694] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0695] Step 1:

[0696] A user puts on a device (smart glasses or a head-mounted display) and starts a conversation. The device's camera captures the user's face, and the microphone collects audio. The input is the user's video and audio data. The output is the captured video and audio data.

[0697] Step 2:

[0698] The device sends the captured voice data to the server, which then translates the voice data in real time using the Google Translate API. The input is the voice data, and the output is the translated text data. Specifically, the voice data is converted into text, and the text is translated into the specified language.

[0699] Step 3:

[0700] The server sends the translated text data to the terminal, which then displays the translated text in the user's field of view. The input is the translated text data, and the output is the translated text displayed in the user's field of view. Specifically, the text is displayed on the display of the smart glasses.

[0701] Step 4:

[0702] The device camera captures the user's facial expressions and recognizes their emotions using the Emotion Recognition API. The input is the user's video data, and the output is the recognized emotion data. Specifically, the video data is analyzed to identify the emotion from the user's facial expressions.

[0703] Step 5:

[0704] The device sends the recognized emotion data to the server, which then generates appropriate dialogue suggestions based on the emotion data. The input is emotion data, and the output is dialogue suggestions. Specifically, the server analyzes the emotion data and makes suggestions such as providing detailed explanations if the user is confused.

[0705] Step 6:

[0706] The server sends the generated dialogue suggestions to the device, which then displays the suggestions in the user's field of view. The input is the dialogue suggestions, and the output is the suggestions displayed in the user's field of view. Specifically, the suggestions are displayed on the display of the smart glasses.

[0707] Step 7:

[0708] The user then carries out the next dialogue based on the displayed suggestions. The input is the user's new voice data, and the output is new voice data to return to the next step 1. Specifically, the user continues the conversation according to the suggestions.

[0709] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0710] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0711] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are listed.

[0712] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0713] [Second embodiment]

[0714] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0715] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0716] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0717] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0718] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0719] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0720] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0721] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0722] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0723] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0724] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0725] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0726] "Example 1"

[0727] One embodiment of the present invention provides a system with generative AI translation and summarization capabilities. This system is installed in a designated space (e.g., a conference room or living room) and translates conversations taking place within that space into multiple languages ​​in real time. The translation results are projected onto any surface, such as a wall or screen. This allows participants who do not share the same language to converse naturally.

[0728] "Example 2"

[0729] As a specific example of its use, consider an international conference. Even if participants speak different languages, this system can translate each participant's remarks into other languages ​​in real time and project them onto the wall. This allows all participants to understand the content of the conference and exchange opinions freely.

[0730] "Example 3"

[0731] This system is also suitable for use at home. For example, if a family member who speaks different languages ​​lives together, installing this system in the living room will facilitate smooth communication between family members. Family conversations are translated in real time and projected onto the wall, allowing for natural conversation without any sense of language barriers.

[0732] The processing flow of each embodiment will be described below.

[0733] "Example 1"

[0734] Step 1: A system with generative AI translation and summarization capabilities is installed in a designated space (e.g., a conference room or living room).

[0735] Step 2: Conversations taking place in the space are translated into multiple languages ​​in real time by the system.

[0736] Step 3: The translation result is projected onto any surface, such as a wall or screen.

[0737] Step 4: Even if participants do not share the same language, they can converse naturally by looking at the translation displayed on the screen.

[0738] "Example 2"

[0739] Step 1: At an international conference where participants speak different languages, the system is installed in the conference room.

[0740] Step 2: The system translates each participant's speech into other languages ​​in real time.

[0741] Step 3: The translation result is projected onto the wall.

[0742] Step 4: All participants can see the translation on the screen, understand the content of the meeting, and freely exchange opinions.

[0743] "Example 3"

[0744] Step 1: If your family members speak different languages, install the system in the living room or other area.

[0745] Step 2: The system translates the family's conversation in real time.

[0746] Step 3: The translation result is projected onto the wall.

[0747] Step 4: By having all family members see the translation displayed on screen, they can have natural conversations without feeling any language barrier.

[0748] Example 1

[0749] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0750] It is difficult for participants who speak different languages ​​to converse naturally in the same space. In particular, language barriers can be an obstacle in meetings or at home, making it difficult to share information smoothly. To solve this problem, real-time multilingual translation and summarization are required.

[0751] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0752] In this invention, the server includes means for capturing speech in real time, means for converting speech data into text data, means for translating and summarizing the text data into multiple languages ​​using a generative AI model, and means for displaying the translation results, thereby enabling participants who speak different languages ​​to converse naturally in the same space.

[0753] "Real-time audio capture means" refers to devices or technologies that instantly collect audio occurring within a designated space and convert it into a digital signal.

[0754] A "means for converting voice data into text data" is software or algorithms that analyzes collected voice signals and converts them into a corresponding text format.

[0755] A "generative AI model" is an advanced artificial intelligence model for natural language processing, capable of translating and summarizing text data.

[0756] "Means for translating and summarizing text data into multiple languages" means techniques and processes that use generative AI models to translate and, where necessary, summarize input text data into multiple specified languages.

[0757] "Means for displaying the translation results" refers to devices or technologies for visually displaying the translated text data, including projectors and displays.

[0758] "Means that enable natural conversation even when the language is not shared" refers to technologies and systems that enable participants who speak different languages ​​to understand conversation content that is translated in real time and communicate smoothly.

[0759] MODE FOR CARRYING OUT THE INVENTION

[0760] This invention is a system that enables participants who speak different languages ​​to converse naturally in the same space. The system includes a series of processes that capture speech in real time, convert the speech data into text data, translate and summarize the text data into multiple languages ​​using a generative AI model, and display the translation results.

[0761] Hardware and software used

[0762] Hardware:

[0763] Microphone: Highly sensitive microphones are used to capture sounds occurring within a designated space in real time.

[0764] Server: Provides the computational resources to process audio data, run generative AI models, and manage translation results.

[0765] Projector or display: Used to project the translation results onto a wall or screen.

[0766] software:

[0767] Speech recognition software (e.g., Google Speech-to-Text API): converts voice data into text data.

[0768] Generative AI models (e.g., GPT-4): Translate text data into multiple languages ​​and summarize it where necessary.

[0769] Translation software (e.g., Google Translate API): Used to supplement generative AI models.

[0770] Specific operation of the system

[0771] The server captures conversations in real time through microphones installed in the designated space. The captured audio data is converted into text using speech recognition software. A generative AI model is then used to translate the text into multiple languages ​​and summarize it as needed. The translated and summarized text is then projected onto a wall or screen via a projector or display.

[0772] Specific examples

[0773] For example, if there are participants who speak Japanese and English in a conference room, the system will operate as follows:

[0774] 1. The server captures speech in Japanese through a microphone installed in the conference room.

[0775] 2. The captured voice data is converted into text data using voice recognition software.

[0776] 3. The server uses a generative AI model to translate the Japanese text data into English.

[0777] 4. If necessary, the generative AI model summarizes the text data.

[0778] 5. The translated and summarized text data is displayed on a screen via a projector.

[0779] 6. The English translation will be displayed on the screen so that English-speaking participants can understand the content.

[0780] Prompt Sentence Examples

[0781] An example of a prompt sentence to input to the generative AI model is as follows:

[0782] "Please translate the conversation in the conference room in real time. Please translate and summarize the following Japanese text into English."

[0783] Using this prompt, the generative AI model performs the specified task, translating and summarizing the conversation.

[0784] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0785] Step 1:

[0786] Audio Capture

[0787] The server captures conversations in real time through microphones installed in the designated space. The input is the sound in the space, and the output is audio data converted into a digital signal. Specifically, the microphone collects the sound, converts it into a digital signal, and sends it to the server.

[0788] Step 2:

[0789] Voice Recognition

[0790] The server converts the received voice data into text data using speech recognition software (e.g., Google Speech-to-Text API). The input is voice data converted into digital signals, and the output is the corresponding text data. Specifically, the server inputs the voice data into the speech recognition software, which then analyzes the voice and generates text data.

[0791] Step 3:

[0792] Translation and summary

[0793] The server uses a generative AI model (e.g., GPT-4) to translate text data into multiple languages ​​and summarize it as needed. The input is text data obtained from speech recognition software, and the output is translated and summarized text data. Specifically, the server inputs text data into the generative AI model, and the model analyzes the text, translates it into the specified language, and summarizes it as needed.

[0794] Step 4:

[0795] display

[0796] The terminal (projector or display) projects the translation results sent from the server onto a wall or screen. The input is translated and summarized text data, and the output is a visually displayed translation result. Specifically, the server sends the translation result to the terminal, and the terminal displays the received data on the projector or display. The displayed translation result is projected in a location where it can be seen by participants.

[0797] (Application example 1)

[0798] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0799] There is a need to provide an environment where people who speak different languages ​​can converse naturally. In particular, in physical stores, it is important that communication between customers who speak a foreign language and store staff is smooth. However, currently, language barriers exist, making it difficult to provide smooth customer service. To solve this problem, a system with real-time translation functionality is needed.

[0800] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0801] In this invention, the server includes means for translating and summarizing using generative AI, means for displaying conversations translated into multiple languages ​​in a specified space (room), means for enabling natural conversation even when the language is not the same, and means for displaying the translation results on a smart device. This allows people who speak different languages ​​to view the translated conversation in real time through their smart devices and converse naturally.

[0802] "Generative AI" refers to artificial intelligence that generates new information based on data.

[0803] The "translation and summarization function" refers to the function that converts input language data into another language or summarizes its contents concisely.

[0804] A "designated space" refers to a specific location or area, such as a room or conference room.

[0805] "Multilingual" means two or more different languages.

[0806] "Means for displaying dialogue" refers to a device or method for visually displaying the translated dialogue.

[0807] "Means that enable natural conversation even when no common language is used" refers to techniques and methods that enable people who speak different languages ​​to communicate smoothly.

[0808] A "smart device" is an electronic device with internet connectivity and advanced computing power, including smartphones and smart glasses.

[0809] "Means for displaying the translation result" refers to a device or method for visually presenting the translated content to the user.

[0810] To implement the present invention, the following system configuration is required.

[0811] System Program

[0812] The system includes a program with generative AI translation and summarization functions. Specifically, it requires a program that integrates speech recognition, translation, and display functions.

[0813] Hardware and Software

[0814] This system uses the following hardware and software:

[0815] Hardware: Smart glasses (e.g., Google Glass), microphone

[0816] Software: Python, Google Speech Recognition API, Google Translate API, OpenCV

[0817] Processing flow

[0818] 1. Speech recognition: The server converts the voice data acquired from the microphone into text data using the Google Speech Recognition API. Through this process, what the user said is captured as text.

[0819] 2. Translation: The server translates the acquired text data into the specified language using the Google Translate API. This process enables communication between users who speak different languages.

[0820] 3. Display: The server uses OpenCV to display the translated text on the smart glasses display, allowing the user to check the translation results in real time.

[0821] Specific examples

[0822] For example, if an English-speaking customer asks, "Where can I find the electronics section?", the server will recognize the speech and translate it into Japanese, displaying it on the smart glasses' display. This allows store staff to provide smooth service without having to worry about language barriers.

[0823] Prompt Sentence Examples

[0824] An example of a prompt sentence to input to the generative AI model is as follows:

[0825] Create a program that translates English speech into Japanese and displays it on the smart glasses display. Use the Google Speech Recognition API for speech recognition and the Google Translate API for translation. Use OpenCV for display.

[0826] In this way, the present invention provides an environment in which people who speak different languages ​​can converse naturally.

[0827] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0828] Step 1:

[0829] The server acquires voice data from the microphone. The user's spoken voice is input. This voice data is sent to the server and becomes input data for voice recognition processing.

[0830] Step 2:

[0831] The server converts the acquired voice data into text data using the Google Speech Recognition API. Voice data is input, and text data is output through a voice recognition algorithm. Through this process, what the user said is captured as text.

[0832] Step 3:

[0833] The server translates the converted text data into the specified language using the Google Translate API. Text data is input, and the translation algorithm outputs text data in the specified language. This process enables communication between users who speak different languages.

[0834] Step 4:

[0835] The server uses OpenCV to display the translated text data on the smart glasses display. The translated text data is input and displayed on the smart glasses display through a display algorithm. This process allows the user to check the translation results in real time.

[0836] Step 5:

[0837] The user checks the translation results displayed on the smart glasses display and continues the conversation. In this step, the user visually checks the translation results and uses them as input data to decide the next action.

[0838] Example 2

[0839] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0840] In situations such as international conferences, when participants speak different languages, language barriers can become an obstacle to communication. To solve this problem, real-time multilingual translation and display is required, but conventional technology has had difficulty in doing so efficiently and accurately.

[0841] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0842] In this invention, the server includes means for acquiring voice data, means for converting the voice data into text data, means for translating the text data into another language, and means for displaying the translated text data, thereby enabling participants who speak different languages ​​to communicate in real time.

[0843] "Voice data" refers to data in which a user's speech or voice is recorded in digital format.

[0844] "Text data" refers to data obtained by converting voice data into character information.

[0845] "Translation" is the process of converting text data written in one language into another language.

[0846] "Display" refers to the visual presentation of the translated text data.

[0847] "Speech recognition software" is software for converting voice data into text data.

[0848] A "generative AI model" is an artificial intelligence model for translating text data into other languages.

[0849] A "projector" is a device for projecting text data onto any surface.

[0850] This invention is a system that enables participants who speak different languages ​​to communicate in real time. The system performs a series of processes: acquiring voice data, converting it into text data, and then translating and displaying it in other languages.

[0851] First, the user speaks using a device (microphone or smartphone). The device captures this speech as voice data and sends it to the server. The server then uses voice recognition software (e.g., Google Cloud Speech-to-Text) to convert the received voice data into text data. Next, the server uses a generative AI model (e.g., OpenAI's GPT-4) to translate the text data into another language. Finally, the server projects the translated text data onto a wall using a projector or display.

[0852] As a concrete example, consider an international conference with participant A who speaks English and participant B who speaks Japanese. If participant A says "Hello, everyone. Welcome to the conference," this utterance is processed as follows:

[0853] 1. Participant A's device sends voice data to the server.

[0854] 2. The server uses Google Cloud Speech-to-Text to convert the speech into text data: "Hello, everyone. Welcome to the conference."

[0855] 3. The server uses a generative AI model (GPT-4) to translate this text data into Japanese: "Hello, everyone. Welcome to the meeting."

[0856] 4. The translated text data is projected onto the wall using a projector.

[0857] Example prompt sentence:

[0858] "Please translate the English text into Japanese: Hello, everyone. Welcome to the conference."

[0859] In this way, participant A's remarks are translated into Japanese in real time, allowing participant B to understand the content of the meeting. This system enables participants who speak different languages ​​to communicate smoothly.

[0860] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0861] Step 1:

[0862] A user speaks using a device (microphone or smartphone), which captures the speech as voice data and sends it to the server.

[0863] Input: User's speech (audio)

[0864] Output: Digital audio data

[0865] Specific operation: The user says, "Hello, everyone. Welcome to the conference." The device records this speech and sends it to the server as digital audio data.

[0866] Step 2:

[0867] The server converts the received voice data into text data using voice recognition software (e.g., Google Cloud Speech-to-Text).

[0868] Input: Digital audio data

[0869] Output: Text data

[0870] Specific operation: The server inputs the voice data into Google Cloud Speech-to-Text and obtains the text data "Hello, everyone. Welcome to the conference."

[0871] Step 3:

[0872] The server uses a generative AI model (e.g., OpenAI's GPT-4) to translate the text data into other languages.

[0873] Input: Text data

[0874] Output: Translated text data

[0875] Specific operation: The server inputs the text data "Hello, everyone. Welcome to the conference." into GPT-4 and obtains the Japanese translation result "Hello, everyone. Welcome to the conference."

[0876] Step 4:

[0877] The server then projects the translated text data onto a wall using a projector or display.

[0878] Input: Translated text data

[0879] Output: Text displayed on the wall

[0880] Specific operation: The server sends the text data "Hello everyone. Welcome to the meeting." to the projector and displays it on the wall.

[0881] In this way, the user's speech is translated into other languages ​​in real time, allowing other participants to understand the content of the conference.

[0882] (Application example 2)

[0883] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0884] The goal is to solve the problem of passengers who speak different languages ​​having difficulty communicating smoothly in self-driving vehicles, and to ensure that passengers can understand destination directions, emergency instructions, and tourist information in real time.

[0885] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means having a generative AI translation and summarization function, means for displaying conversations translated into multiple languages ​​in a specified space (room), means for enabling natural conversation even when the languages ​​are not the same, means for providing translated information in real time via an in-vehicle display device or mobile device when passengers in an autonomous vehicle speak different languages, and means for playing back the translated information by voice. This enables passengers who speak different languages ​​to communicate smoothly in an autonomous vehicle.

[0886] "Generative AI" is a technology that uses artificial intelligence to generate, translate, and summarize data.

[0887] The "translation and summary function" converts input language data into another language, extracts important information, and summarizes it concisely.

[0888] A "designated space" refers to a specific location or area, which in this case means a physical space such as a room or a car.

[0889] "Multiple languages" refers to two or more different languages, including, for example, Japanese, English, and French.

[0890] "Means for displaying the conversation" refers to a technology that visually displays the translated conversation content using a display device such as a monitor or projector.

[0891] "Means that enable natural conversation" are technologies that allow people who speak different languages ​​to communicate smoothly without feeling a language barrier.

[0892] An "autonomous vehicle" is a vehicle that drives autonomously using artificial intelligence and sensor technology.

[0893] A "display device" is a device for visually displaying information, and includes displays, monitors, and the like.

[0894] "Mobile device" refers to a portable electronic device such as a smartphone or tablet.

[0895] "Real-time translated information" means information in which speech or text is instantly converted into another language and provided without delay.

[0896] "Audio playback" refers to a technique that uses speech synthesis technology to output the translated text as audio.

[0897] The following system configuration will be described as an embodiment of the present invention.

[0898] System Configuration

[0899] The system includes a server with generative AI translation and summarization functions, a display device that projects conversations translated into multiple languages ​​in a designated space (room), an audio playback device that enables natural conversation even when the language is not the same, and a mobile device that provides real-time translated information when passengers in an autonomous vehicle speak different languages.

[0900] Program processing

[0901] The server runs a program that integrates speech recognition, translation, and speech synthesis functions. Specifically, it uses the following hardware and software:

[0902] Hardware: microphones, speakers, displays (in-car displays and smartphones)

[0903] Software: Python, SpeechRecognition library, Google Translate API, pyttsx3 library

[0904] Voice Recognition

[0905] The server converts the audio data acquired from the microphone into text data using the SpeechRecognition library. This text data represents what the passenger said.

[0906] translation

[0907] The server uses the Google Translate API to translate the acquired text data into the target language, for example, translating Japanese text into English.

[0908] Speech synthesis

[0909] The server uses the pyttsx3 library to convert the translated text data into audio data and play it through the speaker.

[0910] Specific examples

[0911] For example, if a Japanese-speaking passenger asks, "Where is the next stop?", the system will translate the question into English and display "Where is the next stop?" on the in-car display while simultaneously playing it aloud.

[0912] Prompt Sentence Examples

[0913] Translate the Japanese audio into English, display it on the screen, and play it back aloud.

[0914] In this way, passengers who speak different languages ​​will be able to communicate seamlessly inside an autonomous vehicle.

[0915] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0916] Step 1:

[0917] The user speaks into the microphone. The input is the user's voice data. The server receives this voice data and converts it into text data using the SpeechRecognition library. The output is text data representing the user's speech.

[0918] Step 2:

[0919] The server sends the text data obtained in step 1 to the Google Translate API and translates it into the specified target language. The input is the text data and the target language information. The output is the translated text data.

[0920] Step 3:

[0921] The server converts the translated text data into audio data using the pyttsx3 library. The input is the translated text data. The output is the audio data.

[0922] Step 4:

[0923] The server plays the audio data generated in step 3 through the speaker. The input is the audio data. The output is the audio that the user can hear.

[0924] Step 5:

[0925] The server sends the translated text data to an in-car display or mobile device for display on the display device. The input is the translated text data. The output is the translated text displayed on the display.

[0926] In this way, what the user says is translated in real time and provided as both audio and text.

[0927] Example 3

[0928] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0929] It is difficult for people who speak different languages ​​to converse naturally in the same space. Language barriers often become an obstacle to communication, especially at home or in small gatherings. To solve this problem, real-time speech recognition and translation, as well as a visual display of the translation results, are required.

[0930] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0931] In this invention, the server includes means for capturing voice input, means for transmitting the captured voice data to the server, means for converting the voice data into text, means for translating the text into a specified target language, means for transmitting the translated text to the terminal, and means for displaying the translated text, thereby enabling people who speak different languages ​​to communicate naturally while visually checking translated conversations in real time.

[0932] A "means for capturing voice input" is a device or function for detecting a user's speech and collecting the voice data.

[0933] The "means for transmitting captured voice data to a server" refers to a device or function for transmitting collected voice data to a server via a communication means such as the Internet.

[0934] "Means for converting voice data into text" refers to a device or function for analyzing voice data and converting the content into text format.

[0935] A "means for translating text into a specified target language" is a device or function for converting text data into a different language.

[0936] The "means for transmitting the translated text to the terminal" refers to a device or function for transmitting the translated text data to the terminal via a communication means such as the Internet.

[0937] The "means for displaying translated text" refers to a device or function for visually displaying translated text data.

[0938] This invention is a system that allows people who speak different languages ​​to converse naturally in the same space. The system includes a set of means for capturing voice input, translating it in real time, and visually displaying the results.

[0939] Hardware and software used

[0940] Hardware:

[0941] Device: Smart speaker or smart device with microphone

[0942] Display devices: projectors and smart displays

[0943] software:

[0944] Speech Recognition API: Google Cloud Speech-to-Text API

[0945] Translation API: Google Cloud Translation API

[0946] Data processing and calculation

[0947] Capturing audio input:

[0948] When a user speaks, the device captures the audio. For example, a smart speaker installed in the living room detects the user's speech.

[0949] Sending audio data:

[0950] The captured audio data is transmitted to a server over the Internet.

[0951] Voice Recognition:

[0952] The server converts the audio data into text using the Google Cloud Speech-to-Text API. For example, if a user says, "How was your day?", the API converts this to the text "How was your day?"

[0953] translation:

[0954] The server translates the converted text into the specified target language using the Google Cloud Translation API. For example, if a translation from English to Japanese is required, the API will translate "How was your day?" into "How was your day today?"

[0955] Sending translation results:

[0956] The translated text is sent to the device via the internet.

[0957] Displaying translation results:

[0958] The device then projects the translation results onto a wall or screen using a projector or smart display. For example, the text "How was your day?" could be displayed on a living room wall.

[0959] Specific examples

[0960] As a concrete example, consider a conversation between an English-speaking father and a Japanese-speaking mother in the living room.

[0961] 1. Capture audio input:

[0962] The father says, "How was your day?"

[0963] The device captures this audio and sends it to the server.

[0964] 2. Speech Recognition:

[0965] The server uses the Google Cloud Speech-to-Text API to convert the audio data into text: "How was your day?"

[0966] 3. Translation:

[0967] The server uses the Google Cloud Translation API to translate "How was your day?" to "How was your day?"

[0968] 4. Sending translation results:

[0969] The server sends the translated text "How was your day?" to the device.

[0970] 5. Displaying the translation results:

[0971] The device projects the text "How was your day?" onto the living room wall.

[0972] Example prompt sentence:

[0973] User: "How was your day?"

[0974] Server: "How was your day?"

[0975] This system allows smooth communication between family members who speak different languages ​​through real-time translation. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0976] Step 1:

[0977] Capture audio input

[0978] When a user speaks, the device captures the voice. For example, a smart speaker installed in a living room detects the user's utterance, "How was your day?" The input is the user's voice, and the output is the captured voice data.

[0979] Step 2:

[0980] Sending audio data

[0981] The terminal transmits the captured audio data to a server via the Internet. The input is the captured audio data, and the output is the audio data transmitted to the server.

[0982] Step 3:

[0983] Voice Recognition

[0984] The server converts the voice data into text using the Google Cloud Speech-to-Text API. For example, the voice data "How was your day?" is converted into text "How was your day?". The input is the voice data sent to the server, and the output is the converted text data.

[0985] Step 4:

[0986] translation

[0987] The server translates the converted text into the specified target language using the Google Cloud Translation API. For example, the text "How was your day?" is translated into Japanese as "How was today?". The input is the converted text data, and the output is the translated text data.

[0988] Step 5:

[0989] Sending translation results

[0990] The server transmits the translated text to the terminal via the Internet. The input is the translated text data, and the output is the translated text data transmitted to the terminal.

[0991] Step 6:

[0992] Displaying translation results

[0993] The device then projects the translation results onto a wall or screen via a projector or smart display. For example, the text "How was your day?" is displayed on the wall of your living room. The input is the translated text data sent to the device, and the output is the translated text displayed on the wall or screen.

[0994] (Application example 3)

[0995] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0996] In environments where people speaking different languages ​​coexist, it is important to achieve natural communication without feeling the language barrier. In particular, smooth conversations between multinational customers and store staff are required in brick-and-mortar stores. Conventional translation systems often lack practicality due to limitations in real-time performance and display methods.

[0997] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0998] In this invention, the server includes a means having a generative AI translation and summarization function, a means for projecting conversations translated into multiple languages ​​in a specified space (room), a means for enabling natural conversation even when the language is not the same, and a means for translating conversations with customers who speak different languages ​​in real time using smart glasses and displaying the translation on a display, thereby enabling smooth real-time communication between customers who speak different languages ​​and store clerks.

[0999] "Generative AI" is a system that uses artificial intelligence to generate, translate, and summarize data.

[1000] The "translation and summary function" is a function that converts input language data into another language or summarizes its contents concisely.

[1001] "Designated space" refers to the specific room or area in which the system operates.

[1002] "Multiple languages" refers to two or more different languages ​​and indicates the types of languages ​​that the system supports.

[1003] "Means for displaying dialogue" refers to devices or technologies that visually display the translated dialogue.

[1004] "Means that enable natural conversation" refers to methods and techniques that enable smooth communication without making language differences noticeable.

[1005] "Smart glasses" are glasses-type devices that have built-in displays and sensors and have the ability to visually display information.

[1006] "Means for translating in real time and displaying the results on a display" refers to technology or devices for instantly translating conversations and displaying the results on a display.

[1007] A system for implementing this invention includes means for having generative AI translation and summarization functions, means for displaying conversations translated into multiple languages ​​in a specified space, means for enabling natural conversation even when the languages ​​are not the same, and means for translating conversations with customers who speak different languages ​​in real time using smart glasses and displaying the translation on a display.

[1008] The server uses generative AI to translate and summarize the voice data in real time. Specifically, it converts the voice into text using speech recognition software (e.g., Google Speech Recognition API), and then translates the text into the specified language using translation software (e.g., Google Translate API). The translated text is then displayed on the smart glasses' display.

[1009] The device (smart glasses) receives the translated text sent from the server and displays it on its screen. This allows the store clerk to understand the customer's language and respond appropriately. The smart glasses have a built-in display and sensors, and can display information visually.

[1010] The user (store clerk) wears the smart glasses and converses with customers. What the customer says is translated in real time and displayed on the smart glasses' display, enabling smooth communication without any language barriers.

[1011] As a concrete example, consider a scenario in which an English-speaking tourist visits a brick-and-mortar store in Japan and asks a store clerk for product details. The store clerk is wearing smart glasses, and the tourist's English question is translated into Japanese in real time and displayed on the glasses' display. The clerk then looks at the translation and provides an appropriate response in Japanese.

[1012] Examples of prompts for generative AI models include:

[1013] "An English-speaking tourist visits a brick-and-mortar store in Japan and asks a store clerk for product details. The clerk is wearing smart glasses, and the tourist's English question is translated into Japanese in real time and displayed on the glasses' display. The clerk then looks at the translation and provides an appropriate response in Japanese. Please generate a Python program to realize this scenario."

[1014] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1015] Step 1:

[1016] The server captures the user's (store clerk's) voice through a microphone. The input is voice data, which is converted into text data using speech recognition software (e.g., Google Speech Recognition API). The output is text data.

[1017] Step 2:

[1018] The server inputs the text data obtained in step 1 into translation software (e.g., Google Translate API) and translates it into the specified language. The input is text data, which is converted into another language using translation software. The output is the translated text data.

[1019] Step 3:

[1020] The server sends the translated text data to the smart glasses. The input is the translated text data, which is sent to the smart glasses via the network. The output is the text data displayed on the smart glasses.

[1021] Step 4:

[1022] The terminal (smart glasses) receives the translated text sent from the server and displays it on the display. The input is the translated text data sent from the server, which is displayed on the display. The output is the translated text displayed on the display of the smart glasses.

[1023] Step 5:

[1024] The user (store clerk) reads the translated text displayed on the smart glasses' display and responds appropriately to the customer. The input is the translated text displayed on the smart glasses' display, and the clerk responds to the customer based on this. The output is the appropriate response to the customer.

[1025] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1026] "Example 1"

[1027] One embodiment of the present invention is a system that combines the translation and summarization functions of generative AI with an emotion engine. This system translates and summarizes a user's speech in real time and projects the results on the wall of a specified space (room). Furthermore, the emotion engine recognizes emotions from the user's tone of voice, facial expression, and choice of words, and adjusts the translation and summarization results accordingly. For example, if it recognizes that the user is angry, it adjusts the translation results to appropriately express that anger. It also recognizes emotions such as joy and surprise and adjusts the translation results accordingly. This enables natural conversations that include emotions, even between users who do not share the same language.

[1028] "Example 2"

[1029] The emotion engine can also adjust the content and format of the display. For example, if it recognizes that the user is happy, it can change the color of the displayed text to a brighter color. Or, if it recognizes that the user is surprised, it can make the displayed text larger, providing visual feedback according to the emotion. This allows the user to confirm that their emotions are being recognized by the system and reflected appropriately.

[1030] "Example 3"

[1031] Furthermore, this system not only adjusts the translation and summarization results according to the user's emotions, but also suggests dialogues that correspond to those emotions. For example, if it recognizes that the user is confused, it will identify the parts of the conversation that may be causing that confusion and provide an explanation or commentary. Similarly, if it recognizes that the user is happy, it will suggest replies that share that joy. This allows users to enjoy natural dialogue based on an understanding of their own emotions.

[1032] The processing flow of each embodiment will be described below.

[1033] "Example 1"

[1034] Step 1: The user starts a conversation, which is input to the system via an input device such as a microphone.

[1035] Step 2: Generative AI translates and summarizes the user's speech in real time. This translation and summary translates the user's speech into another language and extracts the main points.

[1036] Step 3: The emotion engine recognizes the user's emotions from their tone of voice, facial expressions, and choice of words.

[1037] Step 4: The generative AI adjusts the translation and summarization results based on the emotions recognized by the emotion engine.

[1038] Step 5: The adjusted translation and summary results are projected onto the wall or other surface of the designated space (room).

[1039] "Example 2"

[1040] Step 1: The user starts a conversation, which is input to the system via an input device such as a microphone.

[1041] Step 2: Generative AI translates and summarizes the user's speech in real time. This translation and summary translates the user's speech into another language and extracts the main points.

[1042] Step 3: The emotion engine recognizes the user's emotions from their tone of voice, facial expressions, and choice of words.

[1043] Step 4: The emotion engine adjusts the content and format of the display depending on the emotion it recognizes. For example, if it recognizes that the user is happy, it changes the color of the displayed text to a brighter color.

[1044] Step 5: The translation and summary results are projected onto the wall of a designated space (room) in the adjusted content and format.

[1045] "Example 3"

[1046] Step 1: The user starts a conversation, which is input to the system via an input device such as a microphone.

[1047] Step 2: Generative AI translates and summarizes the user's speech in real time. This translation and summary translates the user's speech into another language and extracts the main points.

[1048] Step 3: The emotion engine recognizes the user's emotions from their tone of voice, facial expressions, and choice of words.

[1049] Step 4: The generative AI suggests dialogue based on the emotion recognized by the emotion engine. For example, if it recognizes that the user is confused, it will identify parts of the conversation that may be causing that confusion and provide an explanation or commentary.

[1050] Step 5: Project the proposed dialogue onto a wall or other surface in a designated space (room).

[1051] Example 1

[1052] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1053] Conventional translation systems have had problems with natural conversations between users who speak different languages, especially with the difficulty of conveying emotional nuances. Real-time translation and summarization are also inadequate, often disrupting the flow of conversation. Furthermore, there are limited ways to properly display translation results, making them less convenient for users.

[1054] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means having a generative AI translation and summarization function, a means for displaying conversations translated into multiple languages ​​in a specified space, a means for enabling natural conversation even when the languages ​​are not the same, and a means for recognizing user emotions and adjusting the results of translation and summarization according to those emotions. This enables natural conversation, including emotions, even between users who speak different languages, realizes real-time translation and summarization, and allows the translation results to be displayed appropriately on any surface.

[1055] "Generative AI" is a system that uses artificial intelligence technology to generate, translate, and summarize data.

[1056] The "translation and summarization function" converts input text and audio data into other languages ​​and, if necessary, summarizes the content concisely.

[1057] A "designated space" refers to a specific location or area, such as a conference room or living room.

[1058] "Multiple languages" refers to two or more different languages, such as Japanese, English, and French.

[1059] "Conversation projection" refers to a method of displaying translated text on a surface such as a wall or screen using a projector or display.

[1060] "Means that enable natural conversation" refers to methods that facilitate communication between users who speak different languages ​​and enable natural conversation that includes emotions and nuances.

[1061] "Means for recognizing emotions" refers to a method of analyzing emotions from the user's tone of voice, facial expressions, and choice of words, and then providing appropriate translations and summaries based on that information.

[1062] "Adjustment measures" are methods for modifying translation and summarization results based on recognized emotions to make them more appropriate.

[1063] This invention is a system that combines the translation and summarization functions of generative AI with an emotion recognition engine, translating conversations taking place within a specified space into multiple languages ​​in real time and adjusting the translation results by recognizing emotions. Specific embodiments of this system are described below.

[1064] Hardware and software used

[1065] Hardware: high-performance microphones, cameras, projectors, servers

[1066] Software: Generative AI models (e.g., OpenAI's GPT-4), emotion recognition engines (e.g., Microsoft Azure's Emotion API)

[1067] System configuration

[1068] 1. A high-performance microphone installed on the device captures conversations taking place within a designated space in real time.

[1069] 2. A camera installed on the device captures the user's facial expressions.

[1070] 3. The device sends the captured voice and facial expression data to the server via the network.

[1071] 4. The server inputs the received voice data into the generative AI model and converts it into text data.

[1072] 5. At the same time, the server uses an emotion recognition engine to analyze the user's emotions from the facial expression data and voice data.

[1073] 6. The server uses the generative AI model to translate the text data into multiple languages.

[1074] 7. The server adjusts the translation results based on the emotional data analyzed by the emotion recognition engine.

[1075] 8. The server sends the adjusted translation results to the device.

[1076] 9. The device uses a projector to project the translation results onto any surface, such as a wall or screen.

[1077] Specific examples

[1078] For example, suppose there is a Japanese-speaking user A and an English-speaking user B in a conference room. If user A says, "This project is very important," the system processes it as follows:

[1079] 1. The device's microphone captures what User A says.

[1080] 2. The device's camera captures User A's facial expression.

[1081] 3. The device sends the voice data and facial expression data to the server.

[1082] 4. The server converts the voice data into text, and the emotion recognition engine analyzes User A's emotions (e.g., serious expression).

[1083] 5. The server translates the text data into English using a generative AI model.

[1084] 6. Based on the results of the emotion recognition engine, the server adjusts the translation result to "This project is extremely important."

[1085] 7. The server sends the adjusted translation results to the device.

[1086] 8. The terminal uses a projector to project the translation results onto the wall of the conference room.

[1087] Prompt Sentence Examples

[1088] Examples of prompts to be input to a generative AI model include:

[1089] User says: "This project is very important"

[1090] User Sentiment: "Serious"

[1091] Target language: "English"

[1092] By inputting this prompt sentence into a generative AI model, appropriate translation results can be obtained.

[1093] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1094] Step 1:

[1095] A high-performance microphone installed on the device captures conversations taking place within a designated space in real time. The input is a voice signal, and the output is digital voice data. Specifically, the microphone converts the voice signal into digital data.

[1096] Step 2:

[1097] A camera installed on the device captures the user's facial expression. The input is the user's facial expression, and the output is image data. Specifically, the camera generates image data.

[1098] Step 3:

[1099] The device transmits the captured voice and facial expression data to a server via a network. The input is digital voice data and image data, and the output is data transmission to the server. Specifically, the device divides the data into packets and transmits them to the server using the TCP / IP protocol.

[1100] Step 4:

[1101] The server inputs the received voice data into a generative AI model and converts it into text data. The input is digital voice data, and the output is text data. Specifically, the server calls a voice recognition API and converts the voice data into text.

[1102] Step 5:

[1103] At the same time, the server uses an emotion recognition engine to analyze the user's emotions from the facial expression data and voice data. The input is image data and digital voice data, and the output is emotion data. Specifically, the server calls the emotion recognition API and analyzes the facial expression data and voice data.

[1104] Step 6:

[1105] The server uses a generative AI model to translate text data into multiple languages. The input is text data, and the output is translated text data. Specifically, the server calls a translation API to translate the text data into the specified language.

[1106] Step 7:

[1107] The server adjusts the translation results based on the emotional data analyzed by the emotion recognition engine. The input is the translated text data and the emotional data, and the output is the adjusted translated text data. Specifically, the server analyzes the emotional data and applies rules to adjust the tone and expression of the translation results.

[1108] Step 8:

[1109] The server sends the adjusted translation results to the terminal. The input is the adjusted translated text data, and the output is the data sent to the terminal. Specifically, the server divides the data into packets and sends them to the terminal using the TCP / IP protocol.

[1110] Step 9:

[1111] The terminal uses a projector to project the translation results onto any surface, such as a wall or screen. The input is the adjusted translation text data, and the output is the displayed translation result. Specifically, the terminal sends the received translation result to the projector and displays it at the specified location.

[1112] (Application example 1)

[1113] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1114] In a multilingual environment, it is difficult to achieve natural conversations between users who speak different languages. Furthermore, simple translation alone may not convey the nuances of emotions, which can lead to misunderstandings. In brick-and-mortar stores in particular, smooth communication between customers and store clerks is required, so real-time translation and emotional recognition and adjustment are necessary.

[1115] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1116] In this invention, the server includes means for translating and summarizing using generative AI, means for displaying conversations translated into multiple languages ​​in a specified space (room), means for enabling natural conversation even when the languages ​​are not the same, means for recognizing the user's emotions using an emotion recognition engine and adjusting the translation results, and means for displaying the translation results on a smart device, thereby enabling natural conversations that include emotions even between users who speak different languages.

[1117] "Generative AI" refers to artificial intelligence that uses natural language processing to generate, translate, summarize, and otherwise process text.

[1118] The "translation and summary function" is a function that translates input text into another language or summarizes its contents concisely.

[1119] A "designated space" is a specific location or room where the conversation takes place.

[1120] "Multilingual" means two or more different languages.

[1121] "Means for projecting the dialogue" refers to a method or device for projecting the translated text onto a display device such as a wall or screen.

[1122] An "emotion recognition engine" is software or hardware that analyzes and recognizes emotions from a user's tone of voice, facial expressions, and choice of words.

[1123] A "smart device" is a portable electronic device with internet connectivity and advanced computing capabilities.

[1124] "Means for adjusting the translation result" refers to a method or device that appropriately changes the content or expression of the translated text based on the recognized sentiment.

[1125] The system for implementing this invention is configured by combining the translation and summarization functions of generative AI, an emotion recognition engine, and a smart device. A specific embodiment of this system is shown below.

[1126] System configuration

[1127] 1. Generative AI translation and summarization functions

[1128] The server translates and summarizes the user's speech in real time using a generative AI model, such as GPT-4.

[1129] 2. Emotion Recognition Engine

[1130] The server uses an emotion recognition engine, such as Affectiva, to recognize the user's emotions from their tone of voice, facial expressions, and language.

[1131] 3. Smart Devices

[1132] The terminal uses smart devices such as smart glasses or head-mounted displays to display the translation results, allowing store clerks and users to check the translation results in real time. Specific devices that can be used include Google Glass and Microsoft HoloLens.

[1133] Data processing and calculation

[1134] 1. Acquiring voice input

[1135] When the user speaks, the voice data is acquired by the terminal.

[1136] 2. Converting voice data to text

[1137] The acquired voice data is sent to a server and converted into text data by voice recognition software.

[1138] 3. Emotion recognition

[1139] The text data is analyzed by an emotion recognition engine to recognize the user's emotions.

[1140] 4. Translation and Summary

[1141] The generative AI model translates text data into a specified language and summarizes it as needed.

[1142] 5. Adjustment of translation results

[1143] Based on the recognized emotion, the translation result is adjusted appropriately: for example, if the user is angry, the translation result will reflect that emotion.

[1144] 6. Displaying the translation results

[1145] The adjusted translation results are displayed on the smart device, allowing store clerks and users to check the translation results in real time.

[1146] Specific examples

[1147] For example, if a customer says, "I am very happy with the service!", it will be processed as follows:

[1148] 1. Audio data is acquired by the device.

[1149] 2. The audio data is converted into text data.

[1150] 3. The emotion recognition engine recognizes the emotion "happy."

[1151] 4. The generative AI model translates this as "I'm very satisfied with your service!"

[1152] 5. Based on the perceived emotion, it is adjusted to "(with joy) I am very satisfied with the service!"

[1153] 6. The adjusted translation results will be displayed on your smart device.

[1154] Prompt Sentence Examples

[1155] "You will create a program that translates customer utterances in real time, recognizes emotions, and adjusts the translation results accordingly. The emotion recognition engine used is Affectiva, and the translation API is Google Translate API. The translation results will be displayed on the smart glasses. Use the following as an example of a customer utterance: 'I am very happy with the service!'"

[1156] In this way, the present invention can realize natural conversations that include emotions even between users who speak different languages.

[1157] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1158] Step 1:

[1159] The user speaks.

[1160] The user utters, "I am very happy with the service!" This voice data is acquired by the terminal. The input is the user's voice data, and the output is the voice data sent to the terminal.

[1161] Step 2:

[1162] The terminal transmits the voice data to the server.

[1163] The terminal transmits the acquired voice data to the server. The input is the voice data acquired by the terminal, and the output is the voice data transmitted to the server.

[1164] Step 3:

[1165] The server converts the voice data into text data.

[1166] The server uses speech recognition software to convert the voice data into text data: the input is the voice data sent to the server, and the output is text data.

[1167] Step 4:

[1168] The server sends the text data to the emotion recognition engine.

[1169] The server sends the text data to the emotion recognition engine to recognize the user's emotion. The input is the text data, and the output is the recognized emotion data.

[1170] Step 5:

[1171] The server translates the text data using a generative AI model.

[1172] The server uses a generative AI model to translate text data into a specified language. The input is text data, and the output is the translated text data.

[1173] Step 6:

[1174] The server adjusts the translation results based on the emotion data.

[1175] The server adjusts the translation result appropriately based on the recognized emotion data. For example, if the emotion "happy" is recognized, it adds the phrase "(with joy)" to the translation result. The input is the translated text data and emotion data, and the output is the adjusted translation result.

[1176] Step 7:

[1177] The server sends the adjusted translation results to the device.

[1178] The server sends the adjusted translation result to the terminal. The input is the adjusted translation result, and the output is the translation result sent to the terminal.

[1179] Step 8:

[1180] The terminal displays the translation results on the smart device.

[1181] The terminal displays the received translation results on the smart device. The input is the translation result sent to the terminal, and the output is the translation result displayed on the smart device.

[1182] In this way, the user's speech is translated in real time, enabling natural conversation that includes emotions.

[1183] Example 2

[1184] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1185] In situations such as international conferences, when participants speak different languages, language barriers often become an obstacle to communication. It is also difficult to understand the content of the conference and exchange opinions in real time. Furthermore, the quality of communication can be reduced because visual feedback that properly reflects participants' emotions is not provided.

[1186] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means having a translation and summarization function of generative artificial intelligence, means for displaying conversations translated into multiple languages ​​in a specified space, means for enabling natural conversation even when the languages ​​are not the same, means for collecting voice data, means for converting the collected voice data into text data, means for recognizing emotions in the text data, and means for adjusting the content and format of the display based on the emotion recognition results. This enables participants who speak different languages ​​to understand the conversation in real time and receive visual feedback according to their emotions.

[1187] "Generative AI" is an AI that has the ability to analyze voice and text data and translate and summarize.

[1188] The "translation and summary function" is a function that converts input language data into another language or summarizes its contents concisely.

[1189] A "designated space" refers to a specific location such as a conference room or hall.

[1190] "Multiple languages" refers to different languages, including, for example, Japanese, English, and French.

[1191] "Means for projecting the dialogue" refers to a device or method for visually displaying the translated text using a projector, display, or the like.

[1192] "Means for collecting voice data" refers to a method for acquiring speech as digital voice data using a microphone or voice input device.

[1193] "Means for converting into text data" refers to a method of converting voice data into text information using voice recognition software.

[1194] "Emotion recognition means" refers to software or algorithms that analyze and identify a user's emotions from text data or voice data.

[1195] "Means for adjusting the content and format of the display" refers to a method for changing visual attributes such as the color and size of the displayed text based on the results of emotion recognition.

[1196] This invention is a system that enables participants who speak different languages ​​to understand conversations in real time and receive visual feedback according to their emotions in situations such as international conferences. This system includes means for having a generative artificial intelligence translation and summarization function, means for displaying conversations translated into multiple languages ​​in a specified space, means for enabling natural conversation even when the languages ​​are not the same, means for collecting voice data, means for converting the collected voice data into text data, means for recognizing emotions in the text data, and means for adjusting the content and format of the display based on the results of the emotion recognition.

[1197] Hardware and software used

[1198] Hardware:

[1199] High-performance servers (e.g., high-performance servers)

[1200] Projectors (e.g. high-resolution projectors)

[1201] Microphone (e.g. high-sensitivity microphone)

[1202] software:

[1203] Speech recognition software (e.g., speech recognition API)

[1204] Translation software (e.g. translation API)

[1205] Emotion recognition software (e.g., emotion recognition API)

[1206] Program processing explanation

[1207] The server translates the speech of participants into other languages ​​in real time and projects it onto the wall. It also has an emotion engine that adjusts the content and format of the display according to the user's emotions.

[1208] The device uses a microphone to collect speech from meeting participants and sends the audio data to a server. The server uses speech recognition software to convert the audio data into text data. The converted text data is then input into translation software and translated into the specified language. The translated text data is then input into emotion recognition software to analyze the user's emotions. The color and size of the displayed text are adjusted based on the results of the emotion recognition. The adjusted text data is then sent to a projector and projected onto the wall.

[1209] Specific examples

[1210] As a specific use case, consider the following scenario:

[1211] Scenario: At an international conference, there is participant A who speaks Japanese and participant B who speaks English.

[1212] Participant A says, "Hello everyone."

[1213] The device collects this speech with a microphone and transmits the audio data to the server.

[1214] The server uses voice recognition software to convert "Hello everyone" into text.

[1215] The server translates this to "Hello, everyone" using its translation software.

[1216] The server uses emotion recognition software to analyze the emotion in "Hello everyone" and detect joy.

[1217] The server changes the color of the text to a brighter color and sends it to the projector.

[1218] The terminal projects "Hello, everyone" onto the wall via a projector.

[1219] Prompt Sentence Examples

[1220] Write a program that translates the Japanese phrase "Hello everyone" into English, performs emotion recognition, and if joy is detected, changes the color of the text to a brighter color.

[1221] In this way, users can be sure that their emotions are recognized and reflected appropriately by the system.

[1222] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1223] Step 1: Collecting audio data

[1224] The device collects speech from participants using a microphone, and the collected voice data is sent to the server in real time.

[1225] Input: Meeting participant's remarks

[1226] Output: Digital audio data

[1227] How it works: A high-sensitivity microphone connected to the device captures what participants say and sends the audio data digitally to a server.

[1228] Step 2: Voice Recognition

[1229] The server inputs the received voice data into voice recognition software and converts the voice data into text data.

[1230] Input: Digital audio data

[1231] Output: Text data

[1232] Specific operation: The server sends the voice data to a speech recognition API and converts the voice "Hello everyone" into text "Hello everyone."

[1233] Step 3: Translation

[1234] The server inputs the converted text data into translation software and translates it into the specified language.

[1235] Input: Text data

[1236] Output: Translated text data

[1237] Specific operation: The server sends the text data "Hello, everyone" to the translation API, which translates it to "Hello, everyone."

[1238] Step 4: Emotion Recognition

[1239] The server inputs the translated text data into emotion recognition software to analyze the user's emotions.

[1240] Input: Translated text data

[1241] Output: Emotion recognition result

[1242] Specific operation: The server sends the text data "Hello everyone" to the emotion recognition API and detects the emotion of joy.

[1243] Step 5: Adjust the visual feedback

[1244] The server adjusts the color and size of the displayed text based on the results of emotion recognition.

[1245] Input: Emotion recognition results

[1246] Output: Adjusted text data

[1247] Specific operation: Based on the emotion recognition result, the server generates an instruction to change the color of the text to yellow.

[1248] Step 6: Projection

[1249] The terminal sends the adjusted text data to the projector, which projects it onto the wall.

[1250] Input: Adjusted text data

[1251] Output: Text projected on the wall

[1252] Specific operation: The terminal sends the adjusted text data "Hello, everyone" received from the server to the projector, projecting it in yellow letters on the wall.

[1253] In this way, users can be sure that their emotions are recognized and reflected appropriately by the system.

[1254] (Application example 2)

[1255] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1256] In multilingual environments such as international conferences and virtual stores, there is a need to achieve smooth communication across language barriers. Another challenge is to communicate information more intuitively and effectively by providing visual feedback that reflects the user's emotions. In particular, in virtual stores, it is important to provide appropriate information about products that users are interested in, thereby motivating them to purchase.

[1257] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1258] In this invention, the server includes a means for translating and summarizing using generative AI, a means for displaying conversations translated into multiple languages ​​in a specified space, a means for enabling natural conversation even when the language is not the same, a means for recognizing a user's emotions and adjusting the content and format of the display, and a means for displaying product information in a virtual store using smart glasses. This enables real-time translation and visual feedback according to emotions in a multilingual environment, making it easier for users to intuitively understand information. Furthermore, in the virtual store, providing appropriate information for products in which the user is interested can increase purchasing motivation.

[1259] "Generative AI" is a system that uses artificial intelligence to generate and convert data.

[1260] "Translation and summary function" refers to a function that translates input text or audio into another language or summarizes the content concisely.

[1261] A "designated space" refers to a specific location or area, including conference rooms and virtual stores.

[1262] A "conversation translated into multiple languages" is a conversation between multiple users who speak different languages, translated into each of the languages.

[1263] "Displaying means" refers to devices or techniques used to visually display translated text or information.

[1264] "Means that enable natural conversation" are techniques and methods that enable smooth communication beyond language differences.

[1265] "Means for recognizing the user's emotions and adjusting the content and format displayed" refers to technology that detects the user's emotions and changes the color and size of the displayed information accordingly.

[1266] "Smart glasses" are glasses-type devices that have the function of displaying information.

[1267] A "virtual store" is a virtual store that operates on the Internet, where users can browse and purchase products online.

[1268] "Product information" refers to information such as product details, features, and prices.

[1269] A system for implementing this invention includes means for having generative AI translation and summarization functions, means for displaying conversations translated into multiple languages ​​in a specified space, means for enabling natural conversation even when the languages ​​are not the same, means for recognizing the user's emotions and adjusting the content and format of the display, and means for displaying product information in a virtual store using smart glasses.

[1270] The server runs programs for speech recognition, translation, emotion recognition, and visual feedback. Specifically, speech recognition uses a microphone and Google's speech recognition API. Translation uses Google Translate API and emotion recognition uses Emotion Recognition API. Visual feedback uses OpenCV and displays information on smart glasses.

[1271] As a user wears the smart glasses and walks around the virtual store, descriptions of each product are translated into the user's language in real time. The display also changes depending on the user's emotions, providing feedback such as enlarging detailed information about products that interest the user.

[1272] For example, if a user says "I'd like to know more about this product" in a virtual store, this statement is translated into English in real time, and if the user is recognized as interested, the translated text is displayed in a brighter color and larger.

[1273] Examples of prompts to input to a generative AI model include:

[1274] When a user says "I want to know more about this product," create a program that translates this statement into English in real time and, if it recognizes that the user is interested, displays the translated text in a larger, brighter color.

[1275] In this way, the user can easily understand the information intuitively, and in the virtual store, appropriate information can be provided for the product in which the user is interested, thereby increasing the user's desire to purchase.

[1276] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1277] Step 1:

[1278] The server captures the user's speech through a microphone. The input is the user's voice, and the output is voice data. This voice data is converted into text using Google's speech recognition API. Specifically, the server acquires voice data from the microphone, sends it to the speech recognition API, and receives text data.

[1279] Step 2:

[1280] The server sends the text data obtained from the speech recognition API to the Google Translate API and translates it into the specified language. The input is the text data from the speech recognition API, and the output is the translated text data. Specifically, the server sends the text data to the translation API and receives the translation results.

[1281] Step 3:

[1282] The server captures the user's face through the camera and recognizes the user's emotions using the Emotion Recognition API. The input is image data from the camera, and the output is recognized emotion data. Specifically, the server acquires image data from the camera, sends it to the emotion recognition API, and receives the emotion data.

[1283] Step 4:

[1284] The server adjusts the display content and format based on the translated text data and the recognized emotion data. The input is the translated text data and emotion data, and the output is the adjusted display data. Specifically, the server changes the color and size of the text according to the emotion data.

[1285] Step 5:

[1286] The server sends the adjusted display data to the smart glasses and displays it to the user. The input is the adjusted display data, and the output is the information displayed on the smart glasses. In particular, the server sends the display data to the smart glasses, and the user visually confirms it.

[1287] Step 6:

[1288] If the user walks around the virtual store and requests more information about a product they are interested in, the server again performs voice recognition, translation, and emotion recognition and displays the appropriate information. The input is the user's new voice data, and the output is updated display data. Specifically, the server again captures the voice data and repeats steps 1 to 5 described above.

[1289] Example 3

[1290] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1291] When a family member who speaks different languages ​​lives together, the language barrier can become an obstacle to communication. Furthermore, communication can be hindered because appropriate dialogue is not conducted in response to the user's emotions. There is a need to solve these problems and realize natural conversations within the home.

[1292] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1293] In this invention, the server includes means for translating and summarizing using generative AI, means for displaying conversations translated into multiple languages ​​in a specified space, means for enabling natural conversation even when the languages ​​are not the same, means for recognizing the user's emotions and adjusting the translation and summarization results according to those emotions, and means for suggesting dialogue according to the user's emotions.This allows family members who speak different languages ​​to enjoy natural conversation and also allows them to receive dialogue suggestions according to their emotions.

[1294] "Generative AI" refers to technology that uses artificial intelligence to generate, translate, and summarize data.

[1295] "Translation and summary function" refers to the function of converting input text or audio into another language or summarizing the content concisely.

[1296] "Designated space" refers to the specific location or room where the system is installed.

[1297] "Multilingually translated conversation" refers to conversation content that has been translated into different languages.

[1298] "Means of projecting" refers to means of displaying information using a projector, display, etc.

[1299] "Means that enable natural conversation" refers to techniques and methods that enable smooth communication beyond language differences.

[1300] "Means for recognizing user emotions" refers to technology that uses cameras and sensors to analyze the user's facial expressions and tone of voice to determine their emotions.

[1301] "Means for adjusting translation and summary results according to emotions" refers to technology that appropriately changes the content of translations and summaries based on the user's emotional state.

[1302] "Means for suggesting dialogue" refers to technology that generates appropriate dialogue content and responses based on the user's emotions.

[1303] The present invention is a system that allows family members who speak different languages ​​to enjoy natural conversation, and also suggests dialogues according to the user's emotions. A specific embodiment of this system is described below.

[1304] Hardware and software used

[1305] Hardware: high-performance microphones, speakers, projectors, cameras, and servers

[1306] Software: speech recognition software (e.g., Google Speech-to-Text), translation software (e.g., Google Translate API), emotion recognition software (e.g., Microsoft Azure Emotion API), generative AI models (e.g., OpenAI GPT-4)

[1307] System Operation

[1308] 1. Acquiring voice input:

[1309] The server captures the user's voice in real time through a high-performance microphone.

[1310] Example: When family members start talking in the living room, a microphone picks up their voice and sends it to a server.

[1311] 2. Speech Recognition:

[1312] The server uses speech recognition software to convert the captured speech into text.

[1313] Example: Speech "Hello, how was your day?" is converted to text, resulting in the string "Hello, how was your day?"

[1314] 3. Translation:

[1315] The server uses translation software to translate the text into different languages.

[1316] Example: The Japanese text "Hello, how was your day?" is translated into English to produce the text "Hello, how was your day?"

[1317] 4. Emotion recognition:

[1318] The server captures the user's facial expressions with a camera and analyzes the user's emotions using emotion recognition software.

[1319] Example: A camera takes a picture of a user's face and sends the image data to a server, which analyzes the image and determines that the user is confused.

[1320] 5. Translation and summary adjustment:

[1321] The server adjusts the translation and summarization results according to the user's emotions.

[1322] Example: Changing translations to more concise and understandable language for confused users.

[1323] 6. Dialogue Proposal:

[1324] The server uses the generated AI model to suggest dialogues based on the user's emotions.

[1325] Example: For a confused user, identifying the areas that are causing the conversation and generating suggestions like, "I think this part is confusing. Would you like me to explain it more?"

[1326] 7. Displaying the results:

[1327] The terminal (projector) projects the translation results and dialogue suggestions onto the wall.

[1328] Example: The translation "Hello, how was your day?" appears on the wall, along with a suggestion: "This part may be difficult to understand. Would you like me to explain further?"

[1329] Prompt Sentence Examples

[1330] "If the user is confused, identify the part of the conversation that is causing the confusion and provide an explanation or commentary."

[1331] "If the user is happy, suggest a response that shares that joy."

[1332] This system allows family members who speak different languages ​​to enjoy natural conversation and also allows them to receive suggestions for dialogue according to their emotions.The flow of the identification process in the third embodiment will be described with reference to FIG.

[1333] Step 1:

[1334] Acquiring voice input

[1335] The server captures the user's voice in real time through a high-performance microphone.

[1336] Input: User's voice

[1337] Output: Audio data

[1338] How it works: When family members start talking in the living room, the microphone picks up their voice and sends it to the server.

[1339] Step 2:

[1340] Voice Recognition

[1341] The server uses speech recognition software to convert the captured speech into text.

[1342] Input: Audio data

[1343] Output: Text data

[1344] What it does: The speech "Hello, how was your day?" is converted to text, resulting in the string "Hello, how was your day?"

[1345] Step 3:

[1346] translation

[1347] The server uses translation software to translate the text into different languages.

[1348] Input: Text data

[1349] Output: Translated text data

[1350] Specific behavior: The Japanese text "Hello, how was your day?" is translated into English to produce the text "Hello, how was your day?"

[1351] Step 4:

[1352] emotion recognition

[1353] The server captures the user's facial expressions with a camera and analyzes the user's emotions using emotion recognition software.

[1354] Input: User facial expression image data

[1355] Output: Emotion data (e.g., confusion, joy, etc.)

[1356] What it does: The camera takes a picture of the user's face and sends the image data to the server, which analyzes the image and determines that the user is confused.

[1357] Step 5:

[1358] Coordination of translation and summarization

[1359] The server adjusts the translation and summarization results according to the user's emotions.

[1360] Input: Translated text data, emotion data

[1361] Output: Adjusted translation and summary text data

[1362] What it does: It makes the translation more concise and easier to understand for confused users.

[1363] Step 6:

[1364] Dialogue proposal

[1365] The server uses the generated AI model to suggest dialogues based on the user's emotions.

[1366] Input: Emotion data, conversation text data

[1367] Output: Text data of dialogue suggestions

[1368] Specific Behavior: Identify areas of conversation that are causing confusion for confused users and generate suggestions such as, "This part seems confusing. Would you like me to explain it further?"

[1369] Step 7:

[1370] Displaying the results

[1371] The terminal (projector) projects the translation results and dialogue suggestions onto the wall.

[1372] Input: Adjusted translation and summary text data, dialogue suggestion text data

[1373] Output: Text projected on the wall

[1374] What it does: The translated result "Hello, how was your day?" appears on the wall, along with a suggestion: "This part might be difficult to understand. Would you like me to explain further?"

[1375] (Application example 3)

[1376] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1377] In an environment where people who speak different languages ​​coexist, it is necessary to realize natural communication without feeling a language barrier. It is also important to understand the emotions of the other person when communicating and respond appropriately accordingly. Conventional systems have problems with poor real-time translation and emotion recognition, which reduces the quality of communication.

[1378] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes means having a generative AI translation and summarization function, means for displaying conversations translated into multiple languages ​​in a specified space, means for enabling natural conversation even when the languages ​​are not the same, means for recognizing emotions, and means for suggesting dialogue based on the recognized emotions. This enables people who speak different languages ​​to communicate naturally through conversations translated in real time and to respond appropriately according to the other person's emotions.

[1379] "Generative AI" is a system that uses artificial intelligence technology to generate, translate, and summarize data.

[1380] The "translation and summary function" is a function that converts input language data into another language or summarizes its contents concisely.

[1381] "Designated space" refers to the specific location or room where the system is installed.

[1382] "Multilingually translated conversation" refers to conversation content that is translated into different languages ​​in real time.

[1383] "Projection means" refers to any device or technology that allows the translated dialogue to be displayed on a wall, screen, or any other surface.

[1384] "Means that enable natural conversation" refers to techniques and methods that enable smooth communication without making language differences noticeable.

[1385] "Means for recognizing emotions" refers to technology or devices that analyze a user's facial expressions, tone of voice, etc. to identify emotions.

[1386] "Means for making dialogue suggestions" refers to techniques and methods for suggesting appropriate dialogue content and response methods based on recognized emotions.

[1387] A system for implementing this invention includes a generative AI translation and summarization function, a means for displaying conversations translated into multiple languages ​​in a specified space, a means for enabling natural conversation even when the languages ​​are not the same, a means for recognizing emotions, and a means for suggesting dialogue based on the recognized emotions.

[1388] The server provides generative AI translation and summarization functions. Specifically, it uses the Google Translate API to translate input language data into other languages ​​in real time and summarizes the content concisely. This enables smooth communication between users who speak different languages.

[1389] The device provides a means to project translated conversations into multiple languages ​​in a designated space. For example, smart glasses or a head-mounted display can be used to display the translated conversations in the user's field of vision. This allows the user to continue the conversation naturally without being aware of language differences.

[1390] The Emotion Recognition API is used to recognize emotions. The device's camera captures the user's facial expressions and tone of voice, which are then analyzed to identify their emotions. The recognized emotions are sent to the server and used as data to make dialogue suggestions.

[1391] The server provides a means to suggest dialogue based on the perceived emotions. For example, if the user is perceived as confused, the server can identify parts of the conversation that may be causing this confusion and provide explanations or commentary. If the user is perceived as happy, the server can suggest replies that share that joy.

[1392] As a concrete example, consider the following scenario: When a foreign tourist asks a question about a product in a brick-and-mortar store in Japan, the store clerk will translate and display the customer's language in real time through the smart glasses. Furthermore, the clerk will recognize the customer's emotions by analyzing their facial expressions and tone of voice, and suggest appropriate dialogue. For example, a prompt message will be displayed saying, "If the customer is confused, please provide a detailed explanation."

[1393] Using this system, people who speak different languages ​​can communicate naturally through real-time translated conversations and respond appropriately based on the other person's emotions.

[1394] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1395] Step 1:

[1396] A user puts on a device (smart glasses or a head-mounted display) and starts a conversation. The device's camera captures the user's face, and the microphone collects audio. The input is the user's video and audio data. The output is the captured video and audio data.

[1397] Step 2:

[1398] The device sends the captured voice data to the server, which then translates the voice data in real time using the Google Translate API. The input is the voice data, and the output is the translated text data. Specifically, the voice data is converted into text, and the text is translated into the specified language.

[1399] Step 3:

[1400] The server sends the translated text data to the terminal, which then displays the translated text in the user's field of view. The input is the translated text data, and the output is the translated text displayed in the user's field of view. Specifically, the text is displayed on the display of the smart glasses.

[1401] Step 4:

[1402] The device camera captures the user's facial expressions and recognizes their emotions using the Emotion Recognition API. The input is the user's video data, and the output is the recognized emotion data. Specifically, the video data is analyzed to identify the emotion from the user's facial expressions.

[1403] Step 5:

[1404] The device sends the recognized emotion data to the server, which then generates appropriate dialogue suggestions based on the emotion data. The input is emotion data, and the output is dialogue suggestions. Specifically, the server analyzes the emotion data and makes suggestions such as providing detailed explanations if the user is confused.

[1405] Step 6:

[1406] The server sends the generated dialogue suggestions to the device, which then displays the suggestions in the user's field of view. The input is the dialogue suggestions, and the output is the suggestions displayed in the user's field of view. Specifically, the suggestions are displayed on the display of the smart glasses.

[1407] Step 7:

[1408] The user then carries out the next dialogue based on the displayed suggestions. The input is the user's new voice data, and the output is new voice data to return to the next step 1. Specifically, the user continues the conversation according to the suggestions.

[1409] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1410] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1411] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are listed.

[1412] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1413] [Third embodiment]

[1414] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1415] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1416] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1417] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1418] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1419] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1420] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1421] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1422] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1423] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1424] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1425] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1426] "Example 1"

[1427] One embodiment of the present invention provides a system with generative AI translation and summarization capabilities. This system is installed in a designated space (e.g., a conference room or living room) and translates conversations taking place within that space into multiple languages ​​in real time. The translation results are projected onto any surface, such as a wall or screen. This allows participants who do not share the same language to converse naturally.

[1428] "Example 2"

[1429] As a specific example of its use, consider an international conference. Even if participants speak different languages, this system can translate each participant's remarks into other languages ​​in real time and project them onto the wall. This allows all participants to understand the content of the conference and exchange opinions freely.

[1430] "Example 3"

[1431] This system is also suitable for use at home. For example, if a family member who speaks different languages ​​lives together, installing this system in the living room will facilitate smooth communication between family members. Family conversations are translated in real time and projected onto the wall, allowing for natural conversation without any sense of language barriers.

[1432] The processing flow of each embodiment will be described below.

[1433] "Example 1"

[1434] Step 1: A system with generative AI translation and summarization capabilities is installed in a designated space (e.g., a conference room or living room).

[1435] Step 2: Conversations taking place in the space are translated into multiple languages ​​in real time by the system.

[1436] Step 3: The translation result is projected onto any surface, such as a wall or screen.

[1437] Step 4: Even if participants do not share the same language, they can converse naturally by looking at the translation displayed on the screen.

[1438] "Example 2"

[1439] Step 1: At an international conference where participants speak different languages, the system is installed in the conference room.

[1440] Step 2: The system translates each participant's speech into other languages ​​in real time.

[1441] Step 3: The translation result is projected onto the wall.

[1442] Step 4: All participants can see the translation on the screen, understand the content of the meeting, and freely exchange opinions.

[1443] "Example 3"

[1444] Step 1: If your family members speak different languages, install the system in the living room or other area.

[1445] Step 2: The system translates the family's conversation in real time.

[1446] Step 3: The translation result is projected onto the wall.

[1447] Step 4: By having all family members see the translation displayed on screen, they can have natural conversations without feeling any language barrier.

[1448] Example 1

[1449] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1450] It is difficult for participants who speak different languages ​​to converse naturally in the same space. In particular, language barriers can be an obstacle in meetings or at home, making it difficult to share information smoothly. To solve this problem, real-time multilingual translation and summarization are required.

[1451] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1452] In this invention, the server includes means for capturing speech in real time, means for converting speech data into text data, means for translating and summarizing the text data into multiple languages ​​using a generative AI model, and means for displaying the translation results, thereby enabling participants who speak different languages ​​to converse naturally in the same space.

[1453] "Real-time audio capture means" refers to devices or technologies that instantly collect audio occurring within a designated space and convert it into a digital signal.

[1454] A "means for converting voice data into text data" is software or algorithms that analyzes collected voice signals and converts them into a corresponding text format.

[1455] A "generative AI model" is an advanced artificial intelligence model for natural language processing, capable of translating and summarizing text data.

[1456] "Means for translating and summarizing text data into multiple languages" means techniques and processes that use generative AI models to translate and, where necessary, summarize input text data into multiple specified languages.

[1457] "Means for displaying the translation results" refers to devices or technologies for visually displaying the translated text data, including projectors and displays.

[1458] "Means that enable natural conversation even when the language is not shared" refers to technologies and systems that enable participants who speak different languages ​​to understand conversation content that is translated in real time and communicate smoothly.

[1459] MODE FOR CARRYING OUT THE INVENTION

[1460] This invention is a system that enables participants who speak different languages ​​to converse naturally in the same space. The system includes a series of processes that capture speech in real time, convert the speech data into text data, translate and summarize the text data into multiple languages ​​using a generative AI model, and display the translation results.

[1461] Hardware and software used

[1462] Hardware:

[1463] Microphone: Highly sensitive microphones are used to capture sounds occurring within a designated space in real time.

[1464] Server: Provides the computational resources to process audio data, run generative AI models, and manage translation results.

[1465] Projector or display: Used to project the translation results onto a wall or screen.

[1466] software:

[1467] Speech recognition software (e.g., Google Speech-to-Text API): converts voice data into text data.

[1468] Generative AI models (e.g., GPT-4): Translate text data into multiple languages ​​and summarize it where necessary.

[1469] Translation software (e.g., Google Translate API): Used to supplement generative AI models.

[1470] Specific operation of the system

[1471] The server captures conversations in real time through microphones installed in the designated space. The captured audio data is converted into text using speech recognition software. A generative AI model is then used to translate the text into multiple languages ​​and summarize it as needed. The translated and summarized text is then projected onto a wall or screen via a projector or display.

[1472] Specific examples

[1473] For example, if there are participants who speak Japanese and English in a conference room, the system will operate as follows:

[1474] 1. The server captures speech in Japanese through a microphone installed in the conference room.

[1475] 2. The captured voice data is converted into text data using voice recognition software.

[1476] 3. The server uses a generative AI model to translate the Japanese text data into English.

[1477] 4. If necessary, the generative AI model summarizes the text data.

[1478] 5. The translated and summarized text data is displayed on a screen via a projector.

[1479] 6. The English translation will be displayed on the screen so that English-speaking participants can understand the content.

[1480] Prompt Sentence Examples

[1481] An example of a prompt sentence to input to the generative AI model is as follows:

[1482] "Please translate the conversation in the conference room in real time. Please translate and summarize the following Japanese text into English."

[1483] Using this prompt, the generative AI model performs the specified task, translating and summarizing the conversation.

[1484] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1485] Step 1:

[1486] Audio Capture

[1487] The server captures conversations in real time through microphones installed in the designated space. The input is the sound in the space, and the output is audio data converted into a digital signal. Specifically, the microphone collects the sound, converts it into a digital signal, and sends it to the server.

[1488] Step 2:

[1489] Voice Recognition

[1490] The server converts the received voice data into text data using speech recognition software (e.g., Google Speech-to-Text API). The input is voice data converted into digital signals, and the output is the corresponding text data. Specifically, the server inputs the voice data into the speech recognition software, which then analyzes the voice and generates text data.

[1491] Step 3:

[1492] Translation and summary

[1493] The server uses a generative AI model (e.g., GPT-4) to translate text data into multiple languages ​​and summarize it as needed. The input is text data obtained from speech recognition software, and the output is translated and summarized text data. Specifically, the server inputs text data into the generative AI model, and the model analyzes the text, translates it into the specified language, and summarizes it as needed.

[1494] Step 4:

[1495] display

[1496] The terminal (projector or display) projects the translation results sent from the server onto a wall or screen. The input is translated and summarized text data, and the output is a visually displayed translation result. Specifically, the server sends the translation result to the terminal, and the terminal displays the received data on the projector or display. The displayed translation result is projected in a location where it can be seen by participants.

[1497] (Application example 1)

[1498] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1499] There is a need to provide an environment where people who speak different languages ​​can converse naturally. In particular, in physical stores, it is important that communication between customers who speak a foreign language and store staff is smooth. However, currently, language barriers exist, making it difficult to provide smooth customer service. To solve this problem, a system with real-time translation functionality is needed.

[1500] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1501] In this invention, the server includes means for translating and summarizing using generative AI, means for displaying conversations translated into multiple languages ​​in a specified space (room), means for enabling natural conversation even when the language is not the same, and means for displaying the translation results on a smart device. This allows people who speak different languages ​​to view the translated conversation in real time through their smart devices and converse naturally.

[1502] "Generative AI" refers to artificial intelligence that generates new information based on data.

[1503] The "translation and summarization function" refers to the function that converts input language data into another language or summarizes its contents concisely.

[1504] A "designated space" refers to a specific location or area, such as a room or conference room.

[1505] "Multilingual" means two or more different languages.

[1506] "Means for displaying dialogue" refers to a device or method for visually displaying the translated dialogue.

[1507] "Means that enable natural conversation even when no common language is used" refers to techniques and methods that enable people who speak different languages ​​to communicate smoothly.

[1508] A "smart device" is an electronic device with internet connectivity and advanced computing power, including smartphones and smart glasses.

[1509] "Means for displaying the translation result" refers to a device or method for visually presenting the translated content to the user.

[1510] To implement the present invention, the following system configuration is required.

[1511] System Program

[1512] The system includes a program with generative AI translation and summarization functions. Specifically, it requires a program that integrates speech recognition, translation, and display functions.

[1513] Hardware and Software

[1514] This system uses the following hardware and software:

[1515] Hardware: Smart glasses (e.g., Google Glass), microphone

[1516] Software: Python, Google Speech Recognition API, Google Translate API, OpenCV

[1517] Processing flow

[1518] 1. Speech recognition: The server converts the voice data acquired from the microphone into text data using the Google Speech Recognition API. Through this process, what the user said is captured as text.

[1519] 2. Translation: The server translates the acquired text data into the specified language using the Google Translate API. This process enables communication between users who speak different languages.

[1520] 3. Display: The server uses OpenCV to display the translated text on the smart glasses display, allowing the user to check the translation results in real time.

[1521] Specific examples

[1522] For example, if an English-speaking customer asks, "Where can I find the electronics section?", the server will recognize the speech and translate it into Japanese, displaying it on the smart glasses' display. This allows store staff to provide smooth service without having to worry about language barriers.

[1523] Prompt Sentence Examples

[1524] An example of a prompt sentence to input to the generative AI model is as follows:

[1525] Create a program that translates English speech into Japanese and displays it on the smart glasses display. Use the Google Speech Recognition API for speech recognition and the Google Translate API for translation. Use OpenCV for display.

[1526] In this way, the present invention provides an environment in which people who speak different languages ​​can converse naturally.

[1527] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1528] Step 1:

[1529] The server acquires voice data from the microphone. The user's spoken voice is input. This voice data is sent to the server and becomes input data for voice recognition processing.

[1530] Step 2:

[1531] The server converts the acquired voice data into text data using the Google Speech Recognition API. Voice data is input, and text data is output through a voice recognition algorithm. Through this process, what the user said is captured as text.

[1532] Step 3:

[1533] The server translates the converted text data into the specified language using the Google Translate API. Text data is input, and the translation algorithm outputs text data in the specified language. This process enables communication between users who speak different languages.

[1534] Step 4:

[1535] The server uses OpenCV to display the translated text data on the smart glasses display. The translated text data is input and displayed on the smart glasses display through a display algorithm. This process allows the user to check the translation results in real time.

[1536] Step 5:

[1537] The user checks the translation results displayed on the smart glasses display and continues the conversation. In this step, the user visually checks the translation results and uses them as input data to decide the next action.

[1538] Example 2

[1539] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1540] In situations such as international conferences, when participants speak different languages, language barriers can become an obstacle to communication. To solve this problem, real-time multilingual translation and display is required, but conventional technology has had difficulty in doing so efficiently and accurately.

[1541] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1542] In this invention, the server includes means for acquiring voice data, means for converting the voice data into text data, means for translating the text data into another language, and means for displaying the translated text data, thereby enabling participants who speak different languages ​​to communicate in real time.

[1543] "Voice data" refers to data in which a user's speech or voice is recorded in digital format.

[1544] "Text data" refers to data obtained by converting voice data into character information.

[1545] "Translation" is the process of converting text data written in one language into another language.

[1546] "Display" refers to the visual presentation of the translated text data.

[1547] "Speech recognition software" is software for converting voice data into text data.

[1548] A "generative AI model" is an artificial intelligence model for translating text data into other languages.

[1549] A "projector" is a device for projecting text data onto any surface.

[1550] This invention is a system that enables participants who speak different languages ​​to communicate in real time. The system performs a series of processes: acquiring voice data, converting it into text data, and then translating and displaying it in other languages.

[1551] First, the user speaks using a device (microphone or smartphone). The device captures this speech as voice data and sends it to the server. The server then uses voice recognition software (e.g., Google Cloud Speech-to-Text) to convert the received voice data into text data. Next, the server uses a generative AI model (e.g., OpenAI's GPT-4) to translate the text data into another language. Finally, the server projects the translated text data onto a wall using a projector or display.

[1552] As a concrete example, consider an international conference with participant A who speaks English and participant B who speaks Japanese. If participant A says "Hello, everyone. Welcome to the conference," this utterance is processed as follows:

[1553] 1. Participant A's device sends voice data to the server.

[1554] 2. The server uses Google Cloud Speech-to-Text to convert the speech into text data: "Hello, everyone. Welcome to the conference."

[1555] 3. The server uses a generative AI model (GPT-4) to translate this text data into Japanese: "Hello, everyone. Welcome to the meeting."

[1556] 4. The translated text data is projected onto the wall using a projector.

[1557] Example prompt sentence:

[1558] "Please translate the English text into Japanese: Hello, everyone. Welcome to the conference."

[1559] In this way, participant A's remarks are translated into Japanese in real time, allowing participant B to understand the content of the meeting. This system enables participants who speak different languages ​​to communicate smoothly.

[1560] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1561] Step 1:

[1562] A user speaks using a device (microphone or smartphone), which captures the speech as voice data and sends it to the server.

[1563] Input: User's speech (audio)

[1564] Output: Digital audio data

[1565] Specific operation: The user says, "Hello, everyone. Welcome to the conference." The device records this speech and sends it to the server as digital audio data.

[1566] Step 2:

[1567] The server converts the received voice data into text data using voice recognition software (e.g., Google Cloud Speech-to-Text).

[1568] Input: Digital audio data

[1569] Output: Text data

[1570] Specific operation: The server inputs the voice data into Google Cloud Speech-to-Text and obtains the text data "Hello, everyone. Welcome to the conference."

[1571] Step 3:

[1572] The server uses a generative AI model (e.g., OpenAI's GPT-4) to translate the text data into other languages.

[1573] Input: Text data

[1574] Output: Translated text data

[1575] Specific operation: The server inputs the text data "Hello, everyone. Welcome to the conference." into GPT-4 and obtains the Japanese translation result "Hello, everyone. Welcome to the conference."

[1576] Step 4:

[1577] The server then projects the translated text data onto a wall using a projector or display.

[1578] Input: Translated text data

[1579] Output: Text displayed on the wall

[1580] Specific operation: The server sends the text data "Hello everyone. Welcome to the meeting." to the projector and displays it on the wall.

[1581] In this way, the user's speech is translated into other languages ​​in real time, allowing other participants to understand the content of the conference.

[1582] (Application example 2)

[1583] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1584] The goal is to solve the problem of passengers who speak different languages ​​having difficulty communicating smoothly in self-driving vehicles, and to ensure that passengers can understand destination directions, emergency instructions, and tourist information in real time.

[1585] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means having a generative AI translation and summarization function, means for displaying conversations translated into multiple languages ​​in a specified space (room), means for enabling natural conversation even when the languages ​​are not the same, means for providing translated information in real time via an in-vehicle display device or mobile device when passengers in an autonomous vehicle speak different languages, and means for playing back the translated information by voice. This enables passengers who speak different languages ​​to communicate smoothly in an autonomous vehicle.

[1586] "Generative AI" is a technology that uses artificial intelligence to generate, translate, and summarize data.

[1587] The "translation and summary function" converts input language data into another language, extracts important information, and summarizes it concisely.

[1588] A "designated space" refers to a specific location or area, which in this case means a physical space such as a room or a car.

[1589] "Multiple languages" refers to two or more different languages, including, for example, Japanese, English, and French.

[1590] "Means for displaying the conversation" refers to a technology that visually displays the translated conversation content using a display device such as a monitor or projector.

[1591] "Means that enable natural conversation" are technologies that allow people who speak different languages ​​to communicate smoothly without feeling a language barrier.

[1592] An "autonomous vehicle" is a vehicle that drives autonomously using artificial intelligence and sensor technology.

[1593] A "display device" is a device for visually displaying information, and includes displays, monitors, and the like.

[1594] "Mobile device" refers to a portable electronic device such as a smartphone or tablet.

[1595] "Real-time translated information" means information in which speech or text is instantly converted into another language and provided without delay.

[1596] "Audio playback" refers to a technique that uses speech synthesis technology to output the translated text as audio.

[1597] The following system configuration will be described as an embodiment of the present invention.

[1598] System Configuration

[1599] The system includes a server with generative AI translation and summarization functions, a display device that projects conversations translated into multiple languages ​​in a designated space (room), an audio playback device that enables natural conversation even when the language is not the same, and a mobile device that provides real-time translated information when passengers in an autonomous vehicle speak different languages.

[1600] Program processing

[1601] The server runs a program that integrates speech recognition, translation, and speech synthesis functions. Specifically, it uses the following hardware and software:

[1602] Hardware: microphones, speakers, displays (in-car displays and smartphones)

[1603] Software: Python, SpeechRecognition library, Google Translate API, pyttsx3 library

[1604] Voice Recognition

[1605] The server converts the audio data acquired from the microphone into text data using the SpeechRecognition library. This text data represents what the passenger said.

[1606] translation

[1607] The server uses the Google Translate API to translate the acquired text data into the target language, for example, translating Japanese text into English.

[1608] Speech synthesis

[1609] The server uses the pyttsx3 library to convert the translated text data into audio data and play it through the speaker.

[1610] Specific examples

[1611] For example, if a Japanese-speaking passenger asks, "Where is the next stop?", the system will translate the question into English and display "Where is the next stop?" on the in-car display while simultaneously playing it aloud.

[1612] Prompt Sentence Examples

[1613] Translate the Japanese audio into English, display it on the screen, and play it back aloud.

[1614] In this way, passengers who speak different languages ​​will be able to communicate seamlessly inside an autonomous vehicle.

[1615] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1616] Step 1:

[1617] The user speaks into the microphone. The input is the user's voice data. The server receives this voice data and converts it into text data using the SpeechRecognition library. The output is text data representing the user's speech.

[1618] Step 2:

[1619] The server sends the text data obtained in step 1 to the Google Translate API and translates it into the specified target language. The input is the text data and the target language information. The output is the translated text data.

[1620] Step 3:

[1621] The server converts the translated text data into audio data using the pyttsx3 library. The input is the translated text data. The output is audio data.

[1622] Step 4:

[1623] The server plays the audio data generated in step 3 through the speaker. The input is the audio data. The output is the audio that the user can hear.

[1624] Step 5:

[1625] The server sends the translated text data to an in-car display or mobile device for display on the display device. The input is the translated text data. The output is the translated text displayed on the display.

[1626] In this way, what the user says is translated in real time and provided as both audio and text.

[1627] Example 3

[1628] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1629] It is difficult for people who speak different languages ​​to converse naturally in the same space. Language barriers often become an obstacle to communication, especially at home or in small gatherings. To solve this problem, real-time speech recognition and translation, as well as a visual display of the translation results, are required.

[1630] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1631] In this invention, the server includes means for capturing voice input, means for transmitting the captured voice data to the server, means for converting the voice data into text, means for translating the text into a specified target language, means for transmitting the translated text to the terminal, and means for displaying the translated text, thereby enabling people who speak different languages ​​to communicate naturally while visually checking translated conversations in real time.

[1632] A "means for capturing voice input" is a device or function for detecting a user's speech and collecting the voice data.

[1633] The "means for transmitting captured voice data to a server" refers to a device or function for transmitting collected voice data to a server via a communication means such as the Internet.

[1634] "Means for converting voice data into text" refers to a device or function for analyzing voice data and converting the content into text format.

[1635] A "means for translating text into a specified target language" is a device or function for converting text data into a different language.

[1636] The "means for transmitting the translated text to the terminal" refers to a device or function for transmitting the translated text data to the terminal via a communication means such as the Internet.

[1637] The "means for displaying translated text" refers to a device or function for visually displaying translated text data.

[1638] This invention is a system that allows people who speak different languages ​​to converse naturally in the same space. The system includes a set of means for capturing voice input, translating it in real time, and visually displaying the results.

[1639] Hardware and software used

[1640] Hardware:

[1641] Device: Smart speaker or smart device with microphone

[1642] Display devices: projectors and smart displays

[1643] software:

[1644] Speech Recognition API: Google Cloud Speech-to-Text API

[1645] Translation API: Google Cloud Translation API

[1646] Data processing and calculation

[1647] Capturing audio input:

[1648] When a user speaks, the device captures the audio. For example, a smart speaker installed in the living room detects the user's speech.

[1649] Sending audio data:

[1650] The captured audio data is transmitted to a server over the Internet.

[1651] Voice Recognition:

[1652] The server converts the audio data into text using the Google Cloud Speech-to-Text API. For example, if a user says, "How was your day?", the API converts this to the text "How was your day?"

[1653] translation:

[1654] The server translates the converted text into the specified target language using the Google Cloud Translation API. For example, if a translation from English to Japanese is required, the API will translate "How was your day?" into "How was your day today?"

[1655] Sending translation results:

[1656] The translated text is sent to the device via the internet.

[1657] Displaying translation results:

[1658] The device then projects the translation results onto a wall or screen using a projector or smart display. For example, the text "How was your day?" could be displayed on a living room wall.

[1659] Specific examples

[1660] As a concrete example, consider a conversation between an English-speaking father and a Japanese-speaking mother in the living room.

[1661] 1. Capture audio input:

[1662] The father says, "How was your day?"

[1663] The device captures this audio and sends it to the server.

[1664] 2. Speech Recognition:

[1665] The server uses the Google Cloud Speech-to-Text API to convert the audio data into text: "How was your day?"

[1666] 3. Translation:

[1667] The server uses the Google Cloud Translation API to translate "How was your day?" to "How was your day?"

[1668] 4. Sending translation results:

[1669] The server sends the translated text "How was your day?" to the device.

[1670] 5. Displaying the translation results:

[1671] The device projects the text "How was your day?" onto the living room wall.

[1672] Example prompt sentence:

[1673] User: "How was your day?"

[1674] Server: "How was your day?"

[1675] This system allows smooth communication between family members who speak different languages ​​through real-time translation. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1676] Step 1:

[1677] Capture audio input

[1678] When a user speaks, the device captures the voice. For example, a smart speaker installed in a living room detects the user's utterance, "How was your day?" The input is the user's voice, and the output is the captured voice data.

[1679] Step 2:

[1680] Sending audio data

[1681] The terminal transmits the captured audio data to a server via the Internet. The input is the captured audio data, and the output is the audio data transmitted to the server.

[1682] Step 3:

[1683] Voice Recognition

[1684] The server converts the voice data into text using the Google Cloud Speech-to-Text API. For example, the voice data "How was your day?" is converted into text "How was your day?". The input is the voice data sent to the server, and the output is the converted text data.

[1685] Step 4:

[1686] translation

[1687] The server translates the converted text into the specified target language using the Google Cloud Translation API. For example, the text "How was your day?" is translated into Japanese as "How was today?". The input is the converted text data, and the output is the translated text data.

[1688] Step 5:

[1689] Sending translation results

[1690] The server transmits the translated text to the terminal via the Internet. The input is the translated text data, and the output is the translated text data transmitted to the terminal.

[1691] Step 6:

[1692] Displaying translation results

[1693] The device then projects the translation results onto a wall or screen via a projector or smart display. For example, the text "How was your day?" is displayed on the wall of your living room. The input is the translated text data sent to the device, and the output is the translated text displayed on the wall or screen.

[1694] (Application example 3)

[1695] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1696] In environments where people speaking different languages ​​coexist, it is important to achieve natural communication without feeling the language barrier. In particular, smooth conversations between multinational customers and store staff are required in brick-and-mortar stores. Conventional translation systems often lack practicality due to limitations in real-time performance and display methods.

[1697] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1698] In this invention, the server includes a means having a generative AI translation and summarization function, a means for projecting conversations translated into multiple languages ​​in a specified space (room), a means for enabling natural conversation even when the language is not the same, and a means for translating conversations with customers who speak different languages ​​in real time using smart glasses and displaying the translation on a display, thereby enabling smooth real-time communication between customers who speak different languages ​​and store clerks.

[1699] "Generative AI" is a system that uses artificial intelligence to generate, translate, and summarize data.

[1700] The "translation and summary function" is a function that converts input language data into another language or summarizes its contents concisely.

[1701] "Designated space" refers to the specific room or area in which the system operates.

[1702] "Multiple languages" refers to two or more different languages ​​and indicates the types of languages ​​that the system supports.

[1703] "Means for displaying dialogue" refers to devices or technologies that visually display the translated dialogue.

[1704] "Means that enable natural conversation" refers to methods and techniques that enable smooth communication without making language differences noticeable.

[1705] "Smart glasses" are glasses-type devices that have built-in displays and sensors and have the ability to visually display information.

[1706] "Means for translating in real time and displaying the results on a display" refers to technology or devices for instantly translating conversations and displaying the results on a display.

[1707] A system for implementing this invention includes means for having generative AI translation and summarization functions, means for displaying conversations translated into multiple languages ​​in a specified space, means for enabling natural conversation even when the languages ​​are not the same, and means for translating conversations with customers who speak different languages ​​in real time using smart glasses and displaying the translation on a display.

[1708] The server uses generative AI to translate and summarize the voice data in real time. Specifically, it converts the voice into text using speech recognition software (e.g., Google Speech Recognition API), and then translates the text into the specified language using translation software (e.g., Google Translate API). The translated text is then displayed on the smart glasses' display.

[1709] The device (smart glasses) receives the translated text sent from the server and displays it on its screen. This allows the store clerk to understand the customer's language and respond appropriately. The smart glasses have a built-in display and sensors, and can display information visually.

[1710] The user (store clerk) wears the smart glasses and converses with customers. What the customer says is translated in real time and displayed on the smart glasses' display, enabling smooth communication without any language barriers.

[1711] As a concrete example, consider a scenario in which an English-speaking tourist visits a brick-and-mortar store in Japan and asks a store clerk for product details. The store clerk is wearing smart glasses, and the tourist's English question is translated into Japanese in real time and displayed on the glasses' display. The clerk then looks at the translation and provides an appropriate response in Japanese.

[1712] Examples of prompts for generative AI models include:

[1713] "An English-speaking tourist visits a brick-and-mortar store in Japan and asks a store clerk for product details. The clerk is wearing smart glasses, and the tourist's English question is translated into Japanese in real time and displayed on the glasses' display. The clerk then looks at the translation and provides an appropriate response in Japanese. Please generate a Python program to realize this scenario."

[1714] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1715] Step 1:

[1716] The server captures the user's (store clerk's) voice through a microphone. The input is voice data, which is converted into text data using speech recognition software (e.g., Google Speech Recognition API). The output is text data.

[1717] Step 2:

[1718] The server inputs the text data obtained in step 1 into translation software (e.g., Google Translate API) and translates it into the specified language. The input is text data, which is converted into another language using translation software. The output is the translated text data.

[1719] Step 3:

[1720] The server sends the translated text data to the smart glasses. The input is the translated text data, which is sent to the smart glasses via the network. The output is the text data displayed on the smart glasses.

[1721] Step 4:

[1722] The terminal (smart glasses) receives the translated text sent from the server and displays it on the display. The input is the translated text data sent from the server, which is displayed on the display. The output is the translated text displayed on the display of the smart glasses.

[1723] Step 5:

[1724] The user (store clerk) reads the translated text displayed on the smart glasses' display and responds appropriately to the customer. The input is the translated text displayed on the smart glasses' display, and the clerk responds to the customer based on this. The output is the appropriate response to the customer.

[1725] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1726] "Example 1"

[1727] One embodiment of the present invention is a system that combines the translation and summarization functions of generative AI with an emotion engine. This system translates and summarizes a user's speech in real time and projects the results on the wall of a specified space (room). Furthermore, the emotion engine recognizes emotions from the user's tone of voice, facial expression, and choice of words, and adjusts the translation and summarization results accordingly. For example, if it recognizes that the user is angry, it adjusts the translation results to appropriately express that anger. It also recognizes emotions such as joy and surprise and adjusts the translation results accordingly. This enables natural conversations that include emotions, even between users who do not share the same language.

[1728] "Example 2"

[1729] The emotion engine can also adjust the content and format of the display. For example, if it recognizes that the user is happy, it can change the color of the displayed text to a brighter color. Or, if it recognizes that the user is surprised, it can make the displayed text larger, providing visual feedback according to the emotion. This allows the user to confirm that their emotions are being recognized by the system and reflected appropriately.

[1730] "Example 3"

[1731] Furthermore, this system not only adjusts the translation and summarization results according to the user's emotions, but also suggests dialogues that correspond to those emotions. For example, if it recognizes that the user is confused, it will identify the parts of the conversation that may be causing that confusion and provide an explanation or commentary. Similarly, if it recognizes that the user is happy, it will suggest replies that share that joy. This allows users to enjoy natural dialogue based on an understanding of their own emotions.

[1732] The processing flow of each embodiment will be described below.

[1733] "Example 1"

[1734] Step 1: The user starts a conversation, which is input to the system via an input device such as a microphone.

[1735] Step 2: Generative AI translates and summarizes the user's speech in real time. This translation and summary translates the user's speech into another language and extracts the main points.

[1736] Step 3: The emotion engine recognizes the user's emotions from their tone of voice, facial expressions, and choice of words.

[1737] Step 4: The generative AI adjusts the translation and summarization results based on the emotions recognized by the emotion engine.

[1738] Step 5: The adjusted translation and summary results are projected onto the wall or other surface of the designated space (room).

[1739] "Example 2"

[1740] Step 1: The user starts a conversation, which is input to the system via an input device such as a microphone.

[1741] Step 2: Generative AI translates and summarizes the user's speech in real time. This translation and summary translates the user's speech into another language and extracts the main points.

[1742] Step 3: The emotion engine recognizes the user's emotions from their tone of voice, facial expressions, and choice of words.

[1743] Step 4: The emotion engine adjusts the content and format of the display depending on the emotion it recognizes. For example, if it recognizes that the user is happy, it changes the color of the displayed text to a brighter color.

[1744] Step 5: The translation and summary results are projected onto the wall of a designated space (room) in the adjusted content and format.

[1745] "Example 3"

[1746] Step 1: The user starts a conversation, which is input to the system via an input device such as a microphone.

[1747] Step 2: Generative AI translates and summarizes the user's speech in real time. This translation and summary translates the user's speech into another language and extracts the main points.

[1748] Step 3: The emotion engine recognizes the user's emotions from their tone of voice, facial expressions, and choice of words.

[1749] Step 4: The generative AI suggests dialogue based on the emotion recognized by the emotion engine. For example, if it recognizes that the user is confused, it will identify parts of the conversation that may be causing that confusion and provide an explanation or commentary.

[1750] Step 5: Project the proposed dialogue onto a wall or other surface in a designated space (room).

[1751] Example 1

[1752] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1753] Conventional translation systems have had problems with natural conversations between users who speak different languages, especially with the difficulty of conveying emotional nuances. Real-time translation and summarization are also inadequate, often disrupting the flow of conversation. Furthermore, there are limited ways to properly display translation results, making them less convenient for users.

[1754] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means having a generative AI translation and summarization function, a means for displaying conversations translated into multiple languages ​​in a specified space, a means for enabling natural conversation even when the languages ​​are not the same, and a means for recognizing user emotions and adjusting the results of translation and summarization according to those emotions. This enables natural conversation, including emotions, even between users who speak different languages, realizes real-time translation and summarization, and allows the translation results to be displayed appropriately on any surface.

[1755] "Generative AI" is a system that uses artificial intelligence technology to generate, translate, and summarize data.

[1756] The "translation and summarization function" converts input text and audio data into other languages ​​and, if necessary, summarizes the content concisely.

[1757] A "designated space" refers to a specific location or area, such as a conference room or living room.

[1758] "Multiple languages" refers to two or more different languages, such as Japanese, English, and French.

[1759] "Conversation projection" refers to a method of displaying translated text on a surface such as a wall or screen using a projector or display.

[1760] "Means that enable natural conversation" refers to methods that facilitate communication between users who speak different languages ​​and enable natural conversation that includes emotions and nuances.

[1761] "Means for recognizing emotions" refers to a method of analyzing emotions from the user's tone of voice, facial expressions, and choice of words, and then providing appropriate translations and summaries based on that information.

[1762] "Adjustment measures" are methods for modifying translation and summarization results based on recognized emotions to make them more appropriate.

[1763] This invention is a system that combines the translation and summarization functions of generative AI with an emotion recognition engine, translating conversations taking place within a specified space into multiple languages ​​in real time and adjusting the translation results by recognizing emotions. Specific embodiments of this system are described below.

[1764] Hardware and software used

[1765] Hardware: high-performance microphones, cameras, projectors, servers

[1766] Software: Generative AI models (e.g., OpenAI's GPT-4), emotion recognition engines (e.g., Microsoft Azure's Emotion API)

[1767] System configuration

[1768] 1. A high-performance microphone installed on the device captures conversations taking place within a designated space in real time.

[1769] 2. A camera installed on the device captures the user's facial expressions.

[1770] 3. The device sends the captured voice and facial expression data to the server via the network.

[1771] 4. The server inputs the received voice data into the generative AI model and converts it into text data.

[1772] 5. At the same time, the server uses an emotion recognition engine to analyze the user's emotions from the facial expression data and voice data.

[1773] 6. The server uses the generative AI model to translate the text data into multiple languages.

[1774] 7. The server adjusts the translation results based on the emotional data analyzed by the emotion recognition engine.

[1775] 8. The server sends the adjusted translation results to the device.

[1776] 9. The device uses a projector to project the translation results onto any surface, such as a wall or screen.

[1777] Specific examples

[1778] For example, suppose there is a Japanese-speaking user A and an English-speaking user B in a conference room. If user A says, "This project is very important," the system processes it as follows:

[1779] 1. The device's microphone captures what User A says.

[1780] 2. The device's camera captures User A's facial expression.

[1781] 3. The device sends the voice data and facial expression data to the server.

[1782] 4. The server converts the voice data into text, and the emotion recognition engine analyzes User A's emotions (e.g., serious expression).

[1783] 5. The server translates the text data into English using a generative AI model.

[1784] 6. Based on the results of the emotion recognition engine, the server adjusts the translation result to "This project is extremely important."

[1785] 7. The server sends the adjusted translation results to the device.

[1786] 8. The terminal uses a projector to project the translation results onto the wall of the conference room.

[1787] Prompt Sentence Examples

[1788] Examples of prompts to be input to a generative AI model include:

[1789] User says: "This project is very important"

[1790] User Sentiment: "Serious"

[1791] Target language: "English"

[1792] By inputting this prompt sentence into a generative AI model, appropriate translation results can be obtained.

[1793] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1794] Step 1:

[1795] A high-performance microphone installed on the device captures conversations taking place within a designated space in real time. The input is a voice signal, and the output is digital voice data. Specifically, the microphone converts the voice signal into digital data.

[1796] Step 2:

[1797] A camera installed on the device captures the user's facial expression. The input is the user's facial expression, and the output is image data. Specifically, the camera generates image data.

[1798] Step 3:

[1799] The device transmits the captured voice and facial expression data to a server via a network. The input is digital voice data and image data, and the output is data transmission to the server. Specifically, the device divides the data into packets and transmits them to the server using the TCP / IP protocol.

[1800] Step 4:

[1801] The server inputs the received voice data into a generative AI model and converts it into text data. The input is digital voice data, and the output is text data. Specifically, the server calls a voice recognition API and converts the voice data into text.

[1802] Step 5:

[1803] At the same time, the server uses an emotion recognition engine to analyze the user's emotions from the facial expression data and voice data. The input is image data and digital voice data, and the output is emotion data. Specifically, the server calls the emotion recognition API and analyzes the facial expression data and voice data.

[1804] Step 6:

[1805] The server uses a generative AI model to translate text data into multiple languages. The input is text data, and the output is translated text data. Specifically, the server calls a translation API to translate the text data into the specified language.

[1806] Step 7:

[1807] The server adjusts the translation results based on the emotional data analyzed by the emotion recognition engine. The input is the translated text data and the emotional data, and the output is the adjusted translated text data. Specifically, the server analyzes the emotional data and applies rules to adjust the tone and expression of the translation results.

[1808] Step 8:

[1809] The server sends the adjusted translation results to the terminal. The input is the adjusted translated text data, and the output is the data sent to the terminal. Specifically, the server divides the data into packets and sends them to the terminal using the TCP / IP protocol.

[1810] Step 9:

[1811] The terminal uses a projector to project the translation results onto any surface, such as a wall or screen. The input is the adjusted translation text data, and the output is the displayed translation result. Specifically, the terminal sends the received translation result to the projector and displays it at the specified location.

[1812] (Application example 1)

[1813] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1814] In a multilingual environment, it is difficult to achieve natural conversations between users who speak different languages. Furthermore, simple translation alone may not convey the nuances of emotions, which can lead to misunderstandings. In brick-and-mortar stores in particular, smooth communication between customers and store clerks is required, so real-time translation and emotional recognition and adjustment are necessary.

[1815] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1816] In this invention, the server includes means for translating and summarizing using generative AI, means for displaying conversations translated into multiple languages ​​in a specified space (room), means for enabling natural conversation even when the languages ​​are not the same, means for recognizing the user's emotions using an emotion recognition engine and adjusting the translation results, and means for displaying the translation results on a smart device, thereby enabling natural conversations that include emotions even between users who speak different languages.

[1817] "Generative AI" refers to artificial intelligence that uses natural language processing to generate, translate, summarize, and otherwise process text.

[1818] The "translation and summary function" is a function that translates input text into another language or summarizes its contents concisely.

[1819] A "designated space" is a specific location or room where the conversation takes place.

[1820] "Multilingual" means two or more different languages.

[1821] "Means for projecting the dialogue" refers to a method or device for projecting the translated text onto a display device such as a wall or screen.

[1822] An "emotion recognition engine" is software or hardware that analyzes and recognizes emotions from a user's tone of voice, facial expressions, and choice of words.

[1823] A "smart device" is a portable electronic device with internet connectivity and advanced computing capabilities.

[1824] "Means for adjusting the translation result" refers to a method or device that appropriately changes the content or expression of the translated text based on the recognized sentiment.

[1825] The system for implementing this invention is configured by combining the translation and summarization functions of generative AI, an emotion recognition engine, and a smart device. A specific embodiment of this system is shown below.

[1826] System configuration

[1827] 1. Generative AI translation and summarization functions

[1828] The server translates and summarizes the user's speech in real time using a generative AI model, such as GPT-4.

[1829] 2. Emotion Recognition Engine

[1830] The server uses an emotion recognition engine, such as Affectiva, to recognize the user's emotions from their tone of voice, facial expressions, and language.

[1831] 3. Smart Devices

[1832] The terminal uses smart devices such as smart glasses or head-mounted displays to display the translation results, allowing store clerks and users to check the translation results in real time. Specific devices that can be used include Google Glass and Microsoft HoloLens.

[1833] Data processing and calculation

[1834] 1. Acquiring voice input

[1835] When the user speaks, the voice data is acquired by the terminal.

[1836] 2. Converting voice data to text

[1837] The acquired voice data is sent to a server and converted into text data by voice recognition software.

[1838] 3. Emotion recognition

[1839] The text data is analyzed by an emotion recognition engine to recognize the user's emotions.

[1840] 4. Translation and Summary

[1841] The generative AI model translates text data into a specified language and summarizes it as needed.

[1842] 5. Adjustment of translation results

[1843] Based on the recognized emotion, the translation result is adjusted appropriately: for example, if the user is angry, the translation result will reflect that emotion.

[1844] 6. Displaying the translation results

[1845] The adjusted translation results are displayed on the smart device, allowing store clerks and users to check the translation results in real time.

[1846] Specific examples

[1847] For example, if a customer says, "I am very happy with the service!", it will be processed as follows:

[1848] 1. Audio data is acquired by the device.

[1849] 2. The audio data is converted into text data.

[1850] 3. The emotion recognition engine recognizes the emotion "happy."

[1851] 4. The generative AI model translates this as "I'm very satisfied with your service!"

[1852] 5. Based on the perceived emotion, it is adjusted to "(with joy) I am very satisfied with the service!"

[1853] 6. The adjusted translation results will be displayed on your smart device.

[1854] Prompt Sentence Examples

[1855] "You will create a program that translates customer utterances in real time, recognizes emotions, and adjusts the translation results accordingly. The emotion recognition engine used is Affectiva, and the translation API is Google Translate API. The translation results will be displayed on the smart glasses. Use the following as an example of a customer utterance: 'I am very happy with the service!'"

[1856] In this way, the present invention can realize natural conversations that include emotions even between users who speak different languages.

[1857] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1858] Step 1:

[1859] The user speaks.

[1860] The user utters, "I am very happy with the service!" This voice data is acquired by the terminal. The input is the user's voice data, and the output is the voice data sent to the terminal.

[1861] Step 2:

[1862] The terminal transmits the voice data to the server.

[1863] The terminal transmits the acquired voice data to the server. The input is the voice data acquired by the terminal, and the output is the voice data transmitted to the server.

[1864] Step 3:

[1865] The server converts the voice data into text data.

[1866] The server uses speech recognition software to convert the voice data into text data: the input is the voice data sent to the server, and the output is text data.

[1867] Step 4:

[1868] The server sends the text data to the emotion recognition engine.

[1869] The server sends the text data to the emotion recognition engine to recognize the user's emotion. The input is the text data, and the output is the recognized emotion data.

[1870] Step 5:

[1871] The server translates the text data using a generative AI model.

[1872] The server uses a generative AI model to translate text data into a specified language. The input is text data, and the output is the translated text data.

[1873] Step 6:

[1874] The server adjusts the translation results based on the emotion data.

[1875] The server adjusts the translation result appropriately based on the recognized emotion data. For example, if the emotion "happy" is recognized, it adds the phrase "(with joy)" to the translation result. The input is the translated text data and emotion data, and the output is the adjusted translation result.

[1876] Step 7:

[1877] The server sends the adjusted translation results to the device.

[1878] The server sends the adjusted translation result to the terminal. The input is the adjusted translation result, and the output is the translation result sent to the terminal.

[1879] Step 8:

[1880] The terminal displays the translation results on the smart device.

[1881] The terminal displays the received translation results on the smart device. The input is the translation result sent to the terminal, and the output is the translation result displayed on the smart device.

[1882] In this way, the user's speech is translated in real time, enabling natural conversation that includes emotions.

[1883] Example 2

[1884] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1885] In situations such as international conferences, when participants speak different languages, language barriers often become an obstacle to communication. It is also difficult to understand the content of the conference and exchange opinions in real time. Furthermore, the quality of communication can be reduced because visual feedback that properly reflects participants' emotions is not provided.

[1886] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means having a translation and summarization function of generative artificial intelligence, means for displaying conversations translated into multiple languages ​​in a specified space, means for enabling natural conversation even when the languages ​​are not the same, means for collecting voice data, means for converting the collected voice data into text data, means for recognizing emotions in the text data, and means for adjusting the content and format of the display based on the emotion recognition results. This enables participants who speak different languages ​​to understand the conversation in real time and receive visual feedback according to their emotions.

[1887] "Generative AI" is an AI that has the ability to analyze voice and text data and translate and summarize.

[1888] The "translation and summary function" is a function that converts input language data into another language or summarizes its contents concisely.

[1889] A "designated space" refers to a specific location such as a conference room or hall.

[1890] "Multiple languages" refers to different languages, including, for example, Japanese, English, and French.

[1891] "Means for projecting the dialogue" refers to a device or method for visually displaying the translated text using a projector, display, or the like.

[1892] "Means for collecting voice data" refers to a method for acquiring speech as digital voice data using a microphone or voice input device.

[1893] "Means for converting into text data" refers to a method of converting voice data into text information using voice recognition software.

[1894] "Emotion recognition means" refers to software or algorithms that analyze and identify a user's emotions from text data or voice data.

[1895] "Means for adjusting the content and format of the display" refers to a method for changing visual attributes such as the color and size of the displayed text based on the results of emotion recognition.

[1896] This invention is a system that enables participants who speak different languages ​​to understand conversations in real time and receive visual feedback according to their emotions in situations such as international conferences. This system includes means for having a generative artificial intelligence translation and summarization function, means for displaying conversations translated into multiple languages ​​in a specified space, means for enabling natural conversation even when the languages ​​are not the same, means for collecting voice data, means for converting the collected voice data into text data, means for recognizing emotions in the text data, and means for adjusting the content and format of the display based on the results of the emotion recognition.

[1897] Hardware and software used

[1898] Hardware:

[1899] High-performance servers (e.g., high-performance servers)

[1900] Projectors (e.g. high-resolution projectors)

[1901] Microphone (e.g. high-sensitivity microphone)

[1902] software:

[1903] Speech recognition software (e.g., speech recognition API)

[1904] Translation software (e.g. translation API)

[1905] Emotion recognition software (e.g., emotion recognition API)

[1906] Program processing explanation

[1907] The server translates the speech of participants into other languages ​​in real time and projects it onto the wall. It also has an emotion engine that adjusts the content and format of the display according to the user's emotions.

[1908] The device uses a microphone to collect speech from meeting participants and sends the audio data to a server. The server uses speech recognition software to convert the audio data into text data. The converted text data is then input into translation software and translated into the specified language. The translated text data is then input into emotion recognition software to analyze the user's emotions. The color and size of the displayed text are adjusted based on the results of the emotion recognition. The adjusted text data is then sent to a projector and projected onto the wall.

[1909] Specific examples

[1910] As a specific use case, consider the following scenario:

[1911] Scenario: At an international conference, there is participant A who speaks Japanese and participant B who speaks English.

[1912] Participant A says, "Hello everyone."

[1913] The device collects this speech with a microphone and transmits the audio data to the server.

[1914] The server uses voice recognition software to convert "Hello everyone" into text.

[1915] The server translates this to "Hello, everyone" using its translation software.

[1916] The server uses emotion recognition software to analyze the emotion in "Hello everyone" and detect joy.

[1917] The server changes the color of the text to a brighter color and sends it to the projector.

[1918] The terminal projects "Hello, everyone" onto the wall via a projector.

[1919] Prompt Sentence Examples

[1920] Write a program that translates the Japanese phrase "Hello everyone" into English, performs emotion recognition, and if joy is detected, changes the color of the text to a brighter color.

[1921] In this way, users can be sure that their emotions are recognized and reflected appropriately by the system.

[1922] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1923] Step 1: Collecting audio data

[1924] The device collects speech from participants using a microphone, and the collected voice data is sent to the server in real time.

[1925] Input: Meeting participant's remarks

[1926] Output: Digital audio data

[1927] How it works: A high-sensitivity microphone connected to the device captures what participants say and sends the audio data digitally to a server.

[1928] Step 2: Voice Recognition

[1929] The server inputs the received voice data into voice recognition software and converts the voice data into text data.

[1930] Input: Digital audio data

[1931] Output: Text data

[1932] Specific operation: The server sends the voice data to a speech recognition API and converts the voice "Hello everyone" into text "Hello everyone."

[1933] Step 3: Translation

[1934] The server inputs the converted text data into translation software and translates it into the specified language.

[1935] Input: Text data

[1936] Output: Translated text data

[1937] Specific operation: The server sends the text data "Hello, everyone" to the translation API, which translates it to "Hello, everyone."

[1938] Step 4: Emotion Recognition

[1939] The server inputs the translated text data into emotion recognition software to analyze the user's emotions.

[1940] Input: Translated text data

[1941] Output: Emotion recognition result

[1942] Specific operation: The server sends the text data "Hello everyone" to the emotion recognition API and detects the emotion of joy.

[1943] Step 5: Adjust the visual feedback

[1944] The server adjusts the color and size of the displayed text based on the results of emotion recognition.

[1945] Input: Emotion recognition results

[1946] Output: Adjusted text data

[1947] Specific operation: Based on the emotion recognition result, the server generates an instruction to change the color of the text to yellow.

[1948] Step 6: Projection

[1949] The terminal sends the adjusted text data to the projector, which projects it onto the wall.

[1950] Input: Adjusted text data

[1951] Output: Text projected on the wall

[1952] Specific operation: The terminal sends the adjusted text data "Hello, everyone" received from the server to the projector, projecting it in yellow letters on the wall.

[1953] In this way, users can be sure that their emotions are recognized and reflected appropriately by the system.

[1954] (Application example 2)

[1955] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1956] In multilingual environments such as international conferences and virtual stores, there is a need to achieve smooth communication across language barriers. Another challenge is to communicate information more intuitively and effectively by providing visual feedback that reflects the user's emotions. In particular, in virtual stores, it is important to provide appropriate information about products that users are interested in, thereby motivating them to purchase.

[1957] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1958] In this invention, the server includes a means for translating and summarizing using generative AI, a means for displaying conversations translated into multiple languages ​​in a specified space, a means for enabling natural conversation even when the language is not the same, a means for recognizing a user's emotions and adjusting the content and format of the display, and a means for displaying product information in a virtual store using smart glasses. This enables real-time translation and visual feedback according to emotions in a multilingual environment, making it easier for users to intuitively understand information. Furthermore, in the virtual store, providing appropriate information for products in which the user is interested can increase purchasing motivation.

[1959] "Generative AI" is a system that uses artificial intelligence to generate and convert data.

[1960] "Translation and summary function" refers to a function that translates input text or audio into another language or summarizes the content concisely.

[1961] A "designated space" refers to a specific location or area, including conference rooms and virtual stores.

[1962] A "conversation translated into multiple languages" is a conversation between multiple users who speak different languages, translated into each of the languages.

[1963] "Displaying means" refers to devices or techniques used to visually display translated text or information.

[1964] "Means that enable natural conversation" are techniques and methods that enable smooth communication beyond language differences.

[1965] "Means for recognizing the user's emotions and adjusting the content and format displayed" refers to technology that detects the user's emotions and changes the color and size of the displayed information accordingly.

[1966] "Smart glasses" are glasses-type devices that have the function of displaying information.

[1967] A "virtual store" is a virtual store that operates on the Internet, where users can browse and purchase products online.

[1968] "Product information" refers to information such as product details, features, and prices.

[1969] A system for implementing this invention includes means for having generative AI translation and summarization functions, means for displaying conversations translated into multiple languages ​​in a specified space, means for enabling natural conversation even when the languages ​​are not the same, means for recognizing the user's emotions and adjusting the content and format of the display, and means for displaying product information in a virtual store using smart glasses.

[1970] The server runs programs for speech recognition, translation, emotion recognition, and visual feedback. Specifically, speech recognition uses a microphone and Google's speech recognition API. Translation uses Google Translate API and emotion recognition uses Emotion Recognition API. Visual feedback uses OpenCV and displays information on smart glasses.

[1971] As a user wears the smart glasses and walks around the virtual store, descriptions of each product are translated into the user's language in real time. The display also changes depending on the user's emotions, providing feedback such as enlarging detailed information about products that interest the user.

[1972] For example, if a user says "I'd like to know more about this product" in a virtual store, this statement is translated into English in real time, and if the user is recognized as interested, the translated text is displayed in a brighter color and larger.

[1973] Examples of prompts to input to a generative AI model include:

[1974] When a user says "I want to know more about this product," create a program that translates this statement into English in real time and, if it recognizes that the user is interested, displays the translated text in a larger, brighter color.

[1975] In this way, the user can easily understand the information intuitively, and in the virtual store, appropriate information can be provided for the product in which the user is interested, thereby increasing the user's desire to purchase.

[1976] The...

Claims

1. A means for acquiring audio data; means for converting the voice data into text data; A means for translating the text data into a plurality of languages ​​different from the language of the text data using generative AI; means for projecting the translated text data onto a predetermined surface; A system including:

2. The system of claim 1 , wherein the generative AI translating means translates the converted text data in real time.

3. means for recognizing an emotion of a user of the text data using an emotion recognition engine; a means for adjusting the content that is a translation result of the translating means and the display format of the displaying means according to the emotion recognized by the emotion recognizing means; The system of claim 1 further comprising:

Images