System

Abstract

To provide a system for achieving smooth communication across language barriers.SOLUTION: This system includes a means for acquiring a camera video and transmitting the video data to a server, a means for extracting a person's voice from the video data by the server and performing voice recognition to convert the voice data into text data, a means for analyzing the text data and the voice data by the server to analyze the emotion and the persona of an opposite party, a means for translating the text data by the server and transmitting the translated text data and the information on the emotion and the persona to a terminal, a means for displaying the translation result received by the terminal and the information on the emotion and the persona on the visual field of the user, a means for acquiring the voice of the user by the terminal and transmitting the voice to the server, a means for converting the voice data of the user into text data by the server and translating the text data, and a means for reproducing the text data converted by the server.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Language barriers currently hinder communication, resulting in the loss of opportunities for international business and human relationships. Furthermore, learning a foreign language requires significant effort and cost, and while instant translation is particularly necessary for real-time communication, conventional technology suffers from problems with latency and translation accuracy. Furthermore, the inability to grasp the other person's emotions and persona makes it difficult to communicate appropriately. There is a need to provide a system that overcomes these challenges and enables smooth communication across language barriers. [Means for solving the problem]

[0005] This invention provides a system that utilizes smart glasses and smartwatches and provides the following means: It includes a means for acquiring camera images and transmitting the video data to a server. The server includes a means for extracting a person's voice from the video data, performing voice recognition, and converting it into text data. It also includes a means for the server to analyze the text data and video data and analyze the other person's emotions and persona. It also provides a means for the server to translate the text data and transmit the translated text data and emotion / persona information to a terminal. It also includes a means for the terminal to display the translation results and emotion / persona information received in the user's field of view. In addition, the terminal includes a means for acquiring the user's voice and transmitting it to a server, and the server includes a means for converting the user's voice into text data and translating it. It includes a means for the server to convert the translated text data into audio data and transmit it to a terminal, and it also provides a means for the terminal to play the received audio data, thereby realizing a system that solves the above-mentioned problems.

[0006] "Camera image" is visual information data acquired by a camera device.

[0007] "Video data" refers to visual information captured by a camera expressed in digital form.

[0008] A "server" is a computer system that communicates with multiple terminals via a network and processes and manages data.

[0009] "Sound" is acoustic energy transmitted through vibrations in the air, including the human voice.

[0010] "Speech recognition" is a technology that converts voice data into text data that a computer can understand.

[0011] "Text data" is data that represents character information in digital form.

[0012] "Emotions" are psychological states and reactions that humans have to situations.

[0013] "Persona" is a concept that refers to a person's characteristics and personality as inferred from specific questions and actions.

[0014] "Translation" is the process of expressing text in one language in another language.

[0015] A "terminal" is a computing device (e.g., smart glasses, smartphone) that is directly operated by a user.

[0016] "Displaying in the field of view" refers to visualizing information within the range that the user can see.

[0017] "Speech synthesis" is a technology that converts text data into speech that sounds like a human voice. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] MODE FOR CARRYING OUT THE INVENTION

[0040] This invention is a system that combines camera-equipped smart glasses with a smartwatch (or smartphone) to perform real-time translation and emotion and persona analysis, helping users communicate smoothly across language barriers.

[0041] System Program Processing

[0042] This system operates in the following manner.

[0043] 1. Acquiring and transmitting camera images

[0044] The device (smart glasses) uses a built-in camera to capture images of the area in front of it in real time.

[0045] The terminal transmits the video data to the server via the network.

[0046] 2. Video analysis and audio extraction

[0047] The server analyzes the received video data and recognizes people in the video.

[0048] The server extracts the voice data and converts it into text data using a voice recognition engine.

[0049] 3. Emotional and persona analysis

[0050] The server analyzes emotions and personas based on text and video data.

[0051] Emotions are determined by analyzing facial expressions and tone of voice, while personas are inferred from speaking style and vocabulary used.

[0052] 4. Translation Processing

[0053] The server translates the text data obtained by speech recognition into the specified language.

[0054] The translation results and emotion / persona information are sent to the user's smart glasses device.

[0055] 5. Display of Information

[0056] The device (smart glasses) overlays the received translation results and emotion / persona information onto the user's field of vision.

[0057] Users can check the translation results and the other person's emotions and persona in real time.

[0058] 6. Audio input and transmission

[0059] Users talk to their smart glasses or smartphone.

[0060] The terminal captures the audio and transmits the audio data to the server.

[0061] 7. Audio translation and output

[0062] The server converts the voice data into text using a voice recognition engine and translates it into the specified language.

[0063] The translated text data is passed to a speech synthesis engine to generate speech data.

[0064] The server transmits the generated audio data to the smartphone and plays the audio.

[0065] Specific examples

[0066] Example 1: International business meeting

[0067] 1. The device captures the other party's image with its camera and sends it to the server.

[0068] 2. The server analyzes the video and audio and converts the audio into text data.

[0069] 3. The server analyzes the other person's emotions and persona based on the text and video data.

[0070] 4. The server translates the text data and sends the translation results, emotions, and persona information to the device.

[0071] 5. The device displays the translation results and emotion / persona information in the user's field of view.

[0072] 6. The user speaks into the smartphone, and the device sends the voice data to the server.

[0073] 7. The server translates the speech, generates speech data, and sends it to the device.

[0074] 8. The smartphone plays the generated audio and transmits it to the other person.

[0075] The above steps enable users and other parties to communicate in real time, overcoming language differences and effectively eliminating language barriers in international business and everyday conversations. Using this system, communication between multiple languages ​​can be carried out smoothly, and the emotions and intentions of the other party can be accurately understood.

[0076] The processing flow will be explained below.

[0077] Step 1:

[0078] The device (smart glasses) uses a built-in camera to capture real-time images.

[0079] The terminal encodes the acquired video data and transmits it to a server via a network.

[0080] Step 2:

[0081] The server decodes the received video data and uses a video analysis module to detect the movements of a person's face and mouth.

[0082] The server separates the audio data from the video and passes it to a voice recognition engine.

[0083] Step 3:

[0084] The server converts the voice data into text data using a voice recognition engine.

[0085] The server temporarily stores the generated text data in storage.

[0086] Step 4:

[0087] The server analyzes emotions and personas based on the text data and video data.

[0088] Sentiment analysis combines facial expression recognition and voice tone analysis to identify emotional states.

[0089] Persona analysis infers a person's characteristics from the content of their text and the way they speak.

[0090] Step 5:

[0091] The server translates the text data into the specified language in real time.

[0092] A translation module converts the text data into the user's native language.

[0093] The translated text data and emotion / persona information are packaged into packets, the data is encoded, and sent to the device.

[0094] Step 6:

[0095] The device (smart glasses) decodes the received data and overlays the translation results, emotion, and persona information onto the user's field of vision.

[0096] Users can check the translation results, the other person's emotions, and persona information in real time.

[0097] Step 7:

[0098] Users speak into their smart glasses or smartphone.

[0099] The terminal captures the audio data and transmits it to the server over the network.

[0100] Step 8:

[0101] The server converts the received voice data into text data using a voice recognition engine.

[0102] Translate text data into a specified language in real time.

[0103] The translated text data is passed to a speech synthesis engine to generate speech data.

[0104] Step 9:

[0105] The server encodes the generated voice data and transmits it to the terminal.

[0106] The device (smartphone) decodes the received audio data and plays it back through the speaker to transmit it to the other party.

[0107] In this way, real-time translation and emotional persona provision enable smooth communication between users who speak different languages.

[0108] Example 1

[0109] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0110] Conventional translation systems simply translate speech and text, and are insufficient to realize communication that takes into account the emotions and personality of the other party. This makes it difficult to accurately grasp the other party's intentions and emotions, especially in cross-cultural communication, and often leads to misunderstandings and anxiety.

[0111] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0112] In this invention, the server includes means for acquiring camera footage and transmitting the video data to the server, means for the server to extract a person's voice from the video data, perform voice recognition, and convert it into text data, means for the server to analyze the text data and video data and analyze the other person's emotions and personality, means for the server to translate the text data and transmit the translated text data and emotion / personality information to the terminal, means for displaying the translation result and emotion / personality information received by the terminal in the user's field of view, means for the terminal to acquire the user's voice and transmit it to the server, means for the server to convert the user's voice into text data and translate it, means for the server to convert the translated text data into audio data and transmit it to the terminal, and means for the terminal to play back the audio data received. This enables communication that not only provides translation results but also grasps the other person's emotions and personality in real time.

[0113] "Camera footage" is visible light information captured by a camera, and is data recorded as an image or video.

[0114] "Video data" refers to information stored in digital format from camera footage, and is expressed as an image or video file.

[0115] A "server" is a computer system that processes and stores data over a network, and is responsible for accepting and responding to requests from clients.

[0116] "Speech recognition" is a technology that analyzes speech and converts it into text data, recognizing spoken words as text information.

[0117] "Text data" refers to data that stores character information in a digital format and is expressed as language.

[0118] An "emotion" is a psychological reaction or state that a person shows to a particular situation, and can be anything from joy, sadness, surprise, etc.

[0119] "Personality" refers to the character and attitude that can be inferred from an individual's language and behavior patterns, and indicates the other person's personality and characteristics.

[0120] "Translation" is the act of converting text or audio expressed in one language into another, thereby conveying meaning between different languages.

[0121] A "terminal" is a computer system or device that is directly operated by a user, including smart glasses and smartphones.

[0122] A "user" is a person who communicates using this system and is the entity that operates each function of the system.

[0123] "Audio data" refers to information that stores audio in digital form and is expressed as an audio file.

[0124] "Playback" means reading stored data and outputting the original sound or video, making digital information into a form that can be confirmed visually or audibly.

[0125] This invention is a system that combines a camera-equipped device with a portable communication device to perform real-time translation and emotion and personality analysis. This system is designed to help users communicate smoothly with people who speak different languages.

[0126] System configuration

[0127] The system consists of the following main components:

[0128] 1. Camera-equipped devices (e.g., smart glasses)

[0129] 2. Portable communication devices (e.g., smartphones)

[0130] 3. Server Device

[0131] 4. Various software (e.g., facial recognition technology, voice recognition engine, emotion analysis engine, translation engine, voice synthesis engine)

[0132] Specific hardware and software

[0133] 1. Camera-equipped devices acquire images using the built-in camera. Use a high-resolution digital camera (e.g., 1080p camera).

[0134] 2. The portable communication device uses wireless communication technology (e.g., Wi-Fi or Bluetooth) to communicate data between the device and the server.

[0135] 3. The server equipment requires a high-performance processor and large-capacity memory, so it is possible to use a cloud server.

[0136] 4. The following software must be used:

[0137] OpenCV as a facial recognition technology

[0138] Google Cloud Speech-to-Text as the speech recognition engine

[0139] IBM Watson Tone Analyzer as a sentiment analysis engine

[0140] LIWC for personality inference

[0141] Google Translate API as a translation engine

[0142] Amazon Polly as a speech synthesis engine

[0143] Explanation of program processing

[0144] 1. Acquiring and transmitting camera images

[0145] The terminal uses a camera to capture images of the area in front of the user in real time, and transmits the image data to a server via a portable communication device.

[0146] 2. Video analysis and audio extraction

[0147] The server analyzes the transmitted video data and identifies people in the video using facial recognition technology, while simultaneously converting the audio synchronized with the video into text using a voice recognition engine.

[0148] 3. Emotional and personality analysis

[0149] The server uses an emotion analysis engine and personality estimation engine to analyze the other person's emotions and personality based on the acquired text data and facial recognition data.

[0150] 4. Translation Processing

[0151] The server translates the text data obtained by speech recognition into the language specified by the translation engine, and sends the translated text data along with emotion and personality information to the terminal.

[0152] 5. Display of Information

[0153] The device overlays the received translation results and emotional and personality information onto the user's field of vision, allowing the user to see what the other person is saying and how they are feeling in real time.

[0154] 6. Audio input and transmission

[0155] The user inputs their own speech by speaking into the device, and the terminal transmits this voice data to the server.

[0156] 7. Audio translation and output

[0157] The server converts the received voice data into text data using a voice recognition engine and translates it into the specified language. The translated text data is converted into voice data using a voice synthesis engine and sent to the device. The device then plays the received voice data over a speaker.

[0158] Examples and prompts

[0159] Below is a concrete example of how this system can be used in an international business meeting.

[0160] 1. The device captures the other party's image with its camera and sends it to the server.

[0161] 2. The server analyzes the video and audio and converts the audio into text data.

[0162] 3. The server analyzes the other person's emotions and personality based on the text and video data.

[0163] 4. The server translates the text data and sends the translation results, emotions, and personality information to the terminal.

[0164] 5. The device displays the translation results and emotional and personality information in the user's field of vision.

[0165] 6. The user speaks into the device and the voice data is sent to the server.

[0166] 7. The server translates the speech, generates speech data, and sends it to the device.

[0167] 8. The device plays the generated audio and transmits it to the other person.

[0168] Examples of prompt sentences include:

[0169] "Please explain in detail the camera image acquisition capabilities of this system. Please specify what hardware and software you use, and provide specific examples."

[0170] Based on the above, by using the system, users can communicate accurately and smoothly in real time with people who speak different languages.

[0171] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0172] Step 1:

[0173] The device (smart glasses) captures images of the front in real time using a built-in camera. The input is the image that appears in the user's field of vision. This is temporarily stored in internal memory as digital image data. This image data is then sent to a server via Wi-Fi or Bluetooth. The output is the digital image data sent to the server.

[0174] Step 2:

[0175] The server receives the transmitted video data. The input is digital video data. The server uses facial recognition technology (OpenCV) to identify people in the video. At the same time, it uses a speech recognition engine (Google Cloud Speech-to-Text) to convert the audio synchronized with the video into text data. The output is facial recognition data and text data.

[0176] Step 3:

[0177] The server uses an emotion analysis engine (IBM Watson Tone Analyzer) and a personality estimation engine (LIWC) to analyze the other person's emotions and personality based on the acquired text data and facial recognition data. The input is text data and facial recognition data. The output is emotion data and personality data. The server analyzes changes in facial expressions, tone of voice, patterns of words used, etc.

[0178] Step 4:

[0179] The server translates the text data obtained by speech recognition into the specified language using a translation engine (Google Translate API). The input is text data. The translated text data is then aggregated again together with emotion and personality information. The output is the translated text data and emotion / personality information.

[0180] Step 5:

[0181] The server sends the translation results and emotion / personality information to the device. The input is the translated text data and emotion / personality information. The device (smart glasses) overlays the received data in the user's field of vision. The input is the translated text data and emotion / personality information. The output is the translation results and emotion / personality information displayed in the user's field of vision.

[0182] Step 6:

[0183] The user speaks to the smart glasses or smartphone. The input is what the user says. The device captures the voice data and transmits it to the server via wireless communication. The output is the voice data transmitted to the server.

[0184] Step 7:

[0185] The server converts the received voice data into text data using a voice recognition engine. The input is the user's voice data. Next, it translates it into the specified language using a translation engine (Google Translate API). The output is the translated text data. The translated text data is converted into voice data using a voice synthesis engine (Amazon Polly). The output is the generated voice data. This voice data is sent to the terminal, and the terminal plays the received voice data. The input is the generated voice data. The output is the user's spoken content, which is played back as voice.

[0186] (Application example 1)

[0187] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0188] There is a need for a method to facilitate communication between passengers and systems within autonomous vehicles. In particular, it is necessary to provide real-time translation in multilingual environments and analyze passenger emotions and personalities to provide appropriate responses. Such a system would improve convenience for international users and promote the widespread adoption of autonomous vehicles. Furthermore, there is a need for a more efficient and intuitive interface to ensure passenger safety and comfort.

[0189] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0190] In this invention, the server includes a means for acquiring camera footage and transmitting the video data to the server, a means for the server to extract a person's voice from the video data, perform voice recognition, and convert the voice data into text data, and a means for the server to analyze the text data and video data and analyze the other person's emotions and personality. This enables smooth communication with passengers in an autonomous vehicle and real-time translation in a multilingual environment. Furthermore, understanding passenger emotions and personality and providing feedback based on this understanding improves passenger safety and comfort.

[0191] "Camera footage" is real-time visual data captured using a camera.

[0192] "Video data" refers to camera footage converted into a form that can be stored or transmitted in a digital format.

[0193] A "server" is a computer system that receives, analyzes, and transmits data over a network.

[0194] "Person's voice" refers to a voice uttered by a specific person, and is converted into text data by voice recognition.

[0195] "Speech recognition" is a technology that analyzes speech as a digital signal and converts it into text data.

[0196] "Text data" is character information converted from speech by speech recognition.

[0197] "Analysis" is the process of examining the received data in detail and extracting useful information.

[0198] "Emotion" is a psychological state that can be inferred by analyzing a person's facial expression and tone of voice.

[0199] "Personality" refers to a person's characteristics and tendencies that can be inferred based on their behavior, speaking style, and vocabulary used.

[0200] "Translation" is the process of converting text data from one language to another.

[0201] A "terminal" is a hardware device that is directly operated by a user and that displays data and plays audio.

[0202] "Displaying in the field of view" means overlaying information on the device screen or smart glasses display.

[0203] "Capture voice" means that the terminal captures the user's speech with a microphone.

[0204] "Audio data" refers to data obtained by converting audio captured by a microphone into a digital format.

[0205] "Speech synthesis" is a technology that generates synthetic speech from text data and converts it into reproducible speech data.

[0206] An "autonomous vehicle" is a vehicle that can drive autonomously without the operation of a human driver.

[0207] This invention provides a system for realizing smooth communication with passengers in autonomous vehicles. This system links smart glasses, a smartphone, and a server to perform translation and emotion and personality analysis in real time. A specific implementation method is described below.

[0208] System configuration

[0209] Hardware

[0210] 1. Smart Glasses

[0211] It has a built-in camera that captures and transmits images of the area ahead in real time.

[0212] It has a display function and overlays information onto the user's field of vision.

[0213] 2. Smartphone

[0214] It captures the audio and sends the data to the server.

[0215] Play the synthesized speech and output the translated information as audio.

[0216] 3. Server

[0217] A computer system for high-performance data analysis and communications.

[0218] software

[0219] 1. OpenCV (cv2)

[0220] Responsible for acquiring camera footage and processing the video data.

[0221] 2. Speech Recognition Engine

[0222] Convert the audio data into text data.

[0223] 3. Sentiment Analysis Engine

[0224] Analyze emotions from facial expressions and tone of voice.

[0225] 4. Translation Engine

[0226] Translates text obtained by speech recognition into a specified language.

[0227] 5. Speech synthesis engine

[0228] Text data is converted into audio and audio data that can be played on a smartphone is generated.

[0229] System Operation

[0230] The server receives camera footage sent from the smart glasses and analyzes the video data to extract the person's voice. The extracted voice is converted into text data using a voice recognition engine. The server then works with an emotion analysis engine to analyze emotions and personality based on the text and video data. At the same time, a translation engine translates the text data into the specified language. The translation results and emotion / personality information are sent to the smart glasses and displayed as an overlay in the user's field of vision.

[0231] When a user speaks into a smartphone, the voice data is sent to a server. The server converts this voice data into text data using a voice recognition engine and translates it using a translation engine. The translated text data is then converted into voice data using a speech synthesis engine and sent to the smartphone. The smartphone plays back the received voice data, allowing the user to communicate their intentions to the other party in real time.

[0232] Specific examples

[0233] As an example of the system, consider a situation in which an English-speaking passenger communicates with a Japanese-speaking passenger in an autonomous vehicle. The smart glasses capture video of the passenger with a camera and send it to a server. The server analyzes the video and audio data, generates text data from the speech, and analyzes emotions and personality. This information is translated and displayed on the smart glasses' display along with the translated text, "I am feeling anxious." If the passenger says, "I am feeling anxious," the smart glasses will display "I am feeling anxious." The user speaks into their smartphone, and their speech is translated into Japanese and transmitted to the other party.

[0234] Prompt Sentence Examples

[0235] "Please recognize the passenger's voice, analyze and translate their emotions, and display them in Japanese."

[0236] This system will facilitate smooth communication in multilingual environments inside autonomous vehicles and enable accurate understanding of passenger emotions and personalities, thereby improving passenger safety and comfort and promoting the use of autonomous vehicles.

[0237] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0238] Step 1:

[0239] The smart glasses use a camera to capture real-time video of passengers, and this video data is sent over a network to a server.

[0240] Input: Real-time video

[0241] Output: Video data sent to the server

[0242] How it works: The smart glasses use a camera to capture images of what's ahead, convert the data into packets, and send them to a server.

[0243] Step 2:

[0244] The server analyzes the received video data and identifies people in the video, while simultaneously extracting audio from the video and converting it into text using a speech recognition engine.

[0245] Input: Video data

[0246] Output: Speech-to-text data

[0247] Specific operation: The server uses video processing software to identify people from the video, extract audio data, and convert it into text using a voice recognition engine.

[0248] Step 3:

[0249] The server performs emotion and personality analysis based on the text data and video data obtained through speech recognition. An emotion analysis engine is used to analyze facial expressions and tone of voice, and personality is inferred from speaking style and vocabulary used.

[0250] Input: Text data, video data

[0251] Output: Emotion data, personality data

[0252] Specific operation: The server works in conjunction with the sentiment analysis engine to analyze emotions and personality using text data and video data as anchors.

[0253] Step 4:

[0254] The server uses a translation engine to translate the text data obtained by speech recognition into the specified language, and the translation results, along with emotion and personality information, are sent to the smart glasses.

[0255] Input: Text data, emotion data, personality data

[0256] Output: Translated text data, emotion data, personality data

[0257] Specific operation: The server uses a translation engine to translate text data in real time and transmits it to the smart glasses along with emotional and personality data.

[0258] Step 5:

[0259] The smart glasses overlay the received translation results and emotional and personality information onto the user's field of vision.

[0260] Input: translated text data, emotion data, personality data

[0261] Output: Information displayed in the user's field of view

[0262] How it works: The smart glasses use their display function to overlay the collected information onto the user's field of vision.

[0263] Step 6:

[0264] A user speaks into a smartphone, which captures the voice and sends the voice data to a server.

[0265] Input: User's voice

[0266] Output: Audio data sent to the server

[0267] Specific operation: The smartphone captures the user's voice with a microphone, converts the data into packets, and sends them to the server.

[0268] Step 7:

[0269] The server converts the received voice data into text data using a speech recognition engine, translates it into the specified language, and passes the translated text data to a speech synthesis engine, where it is converted into voice data.

[0270] Input: Audio data

[0271] Output: Speech-synthesized translation data

[0272] Specific operation: The server uses a speech recognition engine to convert speech into text, then translates it using a translation engine, and finally converts the translated data into audio using a speech synthesis engine.

[0273] Step 8:

[0274] The smartphone plays the voice data received from the server and conveys what the user said to the other party.

[0275] Input: Speech-synthesized translation data

[0276] Output: Played audio

[0277] Specific operation: The smartphone plays the audio data received from the server using the speaker.

[0278] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0279] MODE FOR CARRYING OUT THE INVENTION

[0280] This invention is a system that links camera-equipped smart glasses with a smartwatch (or smartphone) to perform real-time translation and analyze the emotions and personas of both parties. This system not only helps users communicate smoothly across language barriers, but also incorporates an emotion engine that can recognize and display the emotions of both parties to improve the quality of communication.

[0281] System Program Processing

[0282] This system operates in the following manner.

[0283] 1. Acquiring and transmitting camera images

[0284] The device (smart glasses) uses a built-in camera to capture real-time images.

[0285] The terminal encodes the acquired video data and transmits it to a server via a network.

[0286] 2. Video analysis and audio extraction

[0287] The server decodes the received video data and uses a video analysis module to detect the movements of a person's face and mouth.

[0288] The server separates the audio data from the video and passes it to a voice recognition engine.

[0289] 3. Speech Recognition and Text Conversion

[0290] The server converts the voice data into text data using a voice recognition engine.

[0291] The server temporarily stores the generated text data in storage.

[0292] 4. Emotional and persona analysis

[0293] The server analyzes emotions and personas based on the text data and video data.

[0294] Sentiment analysis combines facial expression recognition and voice tone analysis to identify emotional states.

[0295] Persona analysis infers a person's characteristics from the content of their text and the way they speak.

[0296] 5. User Emotion Recognition

[0297] The devices (smart glasses and smart watches) capture the user's video and audio data through their built-in cameras and microphones.

[0298] The device passes this data to an emotion engine to analyze the user's emotions.

[0299] The user's emotion data is transmitted to the server.

[0300] 6. Translation Processing

[0301] The server translates the text data obtained by speech recognition into the specified language in real time.

[0302] A translation module converts the text data into the user's native language.

[0303] The translated text data and emotion / persona information are packaged into packets, the data is encoded, and sent to the device.

[0304] 7. Display of Information

[0305] The device (smart glasses) decodes the received data and overlays the translation results and both parties' emotional and persona information onto the user's field of vision.

[0306] Users can check the translation results in real time, as well as the other person's and their own emotions and persona information.

[0307] 8. Audio Input and Transmission

[0308] Users speak into their smart glasses or smartphone.

[0309] The terminal captures the audio data and transmits it to the server over the network.

[0310] 9. Audio Translation and Output

[0311] The server converts the voice data into text using a voice recognition engine and translates it into the specified language in real time.

[0312] The translated text data is passed to a speech synthesis engine to generate speech data.

[0313] The server encodes the generated voice data and transmits it to the terminal.

[0314] The device (smartphone) decodes the received audio data and plays it back through the speaker to transmit it to the other party.

[0315] Specific examples

[0316] Example 1: International business meeting

[0317] 1. The device captures the other party's image with its camera and sends it to the server.

[0318] 2. The server analyzes the video and audio and converts the audio into text data.

[0319] 3. The server analyzes the other person's emotions and persona based on the text and video data.

[0320] 4. The device recognizes the user's emotions based on the video and audio captured and transmits the information to the server.

[0321] 5. The server translates the text data and sends the translation results, the other person's emotions, the user's emotions, and persona information to the terminal.

[0322] 6. The device displays the translation results and emotion / persona information in the user's field of view.

[0323] 7. The user speaks into the smartphone, and the device sends the voice data to the server.

[0324] 8. The server translates the speech, generates speech data, and sends it to the device.

[0325] 9. The smartphone plays the generated audio and transmits it to the other person.

[0326] This program processing enables real-time translation and communication while recognizing the emotions of both parties, improving the quality of communication.

[0327] The processing flow will be explained below.

[0328] Step 1:

[0329] The device (smart glasses) captures real-time video using a built-in camera, and the video data is encoded by a built-in encoding chip and sent to a server via a network.

[0330] Step 2:

[0331] The server decodes the received video data and uses a video analysis module to detect facial and mouth movements, thereby identifying the person in the video.

[0332] Step 3:

[0333] The server separates the audio data from the video and passes it to a speech recognition engine, where the audio is converted into text data.

[0334] Step 4:

[0335] The server temporarily stores the generated text data in storage, and simultaneously analyzes emotions and personas based on the text and video data. Emotion analysis is performed using facial expression recognition and voice tone analysis. Persona analysis estimates a person's characteristics from the content of the text and speaking style.

[0336] Step 5:

[0337] The devices (smart glasses and smart watches) capture the user's video and audio data, and use an emotion engine to recognize emotions based on the user's facial expressions and tone of voice, then send the results to the server.

[0338] Step 6:

[0339] The server translates the text data obtained by speech recognition into the specified language in real time. The translation module converts the text data into the user's native language. The translated text data and emotion / persona information are packaged into packets, encoded, and sent to the device.

[0340] Step 7:

[0341] The device (smart glasses) decodes the received data and overlays the translation results, as well as the other person's and the user's emotional and persona information, onto the user's field of vision, allowing the user to check the translation results and emotional and persona information in real time.

[0342] Step 8:

[0343] Users speak into their smart glasses or smartphone, which captures the voice data and transmits it to a server over a network.

[0344] Step 9:

[0345] The server converts the received voice data into text using a speech recognition engine, translates it into the specified language in real time, and passes the translated text data to a speech synthesis engine to generate voice data.

[0346] Step 10:

[0347] The server encodes the generated voice data and sends it to the device, which then decodes the received voice data and plays the translated voice over the speaker to communicate with the other party.

[0348] The above processing steps enable real-time two-way translation and emotion recognition / display, enabling smooth communication.

[0349] Example 2

[0350] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0351] In today's global society, there is a demand for smooth communication between people who speak different languages ​​and for improving the quality of that communication. However, current technology has difficulty not only overcoming language barriers, but also recognizing and adapting to the emotions and personas of both parties in real time. Furthermore, conventional systems do not provide real-time emotional feedback, which can lead to a poor user experience.

[0352] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for acquiring camera images and transmitting the image data to the server, a means for the server to extract a person's voice from the image data, perform voice recognition, and convert the voice data to text data, and a means for the server to analyze the text data and image data and analyze the other person's emotions and persona. This enables smooth communication between people who speak different languages ​​and real-time recognition and adaptation of each other's emotions and personas.

[0353] "Camera footage" is digital data of visual information captured by a camera device.

[0354] "Video data" refers to visual information captured as camera footage that has been converted into digital format.

[0355] A "server" is a computer system that receives, processes, and transmits data over a network.

[0356] A "terminal" is a device used by a user, and includes wearable devices such as smart glasses and smart watches.

[0357] "Person's voice" refers to acoustic data generated by speech such as conversation.

[0358] "Speech recognition" is a technology that analyzes voice data and converts it into text data.

[0359] "Text data" is character information generated by voice recognition.

[0360] "Emotion analysis" is a technology that identifies a person's emotional state based on video and audio data.

[0361] "Persona analysis" is a technique for estimating a person's characteristics and personality based on text data and speaking characteristics.

[0362] "Translation" is the process of converting text written in one language into another.

[0363] "Emotional feedback" is the process of analyzing the emotional state of the user and the other person and adapting in real time based on the results.

[0364] "Real-time" refers to immediate processing or response with little or no delay.

[0365] "Display in the field of view" is a technology in which a device overlays information in front of the user's line of sight.

[0366] MODE FOR CARRYING OUT THE INVENTION

[0367] This invention is a system that links camera-equipped devices (smart glasses, smart watches, or smartphones) to translate in real time and analyze both parties' emotions and personas. This system allows users to communicate smoothly across language barriers, and further improves the quality of communication by incorporating an emotion engine that recognizes and displays emotions on both sides.

[0368] Hardware and Software Overview

[0369] The device contains the following elements:

[0370] 1. Smart glasses: Equipped with a built-in camera, microphone, display, and Wi-Fi or data communication module.

[0371] 2. Smartwatch or smartphone: Equipped with a built-in camera, microphone, speaker, Wi-Fi or data communication module.

[0372] The server includes the following elements:

[0373] 1. Decoding and analysis module: FFmpeg, OpenCV

[0374] 2. Speech recognition engine: Google Speech-to-Text API

[0375] 3. Sentiment and persona analysis engine: IBM Watson Natural Language Understanding API, OpenCV

[0376] 4. Translation Module: Microsoft Translator Text API

[0377] 5. Speech synthesis engine: Amazon Polly

[0378] 6. Storage: AWS S3 and other cloud storage services

[0379] 7. Security protocols: TLS / SSL, etc.

[0380] System operation procedure

[0381] 1. Camera video and audio acquisition:

[0382] The device (smart glasses) uses a built-in camera to capture real-time video and a microphone to capture audio.

[0383] The footage is captured at 1080p resolution and encoded using the H.264 codec.

[0384] The encoded video data is sent to a server using Wi-Fi or a mobile network.

[0385] 2. Video analysis and audio extraction:

[0386] The server uses FFmpeg to decode the video and OpenCV for face detection and audio extraction.

[0387] The extracted audio data is converted into text data using the Google Speech-to-Text API.

[0388] 3. Emotional and persona analysis:

[0389] The server analyzes emotions and personas using IBM Watson Natural Language Understanding API and OpenCV based on text data and video data generated by the Google Speech-to-Text API.

[0390] 4. Translation Processing and Display:

[0391] The server translates the text data in real time using the Microsoft Translator Text API.

[0392] The translated text data and emotion / persona information are packaged into packets, encrypted using the TLS / SSL protocol, and sent to the device.

[0393] The device decodes the received data and displays it as an overlay in the user's field of view using Unity or ARCore.

[0394] 5. Audio feedback:

[0395] The device (smartphone) captures the user's speech with its built-in microphone and sends it to the server.

[0396] The server converts the speech to text and, after translation, generates the speech data using a speech synthesis engine.

[0397] The generated audio data is sent to the terminal and played through the speaker.

[0398] Examples and usage

[0399] Example 1: International business meeting

[0400] 1. The device captures the other party's image with its camera and sends it to the server.

[0401] 2. The server analyzes the video and audio and converts the audio into text data.

[0402] 3. The server analyzes the other person's emotions and persona based on the text and video data.

[0403] 4. The device recognizes the user's emotions based on the video and audio captured and transmits the information to the server.

[0404] 5. The server translates the text data and sends the translation results, the other person's emotions, the user's emotions, and persona information to the terminal.

[0405] 6. The device displays the translation results and emotion / persona information in the user's field of view.

[0406] 7. The user speaks into the smartphone, and the device sends the voice data to the server.

[0407] 8. The server translates the speech, generates speech data, and sends it to the device.

[0408] 9. The smartphone plays the generated audio and transmits it to the other person.

[0409] This program processing enables real-time translation and communication while recognizing the emotions of both parties, improving the quality of communication.

[0410] Examples of prompt statements

[0411] An example of a prompt sentence when generating an explanation of this system using a generative AI model is as follows:

[0412] Please explain in natural language how the following system works. Use either the server, terminal, or user as the subject, and clarify what hardware or software is used for each step and what data processing is performed. Explain with concrete examples.

[0413] (System Overview)

[0414] This system connects camera-equipped smart glasses with a smartwatch (or smartphone) to perform real-time translation and analysis of emotions and personas of both parties.The system incorporates an emotion engine and can recognize and display emotions of both parties.

[0415] Based on this prompt, the generative AI model can automatically generate in the specified format.

[0416] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0417] System program processing flow

[0418] Step 1:

[0419] The device (smart glasses) captures real-time video using its built-in camera. The captured video data is captured at 1080p resolution and 30 frames per second. The captured video is then encoded using the H.264 codec. The encoded video data is then sent to a server via Wi-Fi or mobile network.

[0420] Input: Real-time video from the user's field of view

[0421] Output: The encoded video data is sent to the server

[0422] Step 2:

[0423] The server uses FFmpeg to decode the received video data and OpenCV to detect facial and mouth movements, then the audio data is separated from the video data and passed to a speech recognition engine.

[0424] Input: Encoded video data

[0425] Output: Decoded video data, detected face and mouth movements, extracted audio data

[0426] Step 3:

[0427] The server converts the audio data into text data using the Google Speech-to-Text API, which is then saved in JSON format.

[0428] Input: Extracted audio data

[0429] Output: Text data (JSON format)

[0430] Step 4:

[0431] The server analyzes emotions and personas based on text and video data using IBM Watson Natural Language Understanding API and OpenCV. Emotion analysis is performed by combining facial expression recognition and voice tone analysis, while persona analysis estimates a person's characteristics from the content of the text and speaking style.

[0432] Input: Text data, video data

[0433] Output: Emotion data, persona data

[0434] Step 5:

[0435] The devices (smart glasses and smart watches) capture the user's video and audio data through their built-in cameras and microphones. This data is passed to the emotion engine, and the results of analyzing the user's emotions are sent to the server.

[0436] Input: User's video and audio data

[0437] Output: User emotion data is sent to the server

[0438] Step 6:

[0439] The server uses the Microsoft Translator Text API to translate the text data obtained by speech recognition into the specified language in real time. The translation module converts the text data into the user's native language, and then packages the translated text data and emotion / persona information into packets and sends them to the device using the TLS / SSL protocol.

[0440] Input: Text data, emotion data, persona data

[0441] Output: Translated text data, emotion data, and persona data are sent to the device.

[0442] Step 7:

[0443] The device (smart glasses) decodes the received data and overlays the translation results and both emotional and persona information onto the user's field of view. This is done using Unity or ARCore.

[0444] Input: Translated text data, emotion data, persona data

[0445] Output: Information overlaid on the user's field of view

[0446] Step 8:

[0447] Users speak into their smart glasses or smartphones, and the voice data is captured and sent over the network to a server.

[0448] Input: User's voice data

[0449] Output: Audio data sent to the server

[0450] Step 9:

[0451] The server converts the voice data into text using the Google Speech-to-Text API and translates it into the specified language in real time. The translated text data is then passed to the Amazon Polly speech synthesis engine to generate voice data. The generated voice data is encoded using the Opus codec, sent to the device, and played back through the smartphone speaker.

[0452] Input: Audio data sent to the server

[0453] Output: The generated and encoded audio data is sent to the device and played through the smartphone speaker.

[0454] (Application example 2)

[0455] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0456] The purpose of this invention is to facilitate communication between people who speak different languages ​​and achieve effective communication across language barriers. It also aims to improve the quality of communication by providing real-time emotion and persona information. In particular, it aims to solve the problem of providing fast and accurate customer service in brick-and-mortar stores and improving customer satisfaction.

[0457] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring camera video and transmitting the video data to the server, means for the server to extract the target's voice from the video data, perform voice recognition, and convert it into text data, means for the server to analyze the text data and video data and analyze the other party's emotions and personality, means for the server to translate the text data and transmit the translated text data and emotion / personality information to the terminal, means for the terminal to display the translation result and emotion / personality information received in the user's field of view, means for the terminal to acquire the user's voice and transmit it to the server, means for the server to convert the user's voice into text data and translate it, means for the server to convert the translated text data into audio data and transmit it to the terminal, means for the terminal to play the received audio data, means for the terminal to provide translations and emotional information to customers in the store, and means for the terminal to analyze and display the customer's interest and satisfaction with the product. This enables smooth communication and high-quality customer service that transcends language barriers.

[0458] "Camera image" refers to image data acquired by a device that converts light into electronic signals and records them as image information.

[0459] A "server" is a computer system that receives data from multiple devices over a network and processes, analyzes, stores, and distributes the data.

[0460] "Speech recognition" is a technology that analyzes voice data and converts it into text data.

[0461] "Text data" is digital data that contains text information.

[0462] "Translation" is the process of converting written or spoken content in one language into another.

[0463] "Emotion analysis" is a technology that analyzes a person's facial expressions, tone of voice, text content, etc. to identify their emotional state.

[0464] A "persona" is a model of a user's characteristics and behavioral patterns, represented as a specific role or character.

[0465] "Displaying in the field of view" means superimposing information onto the display of a device worn by the user (e.g., smart glasses).

[0466] "Audio data" refers to data in which audio is recorded in digital format.

[0467] A "terminal" is a device (e.g., smart glasses, smartphone) that is connected to a network and performs various inputs and outputs.

[0468] A "store" is a physical location that offers goods and services.

[0469] "Customer" means a person who purchases or uses goods or services.

[0470] "Translation result" refers to data input in an original language that has been subjected to a translation process and is output in another language.

[0471] "Interest and satisfaction with a product" is information that indicates the degree of interest and satisfaction that a customer has with a particular product or service.

[0472] This invention is a system for facilitating communication between people who speak different languages ​​and achieving effective communication across language barriers. It also aims to improve the quality of communication by providing real-time emotional and persona information. It is particularly intended to improve customer satisfaction by providing prompt and accurate customer service in brick-and-mortar stores.

[0473] This system is implemented using smart glasses or smartphones equipped with cameras. Specifically, the following process is performed:

[0474] First, the device (smart glasses or smartphone) captures camera images and sends the image data to a server. The server extracts the target audio from the captured image data, performs speech recognition, and converts it into text data. For speech recognition, it uses technologies such as Google Cloud Speech-to-Text.

[0475] The server then analyzes the text and video data to determine the other person's emotions and personality. Emotion analysis uses a combination of facial expression recognition and voice tone analysis. This includes the "emotion engine" and "personality analysis module."

[0476] The server translates the text data in real time based on the analysis results. The translation uses the Google Translate API and the Helsinki NLP translation model. The translated text data and emotion / personality information are then sent back to the device.

[0477] The device receives this information and displays it as an overlay in the user's field of vision. Specifically, the translation results and emotional information are overlaid on the smart glasses display, allowing the user to check the translation results, the other person's emotions, and their personality in real time.

[0478] The device then captures the user's voice and sends it to the server. The server converts the user's voice into text data and performs translation processing. The translated text data is converted into voice data and sent back to the device. The device then plays back the received voice data and transmits it to the other party.

[0479] A concrete example of this application is in brick-and-mortar stores. For example, when a foreign customer visits a store and asks a question about a product, the system instantly translates the question and relays it to the store staff. At the same time, the system can analyze the customer's satisfaction and interests in real time and provide this information to the staff. This enables smooth communication that transcends language barriers, improving customer satisfaction.

[0480] An example of a prompt is as follows:

[0481] Please generate a detailed description of the operation: "The server detects faces from video data and determines whether the customer is smiling or not by facial expression recognition." Please also describe the technology used (face detection algorithm, facial expression recognition model, etc.).

[0482] As described above, the present invention makes it possible to provide smooth communication and high-quality customer service that transcends language barriers.

[0483] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0484] Step 1:

[0485] The device (smartglasses or smartphone) captures camera images and sends the image data to a server. The input is the camera image data, and the output is the encoded image data sent to the server. Specifically, the camera built into the smartglasses captures images in real time, and the images are sent to the server via the network.

[0486] Step 2:

[0487] The server extracts audio from the transmitted video data and converts it into text using a speech recognition engine. The input is the audio information in the video data, and the output is text data. Specifically, the server's audio processing module analyzes the video data, separates the audio, and converts it into text using a speech recognition engine such as Google Cloud Speech-to-Text.

[0488] Step 3:

[0489] The server analyzes the generated text data and video data to analyze emotions and personality. The input is text data and video data, and the output is the analysis results of emotions and personality. Specifically, the emotion engine analyzes emotions based on facial expression data and voice tone, and the personality analysis module estimates the user's personality based on the content of the text and speaking style.

[0490] Step 4:

[0491] The server translates text data into a specified language. The input is the original text data, and the output is the translated text data. Specifically, the translation module in the server translates the text in real time using the Google Translate API and the Helsinki NLP translation model.

[0492] Step 5:

[0493] The server encodes the translated text data, emotion, and personality information and sends it to the terminal. The input is the translated text data and the emotion and personality analysis results, and the output is an encoded data packet. Specifically, the server assembles this information into a data packet and sends it to the user's terminal via the network.

[0494] Step 6:

[0495] The device decodes the translation results and emotion / personality information received from the server and overlays them on the user's field of view. The input is an encoded data packet, and the output is the translation results and emotion / personality information displayed on the user's field of view. Specifically, the smart glasses display overlays this information.

[0496] Step 7:

[0497] The device collects the user's voice and sends it to the server. The input is the user's voice, and the output is the voice data sent to the server. Specifically, the microphone in the smart glasses captures the user's speech and sends it to the server.

[0498] Step 8:

[0499] The server converts the user's speech into text data and performs the translation process. The input is speech data, and the output is translated text data. Specifically, the server's speech recognition engine converts the speech into text, and then translates the text in real time.

[0500] Step 9:

[0501] The server converts the translated text data into voice data and sends it to the terminal. The input is the translated text data, and the output is voice data. Specifically, the server's voice synthesis engine converts the text into voice data and sends it to the terminal via the network.

[0502] Step 10:

[0503] The device plays the received voice data and transmits it to the user. The input is voice data, and the output is the reproduced voice. Specifically, the smartphone speaker plays the voice and transmits it to the other party.

[0504] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0505] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0506] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0507] [Second embodiment]

[0508] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0509] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0510] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0511] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0512] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0513] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0514] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0515] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0516] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0517] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0518] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0519] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0520] MODE FOR CARRYING OUT THE INVENTION

[0521] This invention is a system that combines camera-equipped smart glasses with a smartwatch (or smartphone) to perform real-time translation and emotion and persona analysis, helping users communicate smoothly across language barriers.

[0522] System Program Processing

[0523] This system operates in the following manner.

[0524] 1. Acquiring and transmitting camera images

[0525] The device (smart glasses) uses a built-in camera to capture images of the area in front of it in real time.

[0526] The terminal transmits the video data to the server via the network.

[0527] 2. Video analysis and audio extraction

[0528] The server analyzes the received video data and recognizes people in the video.

[0529] The server extracts the voice data and converts it into text data using a voice recognition engine.

[0530] 3. Emotional and persona analysis

[0531] The server analyzes emotions and personas based on text and video data.

[0532] Emotions are determined by analyzing facial expressions and tone of voice, while personas are inferred from speaking style and vocabulary used.

[0533] 4. Translation Processing

[0534] The server translates the text data obtained by speech recognition into the specified language.

[0535] The translation results and emotion / persona information are sent to the user's smart glasses device.

[0536] 5. Display of Information

[0537] The device (smart glasses) overlays the received translation results and emotion / persona information onto the user's field of vision.

[0538] Users can check the translation results and the other person's emotions and persona in real time.

[0539] 6. Audio input and transmission

[0540] Users talk to their smart glasses or smartphone.

[0541] The terminal captures the audio and transmits the audio data to the server.

[0542] 7. Audio translation and output

[0543] The server converts the voice data into text using a voice recognition engine and translates it into the specified language.

[0544] The translated text data is passed to a speech synthesis engine to generate speech data.

[0545] The server transmits the generated audio data to the smartphone and plays the audio.

[0546] Specific examples

[0547] Example 1: International business meeting

[0548] 1. The device captures the other party's image with its camera and sends it to the server.

[0549] 2. The server analyzes the video and audio and converts the audio into text data.

[0550] 3. The server analyzes the other person's emotions and persona based on the text and video data.

[0551] 4. The server translates the text data and sends the translation results, emotions, and persona information to the device.

[0552] 5. The device displays the translation results and emotion / persona information in the user's field of view.

[0553] 6. The user speaks into the smartphone, and the device sends the voice data to the server.

[0554] 7. The server translates the speech, generates speech data, and sends it to the device.

[0555] 8. The smartphone plays the generated audio and transmits it to the other person.

[0556] The above steps enable users and other parties to communicate in real time, overcoming language differences and effectively eliminating language barriers in international business and everyday conversations. Using this system, communication between multiple languages ​​can be carried out smoothly, and the emotions and intentions of the other party can be accurately understood.

[0557] The processing flow will be explained below.

[0558] Step 1:

[0559] The device (smart glasses) uses a built-in camera to capture real-time images.

[0560] The terminal encodes the acquired video data and transmits it to a server via a network.

[0561] Step 2:

[0562] The server decodes the received video data and uses a video analysis module to detect the movements of a person's face and mouth.

[0563] The server separates the audio data from the video and passes it to a voice recognition engine.

[0564] Step 3:

[0565] The server converts the voice data into text data using a voice recognition engine.

[0566] The server temporarily stores the generated text data in storage.

[0567] Step 4:

[0568] The server analyzes emotions and personas based on the text data and video data.

[0569] Sentiment analysis combines facial expression recognition and voice tone analysis to identify emotional states.

[0570] Persona analysis infers a person's characteristics from the content of their text and the way they speak.

[0571] Step 5:

[0572] The server translates the text data into the specified language in real time.

[0573] A translation module converts the text data into the user's native language.

[0574] The translated text data and emotion / persona information are packaged into packets, the data is encoded, and sent to the device.

[0575] Step 6:

[0576] The device (smart glasses) decodes the received data and overlays the translation results, emotion, and persona information onto the user's field of vision.

[0577] Users can check the translation results, the other person's emotions, and persona information in real time.

[0578] Step 7:

[0579] Users speak into their smart glasses or smartphone.

[0580] The terminal captures the audio data and transmits it to the server over the network.

[0581] Step 8:

[0582] The server converts the received voice data into text data using a voice recognition engine.

[0583] Translate text data into a specified language in real time.

[0584] The translated text data is passed to a speech synthesis engine to generate speech data.

[0585] Step 9:

[0586] The server encodes the generated voice data and transmits it to the terminal.

[0587] The device (smartphone) decodes the received audio data and plays it back through the speaker to transmit it to the other party.

[0588] In this way, real-time translation and emotional persona provision enable smooth communication between users who speak different languages.

[0589] Example 1

[0590] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0591] Conventional translation systems simply translate speech and text, and are insufficient to realize communication that takes into account the emotions and personality of the other party. This makes it difficult to accurately grasp the other party's intentions and emotions, especially in cross-cultural communication, and often leads to misunderstandings and anxiety.

[0592] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0593] In this invention, the server includes means for acquiring camera footage and transmitting the video data to the server, means for the server to extract a person's voice from the video data, perform voice recognition, and convert it into text data, means for the server to analyze the text data and video data and analyze the other person's emotions and personality, means for the server to translate the text data and transmit the translated text data and emotion / personality information to the terminal, means for displaying the translation result and emotion / personality information received by the terminal in the user's field of view, means for the terminal to acquire the user's voice and transmit it to the server, means for the server to convert the user's voice into text data and translate it, means for the server to convert the translated text data into audio data and transmit it to the terminal, and means for the terminal to play back the audio data received. This enables communication that not only provides translation results but also grasps the other person's emotions and personality in real time.

[0594] "Camera footage" is visible light information captured by a camera, and is data recorded as an image or video.

[0595] "Video data" refers to information stored in digital format from camera footage, and is expressed as an image or video file.

[0596] A "server" is a computer system that processes and stores data over a network, and is responsible for accepting and responding to requests from clients.

[0597] "Speech recognition" is a technology that analyzes speech and converts it into text data, recognizing spoken words as text information.

[0598] "Text data" refers to data that stores character information in a digital format and is expressed as language.

[0599] An "emotion" is a psychological reaction or state that a person shows to a particular situation, and can be anything from joy, sadness, surprise, etc.

[0600] "Personality" refers to the character and attitude that can be inferred from an individual's language and behavior patterns, and indicates the other person's personality and characteristics.

[0601] "Translation" is the act of converting text or audio expressed in one language into another, thereby conveying meaning between different languages.

[0602] A "terminal" is a computer system or device that is directly operated by a user, including smart glasses and smartphones.

[0603] A "user" is a person who communicates using this system and is the entity that operates each function of the system.

[0604] "Audio data" refers to information that stores audio in digital form and is expressed as an audio file.

[0605] "Playback" means reading stored data and outputting the original sound or video, making digital information into a form that can be confirmed visually or audibly.

[0606] This invention is a system that combines a camera-equipped device with a portable communication device to perform real-time translation and emotion and personality analysis. This system is designed to help users communicate smoothly with people who speak different languages.

[0607] System configuration

[0608] The system consists of the following main components:

[0609] 1. Camera-equipped devices (e.g., smart glasses)

[0610] 2. Portable communication devices (e.g., smartphones)

[0611] 3. Server Device

[0612] 4. Various software (e.g., facial recognition technology, voice recognition engine, emotion analysis engine, translation engine, voice synthesis engine)

[0613] Specific hardware and software

[0614] 1. Camera-equipped devices acquire images using the built-in camera. Use a high-resolution digital camera (e.g., 1080p camera).

[0615] 2. The portable communication device uses wireless communication technology (e.g., Wi-Fi or Bluetooth) to communicate data between the device and the server.

[0616] 3. The server equipment requires a high-performance processor and large-capacity memory, so it is possible to use a cloud server.

[0617] 4. The following software must be used:

[0618] OpenCV as a facial recognition technology

[0619] Google Cloud Speech-to-Text as the speech recognition engine

[0620] IBM Watson Tone Analyzer as a sentiment analysis engine

[0621] LIWC for personality inference

[0622] Google Translate API as a translation engine

[0623] Amazon Polly as a speech synthesis engine

[0624] Explanation of program processing

[0625] 1. Acquiring and transmitting camera images

[0626] The terminal uses a camera to capture images of the area in front of the user in real time, and transmits the image data to a server via a portable communication device.

[0627] 2. Video analysis and audio extraction

[0628] The server analyzes the transmitted video data and identifies people in the video using facial recognition technology, while simultaneously converting the audio synchronized with the video into text using a voice recognition engine.

[0629] 3. Emotional and personality analysis

[0630] The server uses an emotion analysis engine and personality estimation engine to analyze the other person's emotions and personality based on the acquired text data and facial recognition data.

[0631] 4. Translation Processing

[0632] The server translates the text data obtained by speech recognition into the language specified by the translation engine, and sends the translated text data along with emotion and personality information to the terminal.

[0633] 5. Display of Information

[0634] The device overlays the received translation results and emotional and personality information onto the user's field of vision, allowing the user to see what the other person is saying and how they are feeling in real time.

[0635] 6. Audio input and transmission

[0636] The user inputs their own speech by speaking into the device, and the terminal transmits this voice data to the server.

[0637] 7. Audio translation and output

[0638] The server converts the received voice data into text data using a voice recognition engine and translates it into the specified language. The translated text data is converted into voice data using a voice synthesis engine and sent to the device. The device then plays the received voice data over a speaker.

[0639] Examples and prompts

[0640] Below is a concrete example of how this system can be used in an international business meeting.

[0641] 1. The device captures the other party's image with its camera and sends it to the server.

[0642] 2. The server analyzes the video and audio and converts the audio into text data.

[0643] 3. The server analyzes the other person's emotions and personality based on the text and video data.

[0644] 4. The server translates the text data and sends the translation results, emotions, and personality information to the terminal.

[0645] 5. The device displays the translation results and emotional and personality information in the user's field of vision.

[0646] 6. The user speaks into the device and the voice data is sent to the server.

[0647] 7. The server translates the speech, generates speech data, and sends it to the device.

[0648] 8. The device plays the generated audio and transmits it to the other person.

[0649] Examples of prompt sentences include:

[0650] "Please explain in detail the camera image acquisition capabilities of this system. Please specify what hardware and software you use, and provide specific examples."

[0651] Based on the above, by using the system, users can communicate accurately and smoothly in real time with people who speak different languages.

[0652] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0653] Step 1:

[0654] The device (smart glasses) captures images of the front in real time using a built-in camera. The input is the image that appears in the user's field of vision. This is temporarily stored in internal memory as digital image data. This image data is then sent to a server via Wi-Fi or Bluetooth. The output is the digital image data sent to the server.

[0655] Step 2:

[0656] The server receives the transmitted video data. The input is digital video data. The server uses facial recognition technology (OpenCV) to identify people in the video. At the same time, it uses a speech recognition engine (Google Cloud Speech-to-Text) to convert the audio synchronized with the video into text data. The output is facial recognition data and text data.

[0657] Step 3:

[0658] The server uses an emotion analysis engine (IBM Watson Tone Analyzer) and a personality estimation engine (LIWC) to analyze the other person's emotions and personality based on the acquired text data and facial recognition data. The input is text data and facial recognition data. The output is emotion data and personality data. The server analyzes changes in facial expressions, tone of voice, patterns of words used, etc.

[0659] Step 4:

[0660] The server translates the text data obtained by speech recognition into the specified language using a translation engine (Google Translate API). The input is text data. The translated text data is then aggregated again together with emotion and personality information. The output is the translated text data and emotion / personality information.

[0661] Step 5:

[0662] The server sends the translation results and emotion / personality information to the device. The input is the translated text data and emotion / personality information. The device (smart glasses) overlays the received data in the user's field of vision. The input is the translated text data and emotion / personality information. The output is the translation results and emotion / personality information displayed in the user's field of vision.

[0663] Step 6:

[0664] The user speaks to the smart glasses or smartphone. The input is what the user says. The device captures the voice data and transmits it to the server via wireless communication. The output is the voice data transmitted to the server.

[0665] Step 7:

[0666] The server converts the received voice data into text data using a voice recognition engine. The input is the user's voice data. Next, it translates it into the specified language using a translation engine (Google Translate API). The output is the translated text data. The translated text data is converted into voice data using a voice synthesis engine (Amazon Polly). The output is the generated voice data. This voice data is sent to the terminal, and the terminal plays the received voice data. The input is the generated voice data. The output is the user's spoken content, which is played back as voice.

[0667] (Application example 1)

[0668] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0669] There is a need for a method to facilitate communication between passengers and systems within autonomous vehicles. In particular, it is necessary to provide real-time translation in multilingual environments and analyze passenger emotions and personalities to provide appropriate responses. Such a system would improve convenience for international users and promote the widespread adoption of autonomous vehicles. Furthermore, there is a need for a more efficient and intuitive interface to ensure passenger safety and comfort.

[0670] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0671] In this invention, the server includes a means for acquiring camera footage and transmitting the video data to the server, a means for the server to extract a person's voice from the video data, perform voice recognition, and convert the voice data into text data, and a means for the server to analyze the text data and video data and analyze the other person's emotions and personality. This enables smooth communication with passengers in an autonomous vehicle and real-time translation in a multilingual environment. Furthermore, understanding passenger emotions and personality and providing feedback based on this understanding improves passenger safety and comfort.

[0672] "Camera footage" is real-time visual data captured using a camera.

[0673] "Video data" refers to camera footage converted into a form that can be stored or transmitted in a digital format.

[0674] A "server" is a computer system that receives, analyzes, and transmits data over a network.

[0675] "Person's voice" refers to a voice uttered by a specific person, and is converted into text data by voice recognition.

[0676] "Speech recognition" is a technology that analyzes speech as a digital signal and converts it into text data.

[0677] "Text data" is character information converted from speech by speech recognition.

[0678] "Analysis" is the process of examining the received data in detail and extracting useful information.

[0679] "Emotion" is a psychological state that can be inferred by analyzing a person's facial expression and tone of voice.

[0680] "Personality" refers to a person's characteristics and tendencies that can be inferred based on their behavior, speaking style, and vocabulary used.

[0681] "Translation" is the process of converting text data from one language to another.

[0682] A "terminal" is a hardware device that is directly operated by a user and that displays data and plays audio.

[0683] "Displaying in the field of view" means overlaying information on the device screen or smart glasses display.

[0684] "Capture voice" means that the terminal captures the user's speech with a microphone.

[0685] "Audio data" refers to data obtained by converting audio captured by a microphone into a digital format.

[0686] "Speech synthesis" is a technology that generates synthetic speech from text data and converts it into reproducible speech data.

[0687] An "autonomous vehicle" is a vehicle that can drive autonomously without the operation of a human driver.

[0688] This invention provides a system for realizing smooth communication with passengers in autonomous vehicles. This system links smart glasses, a smartphone, and a server to perform translation and emotion and personality analysis in real time. A specific implementation method is described below.

[0689] System configuration

[0690] Hardware

[0691] 1. Smart Glasses

[0692] It has a built-in camera that captures and transmits images of the area ahead in real time.

[0693] It has a display function and overlays information onto the user's field of vision.

[0694] 2. Smartphone

[0695] It captures the audio and sends the data to the server.

[0696] Play the synthesized speech and output the translated information as audio.

[0697] 3. Server

[0698] A computer system for high-performance data analysis and communications.

[0699] software

[0700] 1. OpenCV (cv2)

[0701] Responsible for acquiring camera footage and processing the video data.

[0702] 2. Speech Recognition Engine

[0703] Convert the audio data into text data.

[0704] 3. Sentiment Analysis Engine

[0705] Analyze emotions from facial expressions and tone of voice.

[0706] 4. Translation Engine

[0707] Translates text obtained by speech recognition into a specified language.

[0708] 5. Speech synthesis engine

[0709] Text data is converted into audio and audio data that can be played on a smartphone is generated.

[0710] System Operation

[0711] The server receives camera footage sent from the smart glasses and analyzes the video data to extract the person's voice. The extracted voice is converted into text data using a voice recognition engine. The server then works with an emotion analysis engine to analyze emotions and personality based on the text and video data. At the same time, a translation engine translates the text data into the specified language. The translation results and emotion / personality information are sent to the smart glasses and displayed as an overlay in the user's field of vision.

[0712] When a user speaks into a smartphone, the voice data is sent to a server. The server converts this voice data into text data using a voice recognition engine and translates it using a translation engine. The translated text data is then converted into voice data using a speech synthesis engine and sent to the smartphone. The smartphone plays back the received voice data, allowing the user to communicate their intentions to the other party in real time.

[0713] Specific examples

[0714] As an example of the system, consider a situation in which an English-speaking passenger communicates with a Japanese-speaking passenger in an autonomous vehicle. The smart glasses capture video of the passenger with a camera and send it to a server. The server analyzes the video and audio data, generates text data from the speech, and analyzes emotions and personality. This information is translated and displayed on the smart glasses' display along with the translated text, "I am feeling anxious." If the passenger says, "I am feeling anxious," the smart glasses will display "I am feeling anxious." The user speaks into their smartphone, and their speech is translated into Japanese and transmitted to the other party.

[0715] Prompt Sentence Examples

[0716] "Please recognize the passenger's voice, analyze and translate their emotions, and display them in Japanese."

[0717] This system will facilitate smooth communication in multilingual environments inside autonomous vehicles and enable accurate understanding of passenger emotions and personalities, thereby improving passenger safety and comfort and promoting the use of autonomous vehicles.

[0718] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0719] Step 1:

[0720] The smart glasses use a camera to capture real-time video of passengers, and this video data is sent over a network to a server.

[0721] Input: Real-time video

[0722] Output: Video data sent to the server

[0723] How it works: The smart glasses use a camera to capture images of what's ahead, convert the data into packets, and send them to a server.

[0724] Step 2:

[0725] The server analyzes the received video data and identifies people in the video, while simultaneously extracting audio from the video and converting it into text using a speech recognition engine.

[0726] Input: Video data

[0727] Output: Speech-to-text data

[0728] Specific operation: The server uses video processing software to identify people from the video, extract audio data, and convert it into text using a voice recognition engine.

[0729] Step 3:

[0730] The server performs emotion and personality analysis based on the text data and video data obtained through speech recognition. An emotion analysis engine is used to analyze facial expressions and tone of voice, and personality is inferred from speaking style and vocabulary used.

[0731] Input: Text data, video data

[0732] Output: Emotion data, personality data

[0733] Specific operation: The server works in conjunction with the sentiment analysis engine to analyze emotions and personality using text data and video data as anchors.

[0734] Step 4:

[0735] The server uses a translation engine to translate the text data obtained by speech recognition into the specified language, and the translation results, along with emotion and personality information, are sent to the smart glasses.

[0736] Input: Text data, emotion data, personality data

[0737] Output: Translated text data, emotion data, personality data

[0738] Specific operation: The server uses a translation engine to translate text data in real time and transmits it to the smart glasses along with emotional and personality data.

[0739] Step 5:

[0740] The smart glasses overlay the received translation results and emotional and personality information onto the user's field of vision.

[0741] Input: translated text data, emotion data, personality data

[0742] Output: Information displayed in the user's field of view

[0743] How it works: The smart glasses use their display function to overlay the collected information onto the user's field of vision.

[0744] Step 6:

[0745] A user speaks into a smartphone, which captures the voice and sends the voice data to a server.

[0746] Input: User's voice

[0747] Output: Audio data sent to the server

[0748] Specific operation: The smartphone captures the user's voice with a microphone, converts the data into packets, and sends them to the server.

[0749] Step 7:

[0750] The server converts the received voice data into text data using a speech recognition engine, translates it into the specified language, and passes the translated text data to a speech synthesis engine, where it is converted into voice data.

[0751] Input: Audio data

[0752] Output: Speech-synthesized translation data

[0753] Specific operation: The server uses a speech recognition engine to convert speech into text, then translates it using a translation engine, and finally converts the translated data into audio using a speech synthesis engine.

[0754] Step 8:

[0755] The smartphone plays the voice data received from the server and conveys what the user said to the other party.

[0756] Input: Speech-synthesized translation data

[0757] Output: Played audio

[0758] Specific operation: The smartphone plays the audio data received from the server using the speaker.

[0759] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0760] MODE FOR CARRYING OUT THE INVENTION

[0761] This invention is a system that links camera-equipped smart glasses with a smartwatch (or smartphone) to perform real-time translation and analyze the emotions and personas of both parties. This system not only helps users communicate smoothly across language barriers, but also incorporates an emotion engine that can recognize and display the emotions of both parties to improve the quality of communication.

[0762] System Program Processing

[0763] This system operates in the following manner.

[0764] 1. Acquiring and transmitting camera images

[0765] The device (smart glasses) uses a built-in camera to capture real-time images.

[0766] The terminal encodes the acquired video data and transmits it to a server via a network.

[0767] 2. Video analysis and audio extraction

[0768] The server decodes the received video data and uses a video analysis module to detect the movements of a person's face and mouth.

[0769] The server separates the audio data from the video and passes it to a voice recognition engine.

[0770] 3. Speech Recognition and Text Conversion

[0771] The server converts the voice data into text data using a voice recognition engine.

[0772] The server temporarily stores the generated text data in storage.

[0773] 4. Emotional and persona analysis

[0774] The server analyzes emotions and personas based on the text data and video data.

[0775] Sentiment analysis combines facial expression recognition and voice tone analysis to identify emotional states.

[0776] Persona analysis infers a person's characteristics from the content of their text and the way they speak.

[0777] 5. User Emotion Recognition

[0778] The devices (smart glasses and smart watches) capture the user's video and audio data through their built-in cameras and microphones.

[0779] The device passes this data to an emotion engine to analyze the user's emotions.

[0780] The user's emotion data is transmitted to the server.

[0781] 6. Translation Processing

[0782] The server translates the text data obtained by speech recognition into the specified language in real time.

[0783] A translation module converts the text data into the user's native language.

[0784] The translated text data and emotion / persona information are packaged into packets, the data is encoded, and sent to the device.

[0785] 7. Display of Information

[0786] The device (smart glasses) decodes the received data and overlays the translation results and both parties' emotional and persona information onto the user's field of vision.

[0787] Users can check the translation results in real time, as well as the other person's and their own emotions and persona information.

[0788] 8. Audio Input and Transmission

[0789] Users speak into their smart glasses or smartphone.

[0790] The terminal captures the audio data and transmits it to the server over the network.

[0791] 9. Audio Translation and Output

[0792] The server converts the voice data into text using a voice recognition engine and translates it into the specified language in real time.

[0793] The translated text data is passed to a speech synthesis engine to generate speech data.

[0794] The server encodes the generated voice data and transmits it to the terminal.

[0795] The device (smartphone) decodes the received audio data and plays it back through the speaker to transmit it to the other party.

[0796] Specific examples

[0797] Example 1: International business meeting

[0798] 1. The device captures the other party's image with its camera and sends it to the server.

[0799] 2. The server analyzes the video and audio and converts the audio into text data.

[0800] 3. The server analyzes the other person's emotions and persona based on the text and video data.

[0801] 4. The device recognizes the user's emotions based on the video and audio captured and transmits the information to the server.

[0802] 5. The server translates the text data and sends the translation results, the other person's emotions, the user's emotions, and persona information to the terminal.

[0803] 6. The device displays the translation results and emotion / persona information in the user's field of view.

[0804] 7. The user speaks into the smartphone, and the device sends the voice data to the server.

[0805] 8. The server translates the speech, generates speech data, and sends it to the device.

[0806] 9. The smartphone plays the generated audio and transmits it to the other person.

[0807] This program processing enables real-time translation and communication while recognizing the emotions of both parties, improving the quality of communication.

[0808] The processing flow will be explained below.

[0809] Step 1:

[0810] The device (smart glasses) captures real-time video using a built-in camera, and the video data is encoded by a built-in encoding chip and sent to a server via a network.

[0811] Step 2:

[0812] The server decodes the received video data and uses a video analysis module to detect facial and mouth movements, thereby identifying the person in the video.

[0813] Step 3:

[0814] The server separates the audio data from the video and passes it to a speech recognition engine, where the audio is converted into text data.

[0815] Step 4:

[0816] The server temporarily stores the generated text data in storage, and simultaneously analyzes emotions and personas based on the text and video data. Emotion analysis is performed using facial expression recognition and voice tone analysis. Persona analysis estimates a person's characteristics from the content of the text and speaking style.

[0817] Step 5:

[0818] The devices (smart glasses and smart watches) capture the user's video and audio data, and use an emotion engine to recognize emotions based on the user's facial expressions and tone of voice, then send the results to the server.

[0819] Step 6:

[0820] The server translates the text data obtained by speech recognition into the specified language in real time. The translation module converts the text data into the user's native language. The translated text data and emotion / persona information are packaged into packets, encoded, and sent to the device.

[0821] Step 7:

[0822] The device (smart glasses) decodes the received data and overlays the translation results, as well as the other person's and the user's emotional and persona information, onto the user's field of vision, allowing the user to check the translation results and emotional and persona information in real time.

[0823] Step 8:

[0824] Users speak into their smart glasses or smartphone, which captures the voice data and transmits it to a server over a network.

[0825] Step 9:

[0826] The server converts the received voice data into text using a speech recognition engine, translates it into the specified language in real time, and passes the translated text data to a speech synthesis engine to generate voice data.

[0827] Step 10:

[0828] The server encodes the generated voice data and sends it to the device, which then decodes the received voice data and plays the translated voice over the speaker to communicate with the other party.

[0829] The above processing steps enable real-time two-way translation and emotion recognition / display, enabling smooth communication.

[0830] Example 2

[0831] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0832] In today's global society, there is a demand for smooth communication between people who speak different languages ​​and for improving the quality of that communication. However, current technology has difficulty not only overcoming language barriers, but also recognizing and adapting to the emotions and personas of both parties in real time. Furthermore, conventional systems do not provide real-time emotional feedback, which can lead to a poor user experience.

[0833] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for acquiring camera images and transmitting the image data to the server, a means for the server to extract a person's voice from the image data, perform voice recognition, and convert the voice data to text data, and a means for the server to analyze the text data and image data and analyze the other person's emotions and persona. This enables smooth communication between people who speak different languages ​​and real-time recognition and adaptation of each other's emotions and personas.

[0834] "Camera footage" is digital data of visual information captured by a camera device.

[0835] "Video data" refers to visual information captured as camera footage that has been converted into digital format.

[0836] A "server" is a computer system that receives, processes, and transmits data over a network.

[0837] A "terminal" is a device used by a user, and includes wearable devices such as smart glasses and smart watches.

[0838] "Person's voice" refers to acoustic data generated by speech such as conversation.

[0839] "Speech recognition" is a technology that analyzes voice data and converts it into text data.

[0840] "Text data" is character information generated by voice recognition.

[0841] "Emotion analysis" is a technology that identifies a person's emotional state based on video and audio data.

[0842] "Persona analysis" is a technique for estimating a person's characteristics and personality based on text data and speaking characteristics.

[0843] "Translation" is the process of converting text written in one language into another.

[0844] "Emotional feedback" is the process of analyzing the emotional state of the user and the other person and adapting in real time based on the results.

[0845] "Real-time" refers to immediate processing or response with little or no delay.

[0846] "Display in the field of view" is a technology in which a device overlays information in front of the user's line of sight.

[0847] MODE FOR CARRYING OUT THE INVENTION

[0848] This invention is a system that links camera-equipped devices (smart glasses, smart watches, or smartphones) to translate in real time and analyze both parties' emotions and personas. This system allows users to communicate smoothly across language barriers, and further improves the quality of communication by incorporating an emotion engine that recognizes and displays emotions on both sides.

[0849] Hardware and Software Overview

[0850] The device contains the following elements:

[0851] 1. Smart glasses: Equipped with a built-in camera, microphone, display, and Wi-Fi or data communication module.

[0852] 2. Smartwatch or smartphone: Equipped with a built-in camera, microphone, speaker, Wi-Fi or data communication module.

[0853] The server includes the following elements:

[0854] 1. Decoding and analysis module: FFmpeg, OpenCV

[0855] 2. Speech recognition engine: Google Speech-to-Text API

[0856] 3. Sentiment and persona analysis engine: IBM Watson Natural Language Understanding API, OpenCV

[0857] 4. Translation Module: Microsoft Translator Text API

[0858] 5. Speech synthesis engine: Amazon Polly

[0859] 6. Storage: AWS S3 and other cloud storage services

[0860] 7. Security protocols: TLS / SSL, etc.

[0861] System operation procedure

[0862] 1. Camera video and audio acquisition:

[0863] The device (smart glasses) uses a built-in camera to capture real-time video and a microphone to capture audio.

[0864] The footage is captured at 1080p resolution and encoded using the H.264 codec.

[0865] The encoded video data is sent to a server using Wi-Fi or a mobile network.

[0866] 2. Video analysis and audio extraction:

[0867] The server uses FFmpeg to decode the video and OpenCV for face detection and audio extraction.

[0868] The extracted audio data is converted into text data using the Google Speech-to-Text API.

[0869] 3. Emotional and persona analysis:

[0870] The server analyzes emotions and personas using IBM Watson Natural Language Understanding API and OpenCV based on text data and video data generated by the Google Speech-to-Text API.

[0871] 4. Translation Processing and Display:

[0872] The server translates the text data in real time using the Microsoft Translator Text API.

[0873] The translated text data and emotion / persona information are packaged into packets, encrypted using the TLS / SSL protocol, and sent to the device.

[0874] The device decodes the received data and displays it as an overlay in the user's field of view using Unity or ARCore.

[0875] 5. Audio feedback:

[0876] The device (smartphone) captures the user's speech with its built-in microphone and sends it to the server.

[0877] The server converts the speech to text and, after translation, generates the speech data using a speech synthesis engine.

[0878] The generated audio data is sent to the terminal and played through the speaker.

[0879] Examples and usage

[0880] Example 1: International business meeting

[0881] 1. The device captures the other party's image with its camera and sends it to the server.

[0882] 2. The server analyzes the video and audio and converts the audio into text data.

[0883] 3. The server analyzes the other person's emotions and persona based on the text and video data.

[0884] 4. The device recognizes the user's emotions based on the video and audio captured and transmits the information to the server.

[0885] 5. The server translates the text data and sends the translation results, the other person's emotions, the user's emotions, and persona information to the terminal.

[0886] 6. The device displays the translation results and emotion / persona information in the user's field of view.

[0887] 7. The user speaks into the smartphone, and the device sends the voice data to the server.

[0888] 8. The server translates the speech, generates speech data, and sends it to the device.

[0889] 9. The smartphone plays the generated audio and transmits it to the other person.

[0890] This program processing enables real-time translation and communication while recognizing the emotions of both parties, improving the quality of communication.

[0891] Examples of prompt statements

[0892] An example of a prompt sentence when generating an explanation of this system using a generative AI model is as follows:

[0893] Please explain in natural language how the following system works. Use either the server, terminal, or user as the subject, and clarify what hardware or software is used for each step and what data processing is performed. Explain with concrete examples.

[0894] (System Overview)

[0895] This system connects camera-equipped smart glasses with a smartwatch (or smartphone) to perform real-time translation and analysis of emotions and personas of both parties.The system incorporates an emotion engine and can recognize and display emotions of both parties.

[0896] Based on this prompt, the generative AI model can automatically generate in the specified format.

[0897] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0898] System program processing flow

[0899] Step 1:

[0900] The device (smart glasses) captures real-time video using its built-in camera. The captured video data is captured at 1080p resolution and 30 frames per second. The captured video is then encoded using the H.264 codec. The encoded video data is then sent to a server via Wi-Fi or mobile network.

[0901] Input: Real-time video from the user's field of view

[0902] Output: The encoded video data is sent to the server

[0903] Step 2:

[0904] The server uses FFmpeg to decode the received video data and OpenCV to detect facial and mouth movements, then the audio data is separated from the video data and passed to a speech recognition engine.

[0905] Input: Encoded video data

[0906] Output: Decoded video data, detected face and mouth movements, extracted audio data

[0907] Step 3:

[0908] The server converts the audio data into text data using the Google Speech-to-Text API, which is then saved in JSON format.

[0909] Input: Extracted audio data

[0910] Output: Text data (JSON format)

[0911] Step 4:

[0912] The server analyzes emotions and personas based on text and video data using IBM Watson Natural Language Understanding API and OpenCV. Emotion analysis is performed by combining facial expression recognition and voice tone analysis, while persona analysis estimates a person's characteristics from the content of the text and speaking style.

[0913] Input: Text data, video data

[0914] Output: Emotion data, persona data

[0915] Step 5:

[0916] The devices (smart glasses and smart watches) capture the user's video and audio data through their built-in cameras and microphones. This data is passed to the emotion engine, and the results of analyzing the user's emotions are sent to the server.

[0917] Input: User's video and audio data

[0918] Output: User emotion data is sent to the server

[0919] Step 6:

[0920] The server uses the Microsoft Translator Text API to translate the text data obtained by speech recognition into the specified language in real time. The translation module converts the text data into the user's native language, and then packages the translated text data and emotion / persona information into packets and sends them to the device using the TLS / SSL protocol.

[0921] Input: Text data, emotion data, persona data

[0922] Output: Translated text data, emotion data, and persona data are sent to the device.

[0923] Step 7:

[0924] The device (smart glasses) decodes the received data and overlays the translation results and both emotional and persona information onto the user's field of view. This is done using Unity or ARCore.

[0925] Input: Translated text data, emotion data, persona data

[0926] Output: Information overlaid on the user's field of view

[0927] Step 8:

[0928] Users speak into their smart glasses or smartphones, and the voice data is captured and sent over the network to a server.

[0929] Input: User's voice data

[0930] Output: Audio data sent to the server

[0931] Step 9:

[0932] The server converts the voice data into text using the Google Speech-to-Text API and translates it into the specified language in real time. The translated text data is then passed to the Amazon Polly speech synthesis engine to generate voice data. The generated voice data is encoded using the Opus codec, sent to the device, and played back through the smartphone speaker.

[0933] Input: Audio data sent to the server

[0934] Output: The generated and encoded audio data is sent to the device and played through the smartphone speaker.

[0935] (Application example 2)

[0936] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0937] The purpose of this invention is to facilitate communication between people who speak different languages ​​and achieve effective communication across language barriers. It also aims to improve the quality of communication by providing real-time emotion and persona information. In particular, it aims to solve the problem of providing fast and accurate customer service in brick-and-mortar stores and improving customer satisfaction.

[0938] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring camera video and transmitting the video data to the server, means for the server to extract the target's voice from the video data, perform voice recognition, and convert it into text data, means for the server to analyze the text data and video data and analyze the other party's emotions and personality, means for the server to translate the text data and transmit the translated text data and emotion / personality information to the terminal, means for the terminal to display the translation result and emotion / personality information received in the user's field of view, means for the terminal to acquire the user's voice and transmit it to the server, means for the server to convert the user's voice into text data and translate it, means for the server to convert the translated text data into audio data and transmit it to the terminal, means for the terminal to play the received audio data, means for the terminal to provide translations and emotional information to customers in the store, and means for the terminal to analyze and display the customer's interest and satisfaction with the product. This enables smooth communication and high-quality customer service that transcends language barriers.

[0939] "Camera image" refers to image data acquired by a device that converts light into electronic signals and records them as image information.

[0940] A "server" is a computer system that receives data from multiple devices over a network and processes, analyzes, stores, and distributes the data.

[0941] "Speech recognition" is a technology that analyzes voice data and converts it into text data.

[0942] "Text data" is digital data that contains text information.

[0943] "Translation" is the process of converting written or spoken content in one language into another.

[0944] "Emotion analysis" is a technology that analyzes a person's facial expressions, tone of voice, text content, etc. to identify their emotional state.

[0945] A "persona" is a model of a user's characteristics and behavioral patterns, represented as a specific role or character.

[0946] "Displaying in the field of view" means superimposing information onto the display of a device worn by the user (e.g., smart glasses).

[0947] "Audio data" refers to data in which audio is recorded in digital format.

[0948] A "terminal" is a device (e.g., smart glasses, smartphone) that is connected to a network and performs various inputs and outputs.

[0949] A "store" is a physical location that offers goods and services.

[0950] "Customer" means a person who purchases or uses goods or services.

[0951] "Translation result" refers to data input in an original language that has been subjected to a translation process and is output in another language.

[0952] "Interest and satisfaction with a product" is information that indicates the degree of interest and satisfaction that a customer has with a particular product or service.

[0953] This invention is a system for facilitating communication between people who speak different languages ​​and achieving effective communication across language barriers. It also aims to improve the quality of communication by providing real-time emotional and persona information. It is particularly intended to improve customer satisfaction by providing prompt and accurate customer service in brick-and-mortar stores.

[0954] This system is implemented using smart glasses or smartphones equipped with cameras. Specifically, the following process is performed:

[0955] First, the device (smart glasses or smartphone) captures camera images and sends the image data to a server. The server extracts the target audio from the captured image data, performs speech recognition, and converts it into text data. For speech recognition, it uses technologies such as Google Cloud Speech-to-Text.

[0956] The server then analyzes the text and video data to determine the other person's emotions and personality. Emotion analysis uses a combination of facial expression recognition and voice tone analysis. This includes the "emotion engine" and "personality analysis module."

[0957] The server translates the text data in real time based on the analysis results. The translation uses the Google Translate API and the Helsinki NLP translation model. The translated text data and emotion / personality information are then sent back to the device.

[0958] The device receives this information and displays it as an overlay in the user's field of vision. Specifically, the translation results and emotional information are overlaid on the smart glasses display, allowing the user to check the translation results, the other person's emotions, and their personality in real time.

[0959] The device then captures the user's voice and sends it to the server. The server converts the user's voice into text data and performs translation processing. The translated text data is converted into voice data and sent back to the device. The device then plays back the received voice data and transmits it to the other party.

[0960] A concrete example of this application is in brick-and-mortar stores. For example, when a foreign customer visits a store and asks a question about a product, the system instantly translates the question and relays it to the store staff. At the same time, the system can analyze the customer's satisfaction and interests in real time and provide this information to the staff. This enables smooth communication that transcends language barriers, improving customer satisfaction.

[0961] An example of a prompt is as follows:

[0962] Please generate a detailed description of the operation: "The server detects faces from video data and determines whether the customer is smiling or not by facial expression recognition." Please also describe the technology used (face detection algorithm, facial expression recognition model, etc.).

[0963] As described above, the present invention makes it possible to provide smooth communication and high-quality customer service that transcends language barriers.

[0964] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0965] Step 1:

[0966] The device (smartglasses or smartphone) captures camera images and sends the image data to a server. The input is the camera image data, and the output is the encoded image data sent to the server. Specifically, the camera built into the smartglasses captures images in real time, and the images are sent to the server via the network.

[0967] Step 2:

[0968] The server extracts audio from the transmitted video data and converts it into text using a speech recognition engine. The input is the audio information in the video data, and the output is text data. Specifically, the server's audio processing module analyzes the video data, separates the audio, and converts it into text using a speech recognition engine such as Google Cloud Speech-to-Text.

[0969] Step 3:

[0970] The server analyzes the generated text data and video data to analyze emotions and personality. The input is text data and video data, and the output is the analysis results of emotions and personality. Specifically, the emotion engine analyzes emotions based on facial expression data and voice tone, and the personality analysis module estimates the user's personality based on the content of the text and speaking style.

[0971] Step 4:

[0972] The server translates text data into a specified language. The input is the original text data, and the output is the translated text data. Specifically, the translation module in the server translates the text in real time using the Google Translate API and the Helsinki NLP translation model.

[0973] Step 5:

[0974] The server encodes the translated text data, emotion, and personality information and sends it to the terminal. The input is the translated text data and the emotion and personality analysis results, and the output is an encoded data packet. Specifically, the server assembles this information into a data packet and sends it to the user's terminal via the network.

[0975] Step 6:

[0976] The device decodes the translation results and emotion / personality information received from the server and overlays them on the user's field of view. The input is an encoded data packet, and the output is the translation results and emotion / personality information displayed on the user's field of view. Specifically, the smart glasses display overlays this information.

[0977] Step 7:

[0978] The device collects the user's voice and sends it to the server. The input is the user's voice, and the output is the voice data sent to the server. Specifically, the microphone in the smart glasses captures the user's speech and sends it to the server.

[0979] Step 8:

[0980] The server converts the user's speech into text data and performs the translation process. The input is speech data, and the output is translated text data. Specifically, the server's speech recognition engine converts the speech into text, and then translates the text in real time.

[0981] Step 9:

[0982] The server converts the translated text data into voice data and sends it to the terminal. The input is the translated text data, and the output is voice data. Specifically, the server's voice synthesis engine converts the text into voice data and sends it to the terminal via the network.

[0983] Step 10:

[0984] The device plays the received voice data and transmits it to the user. The input is voice data, and the output is the reproduced voice. Specifically, the smartphone speaker plays the voice and transmits it to the other party.

[0985] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0986] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0987] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0988] [Third embodiment]

[0989] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0990] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0991] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0992] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0993] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0994] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0995] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0996] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0997] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0998] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0999] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1000] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1001] MODE FOR CARRYING OUT THE INVENTION

[1002] This invention is a system that combines camera-equipped smart glasses with a smartwatch (or smartphone) to perform real-time translation and emotion and persona analysis, helping users communicate smoothly across language barriers.

[1003] System Program Processing

[1004] This system operates in the following manner.

[1005] 1. Acquiring and transmitting camera images

[1006] The device (smart glasses) uses a built-in camera to capture images of the area in front of it in real time.

[1007] The terminal transmits the video data to the server via the network.

[1008] 2. Video analysis and audio extraction

[1009] The server analyzes the received video data and recognizes people in the video.

[1010] The server extracts the voice data and converts it into text data using a voice recognition engine.

[1011] 3. Emotional and persona analysis

[1012] The server analyzes emotions and personas based on text and video data.

[1013] Emotions are determined by analyzing facial expressions and tone of voice, while personas are inferred from speaking style and vocabulary used.

[1014] 4. Translation Processing

[1015] The server translates the text data obtained by speech recognition into the specified language.

[1016] The translation results and emotion / persona information are sent to the user's smart glasses device.

[1017] 5. Display of Information

[1018] The device (smart glasses) overlays the received translation results and emotion / persona information onto the user's field of vision.

[1019] Users can check the translation results and the other person's emotions and persona in real time.

[1020] 6. Audio input and transmission

[1021] Users talk to their smart glasses or smartphone.

[1022] The terminal captures the audio and transmits the audio data to the server.

[1023] 7. Audio translation and output

[1024] The server converts the voice data into text using a voice recognition engine and translates it into the specified language.

[1025] The translated text data is passed to a speech synthesis engine to generate speech data.

[1026] The server transmits the generated audio data to the smartphone and plays the audio.

[1027] Specific examples

[1028] Example 1: International business meeting

[1029] 1. The device captures the other party's image with its camera and sends it to the server.

[1030] 2. The server analyzes the video and audio and converts the audio into text data.

[1031] 3. The server analyzes the other person's emotions and persona based on the text and video data.

[1032] 4. The server translates the text data and sends the translation results, emotions, and persona information to the device.

[1033] 5. The device displays the translation results and emotion / persona information in the user's field of view.

[1034] 6. The user speaks into the smartphone, and the device sends the voice data to the server.

[1035] 7. The server translates the speech, generates speech data, and sends it to the device.

[1036] 8. The smartphone plays the generated audio and transmits it to the other person.

[1037] The above steps enable users and other parties to communicate in real time, overcoming language differences and effectively eliminating language barriers in international business and everyday conversations. Using this system, communication between multiple languages ​​can be carried out smoothly, and the emotions and intentions of the other party can be accurately understood.

[1038] The processing flow will be explained below.

[1039] Step 1:

[1040] The device (smart glasses) uses a built-in camera to capture real-time images.

[1041] The terminal encodes the acquired video data and transmits it to a server via a network.

[1042] Step 2:

[1043] The server decodes the received video data and uses a video analysis module to detect the movements of a person's face and mouth.

[1044] The server separates the audio data from the video and passes it to a voice recognition engine.

[1045] Step 3:

[1046] The server converts the voice data into text data using a voice recognition engine.

[1047] The server temporarily stores the generated text data in storage.

[1048] Step 4:

[1049] The server analyzes emotions and personas based on the text data and video data.

[1050] Sentiment analysis combines facial expression recognition and voice tone analysis to identify emotional states.

[1051] Persona analysis infers a person's characteristics from the content of their text and the way they speak.

[1052] Step 5:

[1053] The server translates the text data into the specified language in real time.

[1054] A translation module converts the text data into the user's native language.

[1055] The translated text data and emotion / persona information are packaged into packets, the data is encoded, and sent to the device.

[1056] Step 6:

[1057] The device (smart glasses) decodes the received data and overlays the translation results, emotion, and persona information onto the user's field of vision.

[1058] Users can check the translation results, the other person's emotions, and persona information in real time.

[1059] Step 7:

[1060] Users speak into their smart glasses or smartphone.

[1061] The terminal captures the audio data and transmits it to the server over the network.

[1062] Step 8:

[1063] The server converts the received voice data into text data using a voice recognition engine.

[1064] Translate text data into a specified language in real time.

[1065] The translated text data is passed to a speech synthesis engine to generate speech data.

[1066] Step 9:

[1067] The server encodes the generated voice data and transmits it to the terminal.

[1068] The device (smartphone) decodes the received audio data and plays it back through the speaker to transmit it to the other party.

[1069] In this way, real-time translation and emotional persona provision enable smooth communication between users who speak different languages.

[1070] Example 1

[1071] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1072] Conventional translation systems simply translate speech and text, and are insufficient to realize communication that takes into account the emotions and personality of the other party. This makes it difficult to accurately grasp the other party's intentions and emotions, especially in cross-cultural communication, and often leads to misunderstandings and anxiety.

[1073] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1074] In this invention, the server includes means for acquiring camera footage and transmitting the video data to the server, means for the server to extract a person's voice from the video data, perform voice recognition, and convert it into text data, means for the server to analyze the text data and video data and analyze the other person's emotions and personality, means for the server to translate the text data and transmit the translated text data and emotion / personality information to the terminal, means for displaying the translation result and emotion / personality information received by the terminal in the user's field of view, means for the terminal to acquire the user's voice and transmit it to the server, means for the server to convert the user's voice into text data and translate it, means for the server to convert the translated text data into audio data and transmit it to the terminal, and means for the terminal to play back the audio data received. This enables communication that not only provides translation results but also grasps the other person's emotions and personality in real time.

[1075] "Camera footage" is visible light information captured by a camera, and is data recorded as an image or video.

[1076] "Video data" refers to information stored in digital format from camera footage, and is expressed as an image or video file.

[1077] A "server" is a computer system that processes and stores data over a network, and is responsible for accepting and responding to requests from clients.

[1078] "Speech recognition" is a technology that analyzes speech and converts it into text data, recognizing spoken words as text information.

[1079] "Text data" refers to data that stores character information in a digital format and is expressed as language.

[1080] An "emotion" is a psychological reaction or state that a person shows to a particular situation, and can be anything from joy, sadness, surprise, etc.

[1081] "Personality" refers to the character and attitude that can be inferred from an individual's language and behavior patterns, and indicates the other person's personality and characteristics.

[1082] "Translation" is the act of converting text or audio expressed in one language into another, thereby conveying meaning between different languages.

[1083] A "terminal" is a computer system or device that is directly operated by a user, including smart glasses and smartphones.

[1084] A "user" is a person who communicates using this system and is the entity that operates each function of the system.

[1085] "Audio data" refers to information that stores audio in digital form and is expressed as an audio file.

[1086] "Playback" means reading stored data and outputting the original sound or video, making digital information into a form that can be confirmed visually or audibly.

[1087] This invention is a system that combines a camera-equipped device with a portable communication device to perform real-time translation and emotion and personality analysis. This system is designed to help users communicate smoothly with people who speak different languages.

[1088] System configuration

[1089] The system consists of the following main components:

[1090] 1. Camera-equipped devices (e.g., smart glasses)

[1091] 2. Portable communication devices (e.g., smartphones)

[1092] 3. Server Device

[1093] 4. Various software (e.g., facial recognition technology, voice recognition engine, emotion analysis engine, translation engine, voice synthesis engine)

[1094] Specific hardware and software

[1095] 1. Camera-equipped devices acquire images using the built-in camera. Use a high-resolution digital camera (e.g., 1080p camera).

[1096] 2. The portable communication device uses wireless communication technology (e.g., Wi-Fi or Bluetooth) to communicate data between the device and the server.

[1097] 3. The server equipment requires a high-performance processor and large-capacity memory, so it is possible to use a cloud server.

[1098] 4. The following software must be used:

[1099] OpenCV as a facial recognition technology

[1100] Google Cloud Speech-to-Text as the speech recognition engine

[1101] IBM Watson Tone Analyzer as a sentiment analysis engine

[1102] LIWC for personality inference

[1103] Google Translate API as a translation engine

[1104] Amazon Polly as a speech synthesis engine

[1105] Explanation of program processing

[1106] 1. Acquiring and transmitting camera images

[1107] The terminal uses a camera to capture images of the area in front of the user in real time, and transmits the image data to a server via a portable communication device.

[1108] 2. Video analysis and audio extraction

[1109] The server analyzes the transmitted video data and identifies people in the video using facial recognition technology, while simultaneously converting the audio synchronized with the video into text using a voice recognition engine.

[1110] 3. Emotional and personality analysis

[1111] The server uses an emotion analysis engine and personality estimation engine to analyze the other person's emotions and personality based on the acquired text data and facial recognition data.

[1112] 4. Translation Processing

[1113] The server translates the text data obtained by speech recognition into the language specified by the translation engine, and sends the translated text data along with emotion and personality information to the terminal.

[1114] 5. Display of Information

[1115] The device overlays the received translation results and emotional and personality information onto the user's field of vision, allowing the user to see what the other person is saying and how they are feeling in real time.

[1116] 6. Audio input and transmission

[1117] The user inputs their own speech by speaking into the device, and the terminal transmits this voice data to the server.

[1118] 7. Audio translation and output

[1119] The server converts the received voice data into text data using a voice recognition engine and translates it into the specified language. The translated text data is converted into voice data using a voice synthesis engine and sent to the device. The device then plays the received voice data over a speaker.

[1120] Examples and prompts

[1121] Below is a concrete example of how this system can be used in an international business meeting.

[1122] 1. The device captures the other party's image with its camera and sends it to the server.

[1123] 2. The server analyzes the video and audio and converts the audio into text data.

[1124] 3. The server analyzes the other person's emotions and personality based on the text and video data.

[1125] 4. The server translates the text data and sends the translation results, emotions, and personality information to the terminal.

[1126] 5. The device displays the translation results and emotional and personality information in the user's field of vision.

[1127] 6. The user speaks into the device and the voice data is sent to the server.

[1128] 7. The server translates the speech, generates speech data, and sends it to the device.

[1129] 8. The device plays the generated audio and transmits it to the other person.

[1130] Examples of prompt sentences include:

[1131] "Please explain in detail the camera image acquisition capabilities of this system. Please specify what hardware and software you use, and provide specific examples."

[1132] Based on the above, by using the system, users can communicate accurately and smoothly in real time with people who speak different languages.

[1133] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1134] Step 1:

[1135] The device (smart glasses) captures images of the front in real time using a built-in camera. The input is the image that appears in the user's field of vision. This is temporarily stored in internal memory as digital image data. This image data is then sent to a server via Wi-Fi or Bluetooth. The output is the digital image data sent to the server.

[1136] Step 2:

[1137] The server receives the transmitted video data. The input is digital video data. The server uses facial recognition technology (OpenCV) to identify people in the video. At the same time, it uses a speech recognition engine (Google Cloud Speech-to-Text) to convert the audio synchronized with the video into text data. The output is facial recognition data and text data.

[1138] Step 3:

[1139] The server uses an emotion analysis engine (IBM Watson Tone Analyzer) and a personality estimation engine (LIWC) to analyze the other person's emotions and personality based on the acquired text data and facial recognition data. The input is text data and facial recognition data. The output is emotion data and personality data. The server analyzes changes in facial expressions, tone of voice, patterns of words used, etc.

[1140] Step 4:

[1141] The server translates the text data obtained by speech recognition into the specified language using a translation engine (Google Translate API). The input is text data. The translated text data is then aggregated again together with emotion and personality information. The output is the translated text data and emotion / personality information.

[1142] Step 5:

[1143] The server sends the translation results and emotion / personality information to the device. The input is the translated text data and emotion / personality information. The device (smart glasses) overlays the received data in the user's field of vision. The input is the translated text data and emotion / personality information. The output is the translation results and emotion / personality information displayed in the user's field of vision.

[1144] Step 6:

[1145] The user speaks to the smart glasses or smartphone. The input is what the user says. The device captures the voice data and transmits it to the server via wireless communication. The output is the voice data transmitted to the server.

[1146] Step 7:

[1147] The server converts the received voice data into text data using a voice recognition engine. The input is the user's voice data. Next, it translates it into the specified language using a translation engine (Google Translate API). The output is the translated text data. The translated text data is converted into voice data using a voice synthesis engine (Amazon Polly). The output is the generated voice data. This voice data is sent to the terminal, and the terminal plays the received voice data. The input is the generated voice data. The output is the user's spoken content, which is played back as voice.

[1148] (Application example 1)

[1149] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1150] There is a need for a method to facilitate communication between passengers and systems within autonomous vehicles. In particular, it is necessary to provide real-time translation in multilingual environments and analyze passenger emotions and personalities to provide appropriate responses. Such a system would improve convenience for international users and promote the widespread adoption of autonomous vehicles. Furthermore, there is a need for a more efficient and intuitive interface to ensure passenger safety and comfort.

[1151] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1152] In this invention, the server includes a means for acquiring camera footage and transmitting the video data to the server, a means for the server to extract a person's voice from the video data, perform voice recognition, and convert the voice data into text data, and a means for the server to analyze the text data and video data and analyze the other person's emotions and personality. This enables smooth communication with passengers in an autonomous vehicle and real-time translation in a multilingual environment. Furthermore, understanding passenger emotions and personality and providing feedback based on this understanding improves passenger safety and comfort.

[1153] "Camera footage" is real-time visual data captured using a camera.

[1154] "Video data" refers to camera footage converted into a form that can be stored or transmitted in a digital format.

[1155] A "server" is a computer system that receives, analyzes, and transmits data over a network.

[1156] "Person's voice" refers to a voice uttered by a specific person, and is converted into text data by voice recognition.

[1157] "Speech recognition" is a technology that analyzes speech as a digital signal and converts it into text data.

[1158] "Text data" is character information converted from speech by speech recognition.

[1159] "Analysis" is the process of examining the received data in detail and extracting useful information.

[1160] "Emotion" is a psychological state that can be inferred by analyzing a person's facial expression and tone of voice.

[1161] "Personality" refers to a person's characteristics and tendencies that can be inferred based on their behavior, speaking style, and vocabulary used.

[1162] "Translation" is the process of converting text data from one language to another.

[1163] A "terminal" is a hardware device that is directly operated by a user and that displays data and plays audio.

[1164] "Displaying in the field of view" means overlaying information on the device screen or smart glasses display.

[1165] "Capture voice" means that the terminal captures the user's speech with a microphone.

[1166] "Audio data" refers to data obtained by converting audio captured by a microphone into a digital format.

[1167] "Speech synthesis" is a technology that generates synthetic speech from text data and converts it into reproducible speech data.

[1168] An "autonomous vehicle" is a vehicle that can drive autonomously without the operation of a human driver.

[1169] This invention provides a system for realizing smooth communication with passengers in autonomous vehicles. This system links smart glasses, a smartphone, and a server to perform translation and emotion and personality analysis in real time. A specific implementation method is described below.

[1170] System configuration

[1171] Hardware

[1172] 1. Smart Glasses

[1173] It has a built-in camera that captures and transmits images of the area ahead in real time.

[1174] It has a display function and overlays information onto the user's field of vision.

[1175] 2. Smartphone

[1176] It captures the audio and sends the data to the server.

[1177] Play the synthesized speech and output the translated information as audio.

[1178] 3. Server

[1179] A computer system for high-performance data analysis and communications.

[1180] software

[1181] 1. OpenCV (cv2)

[1182] Responsible for acquiring camera footage and processing the video data.

[1183] 2. Speech Recognition Engine

[1184] Convert the audio data into text data.

[1185] 3. Sentiment Analysis Engine

[1186] Analyze emotions from facial expressions and tone of voice.

[1187] 4. Translation Engine

[1188] Translates text obtained by speech recognition into a specified language.

[1189] 5. Speech synthesis engine

[1190] Text data is converted into audio and audio data that can be played on a smartphone is generated.

[1191] System Operation

[1192] The server receives camera footage sent from the smart glasses and analyzes the video data to extract the person's voice. The extracted voice is converted into text data using a voice recognition engine. The server then works with an emotion analysis engine to analyze emotions and personality based on the text and video data. At the same time, a translation engine translates the text data into the specified language. The translation results and emotion / personality information are sent to the smart glasses and displayed as an overlay in the user's field of vision.

[1193] When a user speaks into a smartphone, the voice data is sent to a server. The server converts this voice data into text data using a voice recognition engine and translates it using a translation engine. The translated text data is then converted into voice data using a speech synthesis engine and sent to the smartphone. The smartphone plays back the received voice data, allowing the user to communicate their intentions to the other party in real time.

[1194] Specific examples

[1195] As an example of the system, consider a situation in which an English-speaking passenger communicates with a Japanese-speaking passenger in an autonomous vehicle. The smart glasses capture video of the passenger with a camera and send it to a server. The server analyzes the video and audio data, generates text data from the speech, and analyzes emotions and personality. This information is translated and displayed on the smart glasses' display along with the translated text, "I am feeling anxious." If the passenger says, "I am feeling anxious," the smart glasses will display "I am feeling anxious." The user speaks into their smartphone, and their speech is translated into Japanese and transmitted to the other party.

[1196] Prompt Sentence Examples

[1197] "Please recognize the passenger's voice, analyze and translate their emotions, and display them in Japanese."

[1198] This system will facilitate smooth communication in multilingual environments inside autonomous vehicles and enable accurate understanding of passenger emotions and personalities, thereby improving passenger safety and comfort and promoting the use of autonomous vehicles.

[1199] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1200] Step 1:

[1201] The smart glasses use a camera to capture real-time video of passengers, and this video data is sent over a network to a server.

[1202] Input: Real-time video

[1203] Output: Video data sent to the server

[1204] How it works: The smart glasses use a camera to capture images of what's ahead, convert the data into packets, and send them to a server.

[1205] Step 2:

[1206] The server analyzes the received video data and identifies people in the video, while simultaneously extracting audio from the video and converting it into text using a speech recognition engine.

[1207] Input: Video data

[1208] Output: Speech-to-text data

[1209] Specific operation: The server uses video processing software to identify people from the video, extract audio data, and convert it into text using a voice recognition engine.

[1210] Step 3:

[1211] The server performs emotion and personality analysis based on the text data and video data obtained through speech recognition. An emotion analysis engine is used to analyze facial expressions and tone of voice, and personality is inferred from speaking style and vocabulary used.

[1212] Input: Text data, video data

[1213] Output: Emotion data, personality data

[1214] Specific operation: The server works in conjunction with the sentiment analysis engine to analyze emotions and personality using text data and video data as anchors.

[1215] Step 4:

[1216] The server uses a translation engine to translate the text data obtained by speech recognition into the specified language, and the translation results, along with emotion and personality information, are sent to the smart glasses.

[1217] Input: Text data, emotion data, personality data

[1218] Output: Translated text data, emotion data, personality data

[1219] Specific operation: The server uses a translation engine to translate text data in real time and transmits it to the smart glasses along with emotional and personality data.

[1220] Step 5:

[1221] The smart glasses overlay the received translation results and emotional and personality information onto the user's field of vision.

[1222] Input: translated text data, emotion data, personality data

[1223] Output: Information displayed in the user's field of view

[1224] How it works: The smart glasses use their display function to overlay the collected information onto the user's field of vision.

[1225] Step 6:

[1226] A user speaks into a smartphone, which captures the voice and sends the voice data to a server.

[1227] Input: User's voice

[1228] Output: Audio data sent to the server

[1229] Specific operation: The smartphone captures the user's voice with a microphone, converts the data into packets, and sends them to the server.

[1230] Step 7:

[1231] The server converts the received voice data into text data using a speech recognition engine, translates it into the specified language, and passes the translated text data to a speech synthesis engine, where it is converted into voice data.

[1232] Input: Audio data

[1233] Output: Speech-synthesized translation data

[1234] Specific operation: The server uses a speech recognition engine to convert speech into text, then translates it using a translation engine, and finally converts the translated data into audio using a speech synthesis engine.

[1235] Step 8:

[1236] The smartphone plays the voice data received from the server and conveys what the user said to the other party.

[1237] Input: Speech-synthesized translation data

[1238] Output: Played audio

[1239] Specific operation: The smartphone plays the audio data received from the server using the speaker.

[1240] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1241] MODE FOR CARRYING OUT THE INVENTION

[1242] This invention is a system that links camera-equipped smart glasses with a smartwatch (or smartphone) to perform real-time translation and analyze the emotions and personas of both parties. This system not only helps users communicate smoothly across language barriers, but also incorporates an emotion engine that can recognize and display the emotions of both parties to improve the quality of communication.

[1243] System Program Processing

[1244] This system operates in the following manner.

[1245] 1. Acquiring and transmitting camera images

[1246] The device (smart glasses) uses a built-in camera to capture real-time images.

[1247] The terminal encodes the acquired video data and transmits it to a server via a network.

[1248] 2. Video analysis and audio extraction

[1249] The server decodes the received video data and uses a video analysis module to detect the movements of a person's face and mouth.

[1250] The server separates the audio data from the video and passes it to a voice recognition engine.

[1251] 3. Speech Recognition and Text Conversion

[1252] The server converts the voice data into text data using a voice recognition engine.

[1253] The server temporarily stores the generated text data in storage.

[1254] 4. Emotional and persona analysis

[1255] The server analyzes emotions and personas based on the text data and video data.

[1256] Sentiment analysis combines facial expression recognition and voice tone analysis to identify emotional states.

[1257] Persona analysis infers a person's characteristics from the content of their text and the way they speak.

[1258] 5. User Emotion Recognition

[1259] The devices (smart glasses and smart watches) capture the user's video and audio data through their built-in cameras and microphones.

[1260] The device passes this data to an emotion engine to analyze the user's emotions.

[1261] The user's emotion data is transmitted to the server.

[1262] 6. Translation Processing

[1263] The server translates the text data obtained by speech recognition into the specified language in real time.

[1264] A translation module converts the text data into the user's native language.

[1265] The translated text data and emotion / persona information are packaged into packets, the data is encoded, and sent to the device.

[1266] 7. Display of Information

[1267] The device (smart glasses) decodes the received data and overlays the translation results and both parties' emotional and persona information onto the user's field of vision.

[1268] Users can check the translation results in real time, as well as the other person's and their own emotions and persona information.

[1269] 8. Audio Input and Transmission

[1270] Users speak into their smart glasses or smartphone.

[1271] The terminal captures the audio data and transmits it to the server over the network.

[1272] 9. Audio Translation and Output

[1273] The server converts the voice data into text using a voice recognition engine and translates it into the specified language in real time.

[1274] The translated text data is passed to a speech synthesis engine to generate speech data.

[1275] The server encodes the generated voice data and transmits it to the terminal.

[1276] The device (smartphone) decodes the received audio data and plays it back through the speaker to transmit it to the other party.

[1277] Specific examples

[1278] Example 1: International business meeting

[1279] 1. The device captures the other party's image with its camera and sends it to the server.

[1280] 2. The server analyzes the video and audio and converts the audio into text data.

[1281] 3. The server analyzes the other person's emotions and persona based on the text and video data.

[1282] 4. The device recognizes the user's emotions based on the video and audio captured and transmits the information to the server.

[1283] 5. The server translates the text data and sends the translation results, the other person's emotions, the user's emotions, and persona information to the terminal.

[1284] 6. The device displays the translation results and emotion / persona information in the user's field of view.

[1285] 7. The user speaks into the smartphone, and the device sends the voice data to the server.

[1286] 8. The server translates the speech, generates speech data, and sends it to the device.

[1287] 9. The smartphone plays the generated audio and transmits it to the other person.

[1288] This program processing enables real-time translation and communication while recognizing the emotions of both parties, improving the quality of communication.

[1289] The processing flow will be explained below.

[1290] Step 1:

[1291] The device (smart glasses) captures real-time video using a built-in camera, and the video data is encoded by a built-in encoding chip and sent to a server via a network.

[1292] Step 2:

[1293] The server decodes the received video data and uses a video analysis module to detect facial and mouth movements, thereby identifying the person in the video.

[1294] Step 3:

[1295] The server separates the audio data from the video and passes it to a speech recognition engine, where the audio is converted into text data.

[1296] Step 4:

[1297] The server temporarily stores the generated text data in storage, and simultaneously analyzes emotions and personas based on the text and video data. Emotion analysis is performed using facial expression recognition and voice tone analysis. Persona analysis estimates a person's characteristics from the content of the text and speaking style.

[1298] Step 5:

[1299] The devices (smart glasses and smart watches) capture the user's video and audio data, and use an emotion engine to recognize emotions based on the user's facial expressions and tone of voice, then send the results to the server.

[1300] Step 6:

[1301] The server translates the text data obtained by speech recognition into the specified language in real time. The translation module converts the text data into the user's native language. The translated text data and emotion / persona information are packaged into packets, encoded, and sent to the device.

[1302] Step 7:

[1303] The device (smart glasses) decodes the received data and overlays the translation results, as well as the other person's and the user's emotional and persona information, onto the user's field of vision, allowing the user to check the translation results and emotional and persona information in real time.

[1304] Step 8:

[1305] Users speak into their smart glasses or smartphone, which captures the voice data and transmits it to a server over a network.

[1306] Step 9:

[1307] The server converts the received voice data into text using a speech recognition engine, translates it into the specified language in real time, and passes the translated text data to a speech synthesis engine to generate voice data.

[1308] Step 10:

[1309] The server encodes the generated voice data and sends it to the device, which then decodes the received voice data and plays the translated voice over the speaker to communicate with the other party.

[1310] The above processing steps enable real-time two-way translation and emotion recognition / display, enabling smooth communication.

[1311] Example 2

[1312] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1313] In today's global society, there is a demand for smooth communication between people who speak different languages ​​and for improving the quality of that communication. However, current technology has difficulty not only overcoming language barriers, but also recognizing and adapting to the emotions and personas of both parties in real time. Furthermore, conventional systems do not provide real-time emotional feedback, which can lead to a poor user experience.

[1314] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for acquiring camera images and transmitting the image data to the server, a means for the server to extract a person's voice from the image data, perform voice recognition, and convert the voice data to text data, and a means for the server to analyze the text data and image data and analyze the other person's emotions and persona. This enables smooth communication between people who speak different languages ​​and real-time recognition and adaptation of each other's emotions and personas.

[1315] "Camera footage" is digital data of visual information captured by a camera device.

[1316] "Video data" refers to visual information captured as camera footage that has been converted into digital format.

[1317] A "server" is a computer system that receives, processes, and transmits data over a network.

[1318] A "terminal" is a device used by a user, and includes wearable devices such as smart glasses and smart watches.

[1319] "Person's voice" refers to acoustic data generated by speech such as conversation.

[1320] "Speech recognition" is a technology that analyzes voice data and converts it into text data.

[1321] "Text data" is character information generated by voice recognition.

[1322] "Emotion analysis" is a technology that identifies a person's emotional state based on video and audio data.

[1323] "Persona analysis" is a technique for estimating a person's characteristics and personality based on text data and speaking characteristics.

[1324] "Translation" is the process of converting text written in one language into another.

[1325] "Emotional feedback" is the process of analyzing the emotional state of the user and the other person and adapting in real time based on the results.

[1326] "Real-time" refers to immediate processing or response with little or no delay.

[1327] "Display in the field of view" is a technology in which a device overlays information in front of the user's line of sight.

[1328] MODE FOR CARRYING OUT THE INVENTION

[1329] This invention is a system that links camera-equipped devices (smart glasses, smart watches, or smartphones) to translate in real time and analyze both parties' emotions and personas. This system allows users to communicate smoothly across language barriers, and further improves the quality of communication by incorporating an emotion engine that recognizes and displays emotions on both sides.

[1330] Hardware and Software Overview

[1331] The device contains the following elements:

[1332] 1. Smart glasses: Equipped with a built-in camera, microphone, display, and Wi-Fi or data communication module.

[1333] 2. Smartwatch or smartphone: Equipped with a built-in camera, microphone, speaker, Wi-Fi or data communication module.

[1334] The server includes the following elements:

[1335] 1. Decoding and analysis module: FFmpeg, OpenCV

[1336] 2. Speech recognition engine: Google Speech-to-Text API

[1337] 3. Sentiment and persona analysis engine: IBM Watson Natural Language Understanding API, OpenCV

[1338] 4. Translation Module: Microsoft Translator Text API

[1339] 5. Speech synthesis engine: Amazon Polly

[1340] 6. Storage: AWS S3 and other cloud storage services

[1341] 7. Security protocols: TLS / SSL, etc.

[1342] System operation procedure

[1343] 1. Camera video and audio acquisition:

[1344] The device (smart glasses) uses a built-in camera to capture real-time video and a microphone to capture audio.

[1345] The footage is captured at 1080p resolution and encoded using the H.264 codec.

[1346] The encoded video data is sent to a server using Wi-Fi or a mobile network.

[1347] 2. Video analysis and audio extraction:

[1348] The server uses FFmpeg to decode the video and OpenCV for face detection and audio extraction.

[1349] The extracted audio data is converted into text data using the Google Speech-to-Text API.

[1350] 3. Emotional and persona analysis:

[1351] The server analyzes emotions and personas using IBM Watson Natural Language Understanding API and OpenCV based on text data and video data generated by the Google Speech-to-Text API.

[1352] 4. Translation Processing and Display:

[1353] The server translates the text data in real time using the Microsoft Translator Text API.

[1354] The translated text data and emotion / persona information are packaged into packets, encrypted using the TLS / SSL protocol, and sent to the device.

[1355] The device decodes the received data and displays it as an overlay in the user's field of view using Unity or ARCore.

[1356] 5. Audio feedback:

[1357] The device (smartphone) captures the user's speech with its built-in microphone and sends it to the server.

[1358] The server converts the speech to text and, after translation, generates the speech data using a speech synthesis engine.

[1359] The generated audio data is sent to the terminal and played through the speaker.

[1360] Examples and usage

[1361] Example 1: International business meeting

[1362] 1. The device captures the other party's image with its camera and sends it to the server.

[1363] 2. The server analyzes the video and audio and converts the audio into text data.

[1364] 3. The server analyzes the other person's emotions and persona based on the text and video data.

[1365] 4. The device recognizes the user's emotions based on the video and audio captured and transmits the information to the server.

[1366] 5. The server translates the text data and sends the translation results, the other person's emotions, the user's emotions, and persona information to the terminal.

[1367] 6. The device displays the translation results and emotion / persona information in the user's field of view.

[1368] 7. The user speaks into the smartphone, and the device sends the voice data to the server.

[1369] 8. The server translates the speech, generates speech data, and sends it to the device.

[1370] 9. The smartphone plays the generated audio and transmits it to the other person.

[1371] This program processing enables real-time translation and communication while recognizing the emotions of both parties, improving the quality of communication.

[1372] Examples of prompt statements

[1373] An example of a prompt sentence when generating an explanation of this system using a generative AI model is as follows:

[1374] Please explain in natural language how the following system works. Use either the server, terminal, or user as the subject, and clarify what hardware or software is used for each step and what data processing is performed. Explain with concrete examples.

[1375] (System Overview)

[1376] This system connects camera-equipped smart glasses with a smartwatch (or smartphone) to perform real-time translation and analysis of emotions and personas of both parties.The system incorporates an emotion engine and can recognize and display emotions of both parties.

[1377] Based on this prompt, the generative AI model can automatically generate in the specified format.

[1378] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1379] System program processing flow

[1380] Step 1:

[1381] The device (smart glasses) captures real-time video using its built-in camera. The captured video data is captured at 1080p resolution and 30 frames per second. The captured video is then encoded using the H.264 codec. The encoded video data is then sent to a server via Wi-Fi or mobile network.

[1382] Input: Real-time video from the user's field of view

[1383] Output: The encoded video data is sent to the server

[1384] Step 2:

[1385] The server uses FFmpeg to decode the received video data and OpenCV to detect facial and mouth movements, then the audio data is separated from the video data and passed to a speech recognition engine.

[1386] Input: Encoded video data

[1387] Output: Decoded video data, detected face and mouth movements, extracted audio data

[1388] Step 3:

[1389] The server converts the audio data into text data using the Google Speech-to-Text API, which is then saved in JSON format.

[1390] Input: Extracted audio data

[1391] Output: Text data (JSON format)

[1392] Step 4:

[1393] The server analyzes emotions and personas based on text and video data using IBM Watson Natural Language Understanding API and OpenCV. Emotion analysis is performed by combining facial expression recognition and voice tone analysis, while persona analysis estimates a person's characteristics from the content of the text and speaking style.

[1394] Input: Text data, video data

[1395] Output: Emotion data, persona data

[1396] Step 5:

[1397] The devices (smart glasses and smart watches) capture the user's video and audio data through their built-in cameras and microphones. This data is passed to the emotion engine, and the results of analyzing the user's emotions are sent to the server.

[1398] Input: User's video and audio data

[1399] Output: User emotion data is sent to the server

[1400] Step 6:

[1401] The server uses the Microsoft Translator Text API to translate the text data obtained by speech recognition into the specified language in real time. The translation module converts the text data into the user's native language, and then packages the translated text data and emotion / persona information into packets and sends them to the device using the TLS / SSL protocol.

[1402] Input: Text data, emotion data, persona data

[1403] Output: Translated text data, emotion data, and persona data are sent to the device.

[1404] Step 7:

[1405] The device (smart glasses) decodes the received data and overlays the translation results and both emotional and persona information onto the user's field of view. This is done using Unity or ARCore.

[1406] Input: Translated text data, emotion data, persona data

[1407] Output: Information overlaid on the user's field of view

[1408] Step 8:

[1409] Users speak into their smart glasses or smartphones, and the voice data is captured and sent over the network to a server.

[1410] Input: User's voice data

[1411] Output: Audio data sent to the server

[1412] Step 9:

[1413] The server converts the voice data into text using the Google Speech-to-Text API and translates it into the specified language in real time. The translated text data is then passed to the Amazon Polly speech synthesis engine to generate voice data. The generated voice data is encoded using the Opus codec, sent to the device, and played back through the smartphone speaker.

[1414] Input: Audio data sent to the server

[1415] Output: The generated and encoded audio data is sent to the device and played through the smartphone speaker.

[1416] (Application example 2)

[1417] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1418] The purpose of this invention is to facilitate communication between people who speak different languages ​​and achieve effective communication across language barriers. It also aims to improve the quality of communication by providing real-time emotion and persona information. In particular, it aims to solve the problem of providing fast and accurate customer service in brick-and-mortar stores and improving customer satisfaction.

[1419] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring camera video and transmitting the video data to the server, means for the server to extract the target's voice from the video data, perform voice recognition, and convert it into text data, means for the server to analyze the text data and video data and analyze the other party's emotions and personality, means for the server to translate the text data and transmit the translated text data and emotion / personality information to the terminal, means for the terminal to display the translation result and emotion / personality information received in the user's field of view, means for the terminal to acquire the user's voice and transmit it to the server, means for the server to convert the user's voice into text data and translate it, means for the server to convert the translated text data into audio data and transmit it to the terminal, means for the terminal to play the received audio data, means for the terminal to provide translations and emotional information to customers in the store, and means for the terminal to analyze and display the customer's interest and satisfaction with the product. This enables smooth communication and high-quality customer service that transcends language barriers.

[1420] "Camera image" refers to image data acquired by a device that converts light into electronic signals and records them as image information.

[1421] A "server" is a computer system that receives data from multiple devices over a network and processes, analyzes, stores, and distributes the data.

[1422] "Speech recognition" is a technology that analyzes voice data and converts it into text data.

[1423] "Text data" is digital data that contains text information.

[1424] "Translation" is the process of converting written or spoken content in one language into another.

[1425] "Emotion analysis" is a technology that analyzes a person's facial expressions, tone of voice, text content, etc. to identify their emotional state.

[1426] A "persona" is a model of a user's characteristics and behavioral patterns, represented as a specific role or character.

[1427] "Displaying in the field of view" means superimposing information onto the display of a device worn by the user (e.g., smart glasses).

[1428] "Audio data" refers to data in which audio is recorded in digital format.

[1429] A "terminal" is a device (e.g., smart glasses, smartphone) that is connected to a network and performs various inputs and outputs.

[1430] A "store" is a physical location that offers goods and services.

[1431] "Customer" means a person who purchases or uses goods or services.

[1432] "Translation result" refers to data input in an original language that has been subjected to a translation process and is output in another language.

[1433] "Interest and satisfaction with a product" is information that indicates the degree of interest and satisfaction that a customer has with a particular product or service.

[1434] This invention is a system for facilitating communication between people who speak different languages ​​and achieving effective communication across language barriers. It also aims to improve the quality of communication by providing real-time emotional and persona information. It is particularly intended to improve customer satisfaction by providing prompt and accurate customer service in brick-and-mortar stores.

[1435] This system is implemented using smart glasses or smartphones equipped with cameras. Specifically, the following process is performed:

[1436] First, the device (smart glasses or smartphone) captures camera images and sends the image data to a server. The server extracts the target audio from the captured image data, performs speech recognition, and converts it into text data. For speech recognition, it uses technologies such as Google Cloud Speech-to-Text.

[1437] The server then analyzes the text and video data to determine the other person's emotions and personality. Emotion analysis uses a combination of facial expression recognition and voice tone analysis. This includes the "emotion engine" and "personality analysis module."

[1438] The server translates the text data in real time based on the analysis results. The translation uses the Google Translate API and the Helsinki NLP translation model. The translated text data and emotion / personality information are then sent back to the device.

[1439] The device receives this information and displays it as an overlay in the user's field of vision. Specifically, the translation results and emotional information are overlaid on the smart glasses display, allowing the user to check the translation results, the other person's emotions, and their personality in real time.

[1440] The device then captures the user's voice and sends it to the server. The server converts the user's voice into text data and performs translation processing. The translated text data is converted into voice data and sent back to the device. The device then plays back the received voice data and transmits it to the other party.

[1441] A concrete example of this application is in brick-and-mortar stores. For example, when a foreign customer visits a store and asks a question about a product, the system instantly translates the question and relays it to the store staff. At the same time, the system can analyze the customer's satisfaction and interests in real time and provide this information to the staff. This enables smooth communication that transcends language barriers, improving customer satisfaction.

[1442] An example of a prompt is as follows:

[1443] Please generate a detailed description of the operation: "The server detects faces from video data and determines whether the customer is smiling or not by facial expression recognition." Please also describe the technology used (face detection algorithm, facial expression recognition model, etc.).

[1444] As described above, the present invention makes it possible to provide smooth communication and high-quality customer service that transcends language barriers.

[1445] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1446] Step 1:

[1447] The device (smartglasses or smartphone) captures camera images and sends the image data to a server. The input is the camera image data, and the output is the encoded image data sent to the server. Specifically, the camera built into the smartglasses captures images in real time, and the images are sent to the server via the network.

[1448] Step 2:

[1449] The server extracts audio from the transmitted video data and converts it into text using a speech recognition engine. The input is the audio information in the video data, and the output is text data. Specifically, the server's audio processing module analyzes the video data, separates the audio, and converts it into text using a speech recognition engine such as Google Cloud Speech-to-Text.

[1450] Step 3:

[1451] The server analyzes the generated text data and video data to analyze emotions and personality. The input is text data and video data, and the output is the analysis results of emotions and personality. Specifically, the emotion engine analyzes emotions based on facial expression data and voice tone, and the personality analysis module estimates the user's personality based on the content of the text and speaking style.

[1452] Step 4:

[1453] The server translates text data into a specified language. The input is the original text data, and the output is the translated text data. Specifically, the translation module in the server translates the text in real time using the Google Translate API and the Helsinki NLP translation model.

[1454] Step 5:

[1455] The server encodes the translated text data, emotion, and personality information and sends it to the terminal. The input is the translated text data and the emotion and personality analysis results, and the output is an encoded data packet. Specifically, the server assembles this information into a data packet and sends it to the user's terminal via the network.

[1456] Step 6:

[1457] The device decodes the translation results and emotion / personality information received from the server and overlays them on the user's field of view. The input is an encoded data packet, and the output is the translation results and emotion / personality information displayed on the user's field of view. Specifically, the smart glasses display overlays this information.

[1458] Step 7:

[1459] The device collects the user's voice and sends it to the server. The input is the user's voice, and the output is the voice data sent to the server. Specifically, the microphone in the smart glasses captures the user's speech and sends it to the server.

[1460] Step 8:

[1461] The server converts the user's speech into text data and performs the translation process. The input is speech data, and the output is translated text data. Specifically, the server's speech recognition engine converts the speech into text, and then translates the text in real time.

[1462] Step 9:

[1463] The server converts the translated text data into voice data and sends it to the terminal. The input is the translated text data, and the output is voice data. Specifically, the server's voice synthesis engine converts the text into voice data and sends it to the terminal via the network.

[1464] Step 10:

[1465] The device plays the received voice data and transmits it to the user. The input is voice data, and the output is the reproduced voice. Specifically, the smartphone speaker plays the voice and transmits it to the other party.

[1466] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1467] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1468] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1469] [Fourth embodiment]

[1470] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1471] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1472] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1473] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1474] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1475] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1476] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1477] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1478] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1479] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1480] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1481] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1482] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1483] MODE FOR CARRYING OUT THE INVENTION

[1484] This invention is a system that combines camera-equipped smart glasses with a smartwatch (or smartphone) to perform real-time translation and emotion and persona analysis, helping users communicate smoothly across language barriers.

[1485] System Program Processing

[1486] This system operates in the following manner.

[1487] 1. Acquiring and transmitting camera images

[1488] The device (smart glasses) uses a built-in camera to capture images of the area in front of it in real time.

[1489] The terminal transmits the video data to the server via the network.

[1490] 2. Video analysis and audio extraction

[1491] The server analyzes the received video data and recognizes people in the video.

[1492] The server extracts the voice data and converts it into text data using a voice recognition engine.

[1493] 3. Emotional and persona analysis

[1494] The server analyzes emotions and personas based on text and video data.

[1495] Emotions are determined by analyzing facial expressions and tone of voice, while personas are inferred from speaking style and vocabulary used.

[1496] 4. Translation Processing

[1497] The server translates the text data obtained by speech recognition into the specified language.

[1498] The translation results and emotion / persona information are sent to the user's smart glasses device.

[1499] 5. Display of Information

[1500] The device (smart glasses) overlays the received translation results and emotion / persona information onto the user's field of vision.

[1501] Users can check the translation results and the other person's emotions and persona in real time.

[1502] 6. Audio input and transmission

[1503] Users talk to their smart glasses or smartphone.

[1504] The terminal captures the audio and transmits the audio data to the server.

[1505] 7. Audio translation and output

[1506] The server converts the voice data into text using a voice recognition engine and translates it into the specified language.

[1507] The translated text data is passed to a speech synthesis engine to generate speech data.

[1508] The server transmits the generated audio data to the smartphone and plays the audio.

[1509] Specific examples

[1510] Example 1: International business meeting

[1511] 1. The device captures the other party's image with its camera and sends it to the server.

[1512] 2. The server analyzes the video and audio and converts the audio into text data.

[1513] 3. The server analyzes the other person's emotions and persona based on the text and video data.

[1514] 4. The server translates the text data and sends the translation results, emotions, and persona information to the device.

[1515] 5. The device displays the translation results and emotion / persona information in the user's field of view.

[1516] 6. The user speaks into the smartphone, and the device sends the voice data to the server.

[1517] 7. The server translates the speech, generates speech data, and sends it to the device.

[1518] 8. The smartphone plays the generated audio and transmits it to the other person.

[1519] The above steps enable users and other parties to communicate in real time, overcoming language differences and effectively eliminating language barriers in international business and everyday conversations. Using this system, communication between multiple languages ​​can be carried out smoothly, and the emotions and intentions of the other party can be accurately understood.

[1520] The processing flow will be explained below.

[1521] Step 1:

[1522] The device (smart glasses) uses a built-in camera to capture real-time images.

[1523] The terminal encodes the acquired video data and transmits it to a server via a network.

[1524] Step 2:

[1525] The server decodes the received video data and uses a video analysis module to detect the movements of a person's face and mouth.

[1526] The server separates the audio data from the video and passes it to a voice recognition engine.

[1527] Step 3:

[1528] The server converts the voice data into text data using a voice recognition engine.

[1529] The server temporarily stores the generated text data in storage.

[1530] Step 4:

[1531] The server analyzes emotions and personas based on the text data and video data.

[1532] Sentiment analysis combines facial expression recognition and voice tone analysis to identify emotional states.

[1533] Persona analysis infers a person's characteristics from the content of their text and the way they speak.

[1534] Step 5:

[1535] The server translates the text data into the specified language in real time.

[1536] A translation module converts the text data into the user's native language.

[1537] The translated text data and emotion / persona information are packaged into packets, the data is encoded, and sent to the device.

[1538] Step 6:

[1539] The device (smart glasses) decodes the received data and overlays the translation results, emotion, and persona information onto the user's field of vision.

[1540] Users can check the translation results, the other person's emotions, and persona information in real time.

[1541] Step 7:

[1542] Users speak into their smart glasses or smartphone.

[1543] The terminal captures the audio data and transmits it to the server over the network.

[1544] Step 8:

[1545] The server converts the received voice data into text data using a voice recognition engine.

[1546] Translate text data into a specified language in real time.

[1547] The translated text data is passed to a speech synthesis engine to generate speech data.

[1548] Step 9:

[1549] The server encodes the generated voice data and transmits it to the terminal.

[1550] The device (smartphone) decodes the received audio data and plays it back through the speaker to transmit it to the other party.

[1551] In this way, real-time translation and emotional persona provision enable smooth communication between users who speak different languages.

[1552] Example 1

[1553] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1554] Conventional translation systems simply translate speech and text, and are insufficient to realize communication that takes into account the emotions and personality of the other party. This makes it difficult to accurately grasp the other party's intentions and emotions, especially in cross-cultural communication, and often leads to misunderstandings and anxiety.

[1555] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1556] In this invention, the server includes means for acquiring camera footage and transmitting the video data to the server, means for the server to extract a person's voice from the video data, perform voice recognition, and convert it into text data, means for the server to analyze the text data and video data and analyze the other person's emotions and personality, means for the server to translate the text data and transmit the translated text data and emotion / personality information to the terminal, means for displaying the translation result and emotion / personality information received by the terminal in the user's field of view, means for the terminal to acquire the user's voice and transmit it to the server, means for the server to convert the user's voice into text data and translate it, means for the server to convert the translated text data into audio data and transmit it to the terminal, and means for the terminal to play back the audio data received. This enables communication that not only provides translation results but also grasps the other person's emotions and personality in real time.

[1557] "Camera footage" is visible light information captured by a camera, and is data recorded as an image or video.

[1558] "Video data" refers to information stored in digital format from camera footage, and is expressed as an image or video file.

[1559] A "server" is a computer system that processes and stores data over a network, and is responsible for accepting and responding to requests from clients.

[1560] "Speech recognition" is a technology that analyzes speech and converts it into text data, recognizing spoken words as text information.

[1561] "Text data" refers to data that stores character information in a digital format and is expressed as language.

[1562] An "emotion" is a psychological reaction or state that a person shows to a particular situation, and can be anything from joy, sadness, surprise, etc.

[1563] "Personality" refers to the character and attitude that can be inferred from an individual's language and behavior patterns, and indicates the other person's personality and characteristics.

[1564] "Translation" is the act of converting text or audio expressed in one language into another, thereby conveying meaning between different languages.

[1565] A "terminal" is a computer system or device that is directly operated by a user, including smart glasses and smartphones.

[1566] A "user" is a person who communicates using this system and is the entity that operates each function of the system.

[1567] "Audio data" refers to information that stores audio in digital form and is expressed as an audio file.

[1568] "Playback" means reading stored data and outputting the original sound or video, making digital information into a form that can be confirmed visually or audibly.

[1569] This invention is a system that combines a camera-equipped device with a portable communication device to perform real-time translation and emotion and personality analysis. This system is designed to help users communicate smoothly with people who speak different languages.

[1570] System configuration

[1571] The system consists of the following main components:

[1572] 1. Camera-equipped devices (e.g., smart glasses)

[1573] 2. Portable communication devices (e.g., smartphones)

[1574] 3. Server Device

[1575] 4. Various software (e.g., facial recognition technology, voice recognition engine, emotion analysis engine, translation engine, voice synthesis engine)

[1576] Specific hardware and software

[1577] 1. Camera-equipped devices acquire images using the built-in camera. Use a high-resolution digital camera (e.g., 1080p camera).

[1578] 2. The portable communication device uses wireless communication technology (e.g., Wi-Fi or Bluetooth) to communicate data between the device and the server.

[1579] 3. The server equipment requires a high-performance processor and large-capacity memory, so it is possible to use a cloud server.

[1580] 4. The following software must be used:

[1581] OpenCV as a facial recognition technology

[1582] Google Cloud Speech-to-Text as the speech recognition engine

[1583] IBM Watson Tone Analyzer as a sentiment analysis engine

[1584] LIWC for personality inference

[1585] Google Translate API as a translation engine

[1586] Amazon Polly as a speech synthesis engine

[1587] Explanation of program processing

[1588] 1. Acquiring and transmitting camera images

[1589] The terminal uses a camera to capture images of the area in front of the user in real time, and transmits the image data to a server via a portable communication device.

[1590] 2. Video analysis and audio extraction

[1591] The server analyzes the transmitted video data and identifies people in the video using facial recognition technology, while simultaneously converting the audio synchronized with the video into text using a voice recognition engine.

[1592] 3. Emotional and personality analysis

[1593] The server uses an emotion analysis engine and personality estimation engine to analyze the other person's emotions and personality based on the acquired text data and facial recognition data.

[1594] 4. Translation Processing

[1595] The server translates the text data obtained by speech recognition into the language specified by the translation engine, and sends the translated text data along with emotion and personality information to the terminal.

[1596] 5. Display of Information

[1597] The device overlays the received translation results and emotional and personality information onto the user's field of vision, allowing the user to see what the other person is saying and how they are feeling in real time.

[1598] 6. Audio input and transmission

[1599] The user inputs their own speech by speaking into the device, and the terminal transmits this voice data to the server.

[1600] 7. Audio translation and output

[1601] The server converts the received voice data into text data using a voice recognition engine and translates it into the specified language. The translated text data is converted into voice data using a voice synthesis engine and sent to the device. The device then plays the received voice data over a speaker.

[1602] Examples and prompts

[1603] Below is a concrete example of how this system can be used in an international business meeting.

[1604] 1. The device captures the other party's image with its camera and sends it to the server.

[1605] 2. The server analyzes the video and audio and converts the audio into text data.

[1606] 3. The server analyzes the other person's emotions and personality based on the text and video data.

[1607] 4. The server translates the text data and sends the translation results, emotions, and personality information to the terminal.

[1608] 5. The device displays the translation results and emotional and personality information in the user's field of vision.

[1609] 6. The user speaks into the device and the voice data is sent to the server.

[1610] 7. The server translates the speech, generates speech data, and sends it to the device.

[1611] 8. The device plays the generated audio and transmits it to the other person.

[1612] Examples of prompt sentences include:

[1613] "Please explain in detail the camera image acquisition capabilities of this system. Please specify what hardware and software you use, and provide specific examples."

[1614] Based on the above, by using the system, users can communicate accurately and smoothly in real time with people who speak different languages.

[1615] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1616] Step 1:

[1617] The device (smart glasses) captures images of the front in real time using a built-in camera. The input is the image that appears in the user's field of vision. This is temporarily stored in internal memory as digital image data. This image data is then sent to a server via Wi-Fi or Bluetooth. The output is the digital image data sent to the server.

[1618] Step 2:

[1619] The server receives the transmitted video data. The input is digital video data. The server uses facial recognition technology (OpenCV) to identify people in the video. At the same time, it uses a speech recognition engine (Google Cloud Speech-to-Text) to convert the audio synchronized with the video into text data. The output is facial recognition data and text data.

[1620] Step 3:

[1621] The server uses an emotion analysis engine (IBM Watson Tone Analyzer) and a personality estimation engine (LIWC) to analyze the other person's emotions and personality based on the acquired text data and facial recognition data. The input is text data and facial recognition data. The output is emotion data and personality data. The server analyzes changes in facial expressions, tone of voice, patterns of words used, etc.

[1622] Step 4:

[1623] The server translates the text data obtained by speech recognition into the specified language using a translation engine (Google Translate API). The input is text data. The translated text data is then aggregated again together with emotion and personality information. The output is the translated text data and emotion / personality information.

[1624] Step 5:

[1625] The server sends the translation results and emotion / personality information to the device. The input is the translated text data and emotion / personality information. The device (smart glasses) overlays the received data in the user's field of vision. The input is the translated text data and emotion / personality information. The output is the translation results and emotion / personality information displayed in the user's field of vision.

[1626] Step 6:

[1627] The user speaks to the smart glasses or smartphone. The input is what the user says. The device captures the voice data and transmits it to the server via wireless communication. The output is the voice data transmitted to the server.

[1628] Step 7:

[1629] The server converts the received voice data into text data using a voice recognition engine. The input is the user's voice data. Next, it translates it into the specified language using a translation engine (Google Translate API). The output is the translated text data. The translated text data is converted into voice data using a voice synthesis engine (Amazon Polly). The output is the generated voice data. This voice data is sent to the terminal, and the terminal plays the received voice data. The input is the generated voice data. The output is the user's spoken content, which is played back as voice.

[1630] (Application example 1)

[1631] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1632] There is a need for a method to facilitate communication between passengers and systems within autonomous vehicles. In particular, it is necessary to provide real-time translation in multilingual environments and analyze passenger emotions and personalities to provide appropriate responses. Such a system would improve convenience for international users and promote the widespread adoption of autonomous vehicles. Furthermore, there is a need for a more efficient and intuitive interface to ensure passenger safety and comfort.

[1633] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1634] In this invention, the server includes a means for acquiring camera footage and transmitting the video data to the server, a means for the server to extract a person's voice from the video data, perform voice recognition, and convert the voice data into text data, and a means for the server to analyze the text data and video data and analyze the other person's emotions and personality. This enables smooth communication with passengers in an autonomous vehicle and real-time translation in a multilingual environment. Furthermore, understanding passenger emotions and personality and providing feedback based on this understanding improves passenger safety and comfort.

[1635] "Camera footage" is real-time visual data captured using a camera.

[1636] "Video data" refers to camera footage converted into a form that can be stored or transmitted in a digital format.

[1637] A "server" is a computer system that receives, analyzes, and transmits data over a network.

[1638] "Person's voice" refers to a voice uttered by a specific person, and is converted into text data by voice recognition.

[1639] "Speech recognition" is a technology that analyzes speech as a digital signal and converts it into text data.

[1640] "Text data" is character information converted from speech by speech recognition.

[1641] "Analysis" is the process of examining the received data in detail and extracting useful information.

[1642] "Emotion" is a psychological state that can be inferred by analyzing a person's facial expression and tone of voice.

[1643] "Personality" refers to a person's characteristics and tendencies that can be inferred based on their behavior, speaking style, and vocabulary used.

[1644] "Translation" is the process of converting text data from one language to another.

[1645] A "terminal" is a hardware device that is directly operated by a user and that displays data and plays audio.

[1646] "Displaying in the field of view" means overlaying information on the device screen or smart glasses display.

[1647] "Capture voice" means that the terminal captures the user's speech with a microphone.

[1648] "Audio data" refers to data obtained by converting audio captured by a microphone into a digital format.

[1649] "Speech synthesis" is a technology that generates synthetic speech from text data and converts it into reproducible speech data.

[1650] An "autonomous vehicle" is a vehicle that can drive autonomously without the operation of a human driver.

[1651] This invention provides a system for realizing smooth communication with passengers in autonomous vehicles. This system links smart glasses, a smartphone, and a server to perform translation and emotion and personality analysis in real time. A specific implementation method is described below.

[1652] System configuration

[1653] Hardware

[1654] 1. Smart Glasses

[1655] It has a built-in camera that captures and transmits images of the area ahead in real time.

[1656] It has a display function and overlays information onto the user's field of vision.

[1657] 2. Smartphone

[1658] It captures the audio and sends the data to the server.

[1659] Play the synthesized speech and output the translated information as audio.

[1660] 3. Server

[1661] A computer system for high-performance data analysis and communications.

[1662] software

[1663] 1. OpenCV (cv2)

[1664] Responsible for acquiring camera footage and processing the video data.

[1665] 2. Speech Recognition Engine

[1666] Convert the audio data into text data.

[1667] 3. Sentiment Analysis Engine

[1668] Analyze emotions from facial expressions and tone of voice.

[1669] 4. Translation Engine

[1670] Translates text obtained by speech recognition into a specified language.

[1671] 5. Speech synthesis engine

[1672] Text data is converted into audio and audio data that can be played on a smartphone is generated.

[1673] System Operation

[1674] The server receives camera footage sent from the smart glasses and analyzes the video data to extract the person's voice. The extracted voice is converted into text data using a voice recognition engine. The server then works with an emotion analysis engine to analyze emotions and personality based on the text and video data. At the same time, a translation engine translates the text data into the specified language. The translation results and emotion / personality information are sent to the smart glasses and displayed as an overlay in the user's field of vision.

[1675] When a user speaks into a smartphone, the voice data is sent to a server. The server converts this voice data into text data using a voice recognition engine and translates it using a translation engine. The translated text data is then converted into voice data using a speech synthesis engine and sent to the smartphone. The smartphone plays back the received voice data, allowing the user to communicate their intentions to the other party in real time.

[1676] Specific examples

[1677] As an example of the system, consider a situation in which an English-speaking passenger communicates with a Japanese-speaking passenger in an autonomous vehicle. The smart glasses capture video of the passenger with a camera and send it to a server. The server analyzes the video and audio data, generates text data from the speech, and analyzes emotions and personality. This information is translated and displayed on the smart glasses' display along with the translated text, "I am feeling anxious." If the passenger says, "I am feeling anxious," the smart glasses will display "I am feeling anxious." The user speaks into their smartphone, and their speech is translated into Japanese and transmitted to the other party.

[1678] Prompt Sentence Examples

[1679] "Please recognize the passenger's voice, analyze and translate their emotions, and display them in Japanese."

[1680] This system will facilitate smooth communication in multilingual environments inside autonomous vehicles and enable accurate understanding of passenger emotions and personalities, thereby improving passenger safety and comfort and promoting the use of autonomous vehicles.

[1681] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1682] Step 1:

[1683] The smart glasses use a camera to capture real-time video of passengers, and this video data is sent over a network to a server.

[1684] Input: Real-time video

[1685] Output: Video data sent to the server

[1686] How it works: The smart glasses use a camera to capture images of what's ahead, convert the data into packets, and send them to a server.

[1687] Step 2:

[1688] The server analyzes the received video data and identifies people in the video, while simultaneously extracting audio from the video and converting it into text using a speech recognition engine.

[1689] Input: Video data

[1690] Output: Speech-to-text data

[1691] Specific operation: The server uses video processing software to identify people from the video, extract audio data, and convert it into text using a voice recognition engine.

[1692] Step 3:

[1693] The server performs emotion and personality analysis based on the text data and video data obtained through speech recognition. An emotion analysis engine is used to analyze facial expressions and tone of voice, and personality is inferred from speaking style and vocabulary used.

[1694] Input: Text data, video data

[1695] Output: Emotion data, personality data

[1696] Specific operation: The server works in conjunction with the sentiment analysis engine to analyze emotions and personality using text data and video data as anchors.

[1697] Step 4:

[1698] The server uses a translation engine to translate the text data obtained by speech recognition into the specified language, and the translation results, along with emotion and personality information, are sent to the smart glasses.

[1699] Input: Text data, emotion data, personality data

[1700] Output: Translated text data, emotion data, personality data

[1701] Specific operation: The server uses a translation engine to translate text data in real time and transmits it to the smart glasses along with emotional and personality data.

[1702] Step 5:

[1703] The smart glasses overlay the received translation results and emotional and personality information onto the user's field of vision.

[1704] Input: translated text data, emotion data, personality data

[1705] Output: Information displayed in the user's field of view

[1706] How it works: The smart glasses use their display function to overlay the collected information onto the user's field of vision.

[1707] Step 6:

[1708] A user speaks into a smartphone, which captures the voice and sends the voice data to a server.

[1709] Input: User's voice

[1710] Output: Audio data sent to the server

[1711] Specific operation: The smartphone captures the user's voice with a microphone, converts the data into packets, and sends them to the server.

[1712] Step 7:

[1713] The server converts the received voice data into text data using a speech recognition engine, translates it into the specified language, and passes the translated text data to a speech synthesis engine, where it is converted into voice data.

[1714] Input: Audio data

[1715] Output: Speech-synthesized translation data

[1716] Specific operation: The server uses a speech recognition engine to convert speech into text, then translates it using a translation engine, and finally converts the translated data into audio using a speech synthesis engine.

[1717] Step 8:

[1718] The smartphone plays the voice data received from the server and conveys what the user said to the other party.

[1719] Input: Speech-synthesized translation data

[1720] Output: Played audio

[1721] Specific operation: The smartphone plays the audio data received from the server using the speaker.

[1722] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1723] MODE FOR CARRYING OUT THE INVENTION

[1724] This invention is a system that links camera-equipped smart glasses with a smartwatch (or smartphone) to perform real-time translation and analyze the emotions and personas of both parties. This system not only helps users communicate smoothly across language barriers, but also incorporates an emotion engine that can recognize and display the emotions of both parties to improve the quality of communication.

[1725] System Program Processing

[1726] This system operates in the following manner.

[1727] 1. Acquiring and transmitting camera images

[1728] The device (smart glasses) uses a built-in camera to capture real-time images.

[1729] The terminal encodes the acquired video data and transmits it to a server via a network.

[1730] 2. Video analysis and audio extraction

[1731] The server decodes the received video data and uses a video analysis module to detect the movements of a person's face and mouth.

[1732] The server separates the audio data from the video and passes it to a voice recognition engine.

[1733] 3. Speech Recognition and Text Conversion

[1734] The server converts the voice data into text data using a voice recognition engine.

[1735] The server temporarily stores the generated text data in storage.

[1736] 4. Emotional and persona analysis

[1737] The server analyzes emotions and personas based on the text data and video data.

[1738] Sentiment analysis combines facial expression recognition and voice tone analysis to identify emotional states.

[1739] Persona analysis infers a person's characteristics from the content of their text and the way they speak.

[1740] 5. User Emotion Recognition

[1741] The devices (smart glasses and smart watches) capture the user's video and audio data through their built-in cameras and microphones.

[1742] The device passes this data to an emotion engine to analyze the user's emotions.

[1743] The user's emotion data is transmitted to the server.

[1744] 6. Translation Processing

[1745] The server translates the text data obtained by speech recognition into the specified language in real time.

[1746] A translation module converts the text data into the user's native language.

[1747] The translated text data and emotion / persona information are packaged into packets, the data is encoded, and sent to the device.

[1748] 7. Display of Information

[1749] The device (smart glasses) decodes the received data and overlays the translation results and both parties' emotional and persona information onto the user's field of vision.

[1750] Users can check the translation results in real time, as well as the other person's and their own emotions and persona information.

[1751] 8. Audio Input and Transmission

[1752] Users speak into their smart glasses or smartphone.

[1753] The terminal captures the audio data and transmits it to the server over the network.

[1754] 9. Audio Translation and Output

[1755] The server converts the voice data into text using a voice recognition engine and translates it into the specified language in real time.

[1756] The translated text data is passed to a speech synthesis engine to generate speech data.

[1757] The server encodes the generated voice data and transmits it to the terminal.

[1758] The device (smartphone) decodes the received audio data and plays it back through the speaker to transmit it to the other party.

[1759] Specific examples

[1760] Example 1: International business meeting

[1761] 1. The device captures the other party's image with its camera and sends it to the server.

[1762] 2. The server analyzes the video and audio and converts the audio into text data.

[1763] 3. The server analyzes the other person's emotions and persona based on the text and video data.

[1764] 4. The device recognizes the user's emotions based on the video and audio captured and transmits the information to the server.

[1765] 5. The server translates the text data and sends the translation results, the other person's emotions, the user's emotions, and persona information to the terminal.

[1766] 6. The device displays the translation results and emotion / persona information in the user's field of view.

[1767] 7. The user speaks into the smartphone, and the device sends the voice data to the server.

[1768] 8. The server translates the speech, generates speech data, and sends it to the device.

[1769] 9. The smartphone plays the generated audio and transmits it to the other person.

[1770] This program processing enables real-time translation and communication while recognizing the emotions of both parties, improving the quality of communication.

[1771] The processing flow will be explained below.

[1772] Step 1:

[1773] The device (smart glasses) captures real-time video using a built-in camera, and the video data is encoded by a built-in encoding chip and sent to a server via a network.

[1774] Step 2:

[1775] The server decodes the received video data and uses a video analysis module to detect facial and mouth movements, thereby identifying the person in the video.

[1776] Step 3:

[1777] The server separates the audio data from the video and passes it to a speech recognition engine, where the audio is converted into text data.

[1778] Step 4:

[1779] The server temporarily stores the generated text data in storage, and simultaneously analyzes emotions and personas based on the text and video data. Emotion analysis is performed using facial expression recognition and voice tone analysis. Persona analysis estimates a person's characteristics from the content of the text and speaking style.

[1780] Step 5:

[1781] The devices (smart glasses and smart watches) capture the user's video and audio data, and use an emotion engine to recognize emotions based on the user's facial expressions and tone of voice, then send the results to the server.

[1782] Step 6:

[1783] The server translates the text data obtained by speech recognition into the specified language in real time. The translation module converts the text data into the user's native language. The translated text data and emotion / persona information are packaged into packets, encoded, and sent to the device.

[1784] Step 7:

[1785] The device (smart glasses) decodes the received data and overlays the translation results, as well as the other person's and the user's emotional and persona information, onto the user's field of vision, allowing the user to check the translation results and emotional and persona information in real time.

[1786] Step 8:

[1787] Users speak into their smart glasses or smartphone, which captures the voice data and transmits it to a server over a network.

[1788] Step 9:

[1789] The server converts the received voice data into text using a speech recognition engine, translates it into the specified language in real time, and passes the translated text data to a speech synthesis engine to generate voice data.

[1790] Step 10:

[1791] The server encodes the generated voice data and sends it to the device, which then decodes the received voice data and plays the translated voice over the speaker to communicate with the other party.

[1792] The above processing steps enable real-time two-way translation and emotion recognition / display, enabling smooth communication.

[1793] Example 2

[1794] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1795] In today's global society, there is a demand for smooth communication between people who speak different languages ​​and for improving the quality of that communication. However, current technology has difficulty not only overcoming language barriers, but also recognizing and adapting to the emotions and personas of both parties in real time. Furthermore, conventional systems do not provide real-time emotional feedback, which can lead to a poor user experience.

[1796] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for acquiring camera images and transmitting the image data to the server, a means for the server to extract a person's voice from the image data, perform voice recognition, and convert the voice data to text data, and a means for the server to analyze the text data and image data and analyze the other person's emotions and persona. This enables smooth communication between people who speak different languages ​​and real-time recognition and adaptation of each other's emotions and personas.

[1797] "Camera footage" is digital data of visual information captured by a camera device.

[1798] "Video data" refers to visual information captured as camera footage that has been converted into digital format.

[1799] A "server" is a computer system that receives, processes, and transmits data over a network.

[1800] A "terminal" is a device used by a user, and includes wearable devices such as smart glasses and smart watches.

[1801] "Person's voice" refers to acoustic data generated by speech such as conversation.

[1802] "Speech recognition" is a technology that analyzes voice data and converts it into text data.

[1803] "Text data" is character information generated by voice recognition.

[1804] "Emotion analysis" is a technology that identifies a person's emotional state based on video and audio data.

[1805] "Persona analysis" is a technique for estimating a person's characteristics and personality based on text data and speaking characteristics.

[1806] "Translation" is the process of converting text written in one language into another.

[1807] "Emotional feedback" is the process of analyzing the emotional state of the user and the other person and adapting in real time based on the results.

[1808] "Real-time" refers to immediate processing or response with little or no delay.

[1809] "Display in the field of view" is a technology in which a device overlays information in front of the user's line of sight.

[1810] MODE FOR CARRYING OUT THE INVENTION

[1811] This invention is a system that links camera-equipped devices (smart glasses, smart watches, or smartphones) to translate in real time and analyze both parties' emotions and personas. This system allows users to communicate smoothly across language barriers, and further improves the quality of communication by incorporating an emotion engine that recognizes and displays emotions on both sides.

[1812] Hardware and Software Overview

[1813] The device contains the following elements:

[1814] 1. Smart glasses: Equipped with a built-in camera, microphone, display, and Wi-Fi or data communication module.

[1815] 2. Smartwatch or smartphone: Equipped with a built-in camera, microphone, speaker, Wi-Fi or data communication module.

[1816] The server includes the following elements:

[1817] 1. Decoding and analysis module: FFmpeg, OpenCV

[1818] 2. Speech recognition engine: Google Speech-to-Text API

[1819] 3. Sentiment and persona analysis engine: IBM Watson Natural Language Understanding API, OpenCV

[1820] 4. Translation Module: Microsoft Translator Text API

[1821] 5. Speech synthesis engine: Amazon Polly

[1822] 6. Storage: AWS S3 and other cloud storage services

[1823] 7. Security protocols: TLS / SSL, etc.

[1824] System operation procedure

[1825] 1. Camera video and audio acquisition:

[1826] The device (smart glasses) uses a built-in camera to capture real-time video and a microphone to capture audio.

[1827] The footage is captured at 1080p resolution and encoded using the H.264 codec.

[1828] The encoded video data is sent to a server using Wi-Fi or a mobile network.

[1829] 2. Video analysis and audio extraction:

[1830] The server uses FFmpeg to decode the video and OpenCV for face detection and audio extraction.

[1831] The extracted audio data is converted into text data using the Google Speech-to-Text API.

[1832] 3. Emotional and persona analysis:

[1833] The server analyzes emotions and personas using IBM Watson Natural Language Understanding API and OpenCV based on text data and video data generated by the Google Speech-to-Text API.

[1834] 4. Translation Processing and Display:

[1835] The server translates the text data in real time using the Microsoft Translator Text API.

[1836] The translated text data and emotion / persona information are packaged into packets, encrypted using the TLS / SSL protocol, and sent to the device.

[1837] The device decodes the received data and displays it as an overlay in the user's field of view using Unity or ARCore.

[1838] 5. Audio feedback:

[1839] The device (smartphone) captures the user's speech with its built-in microphone and sends it to the server.

[1840] The server converts the speech to text and, after translation, generates the speech data using a speech synthesis engine.

[1841] The generated audio data is sent to the terminal and played through the speaker.

[1842] Examples and usage

[1843] Example 1: International business meeting

[1844] 1. The device captures the other party's image with its camera and sends it to the server.

[1845] 2. The server analyzes the video and audio and converts the audio into text data.

[1846] 3. The server analyzes the other person's emotions and persona based on the text and video data.

[1847] 4. The device recognizes the user's emotions based on the video and audio captured and transmits the information to the server.

[1848] 5. The server translates the text data and sends the translation results, the other person's emotions, the user's emotions, and persona information to the terminal.

[1849] 6. The device displays the translation results and emotion / persona information in the user's field of view.

[1850] 7. The user speaks into the smartphone, and the device sends the voice data to the server.

[1851] 8. The server translates the speech, generates speech data, and sends it to the device.

[1852] 9. The smartphone plays the generated audio and transmits it to the other person.

[1853] This program processing enables real-time translation and communication while recognizing the emotions of both parties, improving the quality of communication.

[1854] Examples of prompt statements

[1855] An example of a prompt sentence when generating an explanation of this system using a generative AI model is as follows:

[1856] Please explain in natural language how the following system works. Use either the server, terminal, or user as the subject, and clarify what hardware or software is used for each step and what data processing is performed. Explain with concrete examples.

[1857] (System Overview)

[1858] This system connects camera-equipped smart glasses with a smartwatch (or smartphone) to perform real-time translation and analysis of emotions and personas of both parties.The system incorporates an emotion engine and can recognize and display emotions of both parties.

[1859] Based on this prompt, the generative AI model can automatically generate in the specified format.

[1860] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1861] System program processing flow

[1862] Step 1:

[1863] The device (smart glasses) captures real-time video using its built-in camera. The captured video data is captured at 1080p resolution and 30 frames per second. The captured video is then encoded using the H.264 codec. The encoded video data is then sent to a server via Wi-Fi or mobile network.

[1864] Input: Real-time video from the user's field of view

[1865] Output: The encoded video data is sent to the server

[1866] Step 2:

[1867] The server uses FFmpeg to decode the received video data and OpenCV to detect facial and mouth movements, then the audio data is separated from the video data and passed to a speech recognition engine.

[1868] Input: Encoded video data

[1869] Output: Decoded video data, detected face and mouth movements, extracted audio data

[1870] Step 3:

[1871] The server converts the audio data into text data using the Google Speech-to-Text API, which is then saved in JSON format.

[1872] Input: Extracted audio data

[1873] Output: Text data (JSON format)

[1874] Step 4:

[1875] The server analyzes emotions and personas based on text and video data using IBM Watson Natural Language Understanding API and OpenCV. Emotion analysis is performed by combining facial expression recognition and voice tone analysis, while persona analysis estimates a person's characteristics from the content of the text and speaking style.

[1876] Input: Text data, video data

[1877] Output: Emotion data, persona data

[1878] Step 5:

[1879] The devices (smart glasses and smart watches) capture the user's video and audio data through their built-in cameras and microphones. This data is passed to the emotion engine, and the results of analyzing the user's emotions are sent to the server.

[1880] Input: User's video and audio data

[1881] Output: User emotion data is sent to the server

[1882] Step 6:

[1883] The server uses the Microsoft Translator Text API to translate the text data obtained by speech recognition into the specified language in real time. The translation module converts the text data into the user's native language, and then packages the translated text data and emotion / persona information into packets and sends them to the device using the TLS / SSL protocol.

[1884] Input: Text data, emotion data, persona data

[1885] Output: Translated text data, emotion data, and persona data are sent to the device.

[1886] Step 7:

[1887] The device (smart glasses) decodes the received data and overlays the translation results and both emotional and persona information onto the user's field of view. This is done using Unity or ARCore.

[1888] Input: Translated text data, emotion data, persona data

[1889] Output: Information overlaid on the user's field of view

[1890] Step 8:

[1891] Users speak into their smart glasses or smartphones, and the voice data is captured and sent over the network to a server.

[1892] Input: User's voice data

[1893] Output: Audio data sent to the server

[1894] Step 9:

[1895] The server converts the voice data into text using the Google Speech-to-Text API and translates it into the specified language in real time. The translated text data is then passed to the Amazon Polly speech synthesis engine to generate voice data. The generated voice data is encoded using the Opus codec, sent to the device, and played back through the smartphone speaker.

[1896] Input: Audio data sent to the server

[1897] Output: The generated and encoded audio data is sent to the device and played through the smartphone speaker.

[1898] (Application example 2)

[1899] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1900] The purpose of this invention is to facilitate communication between people who speak different languages ​​and achieve effective communication across language barriers. It also aims to improve the quality of communication by providing real-time emotion and persona information. In particular, it aims to solve the problem of providing fast and accurate customer service in brick-and-mortar stores and improving customer satisfaction.

[1901] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring camera video and transmitting the video data to the server, means for the server to extract the target's voice from the video data, perform voice recognition, and convert it into text data, means for the server to analyze the text data and video data and analyze the other party's emotions and personality, means for the server to translate the text data and transmit the translated text data and emotion / personality information to the terminal, means for the terminal to display the translation result and emotion / personality information received in the user's field of view, means for the terminal to acquire the user's voice and transmit it to the server, means for the server to convert the user's voice into text data and translate it, means for the server to convert the translated text data into audio data and transmit it to the terminal, means for the terminal to play the received audio data, means for the terminal to provide translations and emotional information to customers in the store, and means for the terminal to analyze and display the customer's interest and satisfaction with the product. This enables smooth communication and high-quality customer service that transcends language barriers.

[1902] "Camera image" refers to image data acquired by a device that converts light into electronic signals and records them as image information.

[1903] A "server" is a computer system that receives data from multiple devices over a network and processes, analyzes, stores, and distributes the data.

[1904] "Speech recognition" is a technology that analyzes voice data and converts it into text data.

[1905] "Text data" is digital data that contains text information.

[1906] "Translation" is the process of converting written or spoken content in one language into another.

[1907] "Emotion analysis" is a technology that analyzes a person's facial expressions, tone of voice, text content, etc. to identify their emotional state.

[1908] A "persona" is a model of a user's characteristics and behavioral patterns, represented as a specific role or character.

[1909] "Displaying in the field of view" means superimposing information onto the display of a device worn by the user (e.g., smart glasses).

[1910] "Audio data" refers to data in which audio is recorded in digital format.

[1911] A "terminal" is a device (e.g., smart glasses, smartphone) that is connected to a network and performs various inputs and outputs.

[1912] A "store" is a physical location that offers goods and services.

[1913] "Customer" means a person who purchases or uses goods or services.

[1914] "Translation result" refers to data input in an original language that has been subjected to a translation process and is output in another language.

[1915] "Interest and satisfaction with a product" is information that indicates the degree of interest and satisfaction that a customer has with a particular product or service.

[1916] This invention is a system for facilitating communication between people who speak different languages ​​and achieving effective communication across language barriers. It also aims to improve the quality of communication by providing real-time emotional and persona information. It is particularly intended to improve customer satisfaction by providing prompt and accurate customer service in brick-and-mortar stores.

[1917] This system is implemented using smart glasses or smartphones equipped with cameras. Specifically, the following process is performed:

[1918] First, the device (smart glasses or smartphone) captures camera images and sends the image data to a server. The server extracts the target audio from the captured image data, performs speech recognition, and converts it into text data. For speech recognition, it uses technologies such as Google Cloud Speech-to-Text.

[1919] The server then analyzes the text and video data to determine the other person's emotions and personality. Emotion analysis uses a combination of facial expression recognition and voice tone analysis. This includes the "emotion engine" and "personality analysis module."

[1920] The server translates the text data in real time based on the analysis results. The translation uses the Google Translate API and the Helsinki NLP translation model. The translated text data and emotion / personality information are then sent back to the device.

[1921] The device receives this information and displays it as an overlay in the user's field of vision. Specifically, the translation results and emotional information are overlaid on the smart glasses display, allowing the user to check the translation results, the other person's emotions, and their personality in real time.

[1922] The device then captures the user's voice and sends it to the server. The server converts the user's voice into text data and performs translation processing. The translated text data is converted into voice data and sent back to the device. The device then plays back the received voice data and transmits it to the other party.

[1923] A concrete example of this application is in brick-and-mortar stores. For example, when a foreign customer visits a store and asks a question about a product, the system instantly translates the question and relays it to the store staff. At the same time, the system can analyze the customer's satisfaction and interests in real time and provide this information to the staff. This enables smooth communication that transcends language barriers, improving customer satisfaction.

[1924] An example of a prompt is as follows:

[1925] Please generate a detailed description of the operation: "The server detects faces from video data and determines whether the customer is smiling or not by facial expression recognition." Please also describe the technology used (face detection algorithm, facial expression recognition model, etc.).

[1926] As described above, the present invention makes it possible to provide smooth communication and high-quality customer service that transcends language barriers.

[1927] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1928] Step 1:

[1929] The device (smartglasses or smartphone) captures camera images and sends the image data to a server. The input is the camera image data, and the output is the encoded image data sent to the server. Specifically, the camera built into the smartglasses captures images in real time, and the images are sent to the server via the network.

[1930] Step 2:

[1931] The server extracts audio from the transmitted video data and converts it into text using a speech recognition engine. The input is the audio information in the video data, and the output is text data. Specifically, the server's audio processing module analyzes the video data, separates the audio, and converts it into text using a speech recognition engine such as Google Cloud Speech-to-Text.

[1932] Step 3:

[1933] The server analyzes the generated text data and video data to analyze emotions and personality. The input is text data and video data, and the output is the analysis results of emotions and personality. Specifically, the emotion engine analyzes emotions based on facial expression data and voice tone, and the personality analysis module estimates the user's personality based on the content of the text and speaking style.

[1934] Step 4:

[1935] The server translates text data into a specified language. The input is the original text data, and the output is the translated text data. Specifically, the translation module in the server translates the text in real time using the Google Translate API and the Helsinki NLP translation model.

[1936] Step 5:

[1937] The server encodes the translated text data, emotion, and personality information and sends it to the terminal. The input is the translated text data and the emotion and personality analysis results, and the output is an encoded data packet. Specifically, the server assembles this information into a data packet and sends it to the user's terminal via the network.

[1938] Step 6:

[1939] The device decodes the translation results and emotion / personality information received from the server and overlays them on the user's field of view. The input is an encoded data packet, and the output is the translation results and emotion / personality information displayed on the user's field of view. Specifically, the smart glasses display overlays this information.

[1940] Step 7:

[1941] The device collects the user's voice and sends it to the server. The input is the user's voice, and the output is the voice data sent to the server. Specifically, the microphone in the smart glasses captures the user's speech and sends it to the server.

[1942] Step 8:

[1943] The server converts the user's speech into text data and performs the translation process. The input is speech data, and the output is translated text data. Specifically, the server's speech recognition engine converts the speech into text, and then translates the text in real time.

[1944] Step 9:

[1945] The server converts the translated text data into voice data and sends it to the terminal. The input is the translated text data, and the output is voice data. Specifically, the server's voice synthesis engine converts the text into voice data and sends it to the terminal via the network.

[1946] Step 10:

[1947] The device plays the received voice data and transmits it to the user. The input is voice data, and the output is the reproduced voice. Specifically, the smartphone speaker plays the voice and transmits it to the other party.

[1948] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1949] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1950] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1951] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1952] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1953] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1954] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1955] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1956] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1957] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1958] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1959] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1960] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1961] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1962] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1963] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1964] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1965] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1966] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1967] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1968] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1969] The following is further disclosed regarding the above embodiment.

[1970] (Claim 1)

[1971] means for acquiring camera images and transmitting the image data to a server;

[1972] A means for the server to extract the voice of a person from the video data, perform voice recognition, and convert it into text data;

[1973] A means for the server to analyze the text data and video data and analyze the emotions and persona of the other party;

[1974] A means for the server to translate the text data and transmit the translated text data and emotion / persona information to the terminal;

[1975] A means for displaying the translation results and emotion / persona information received by the terminal in the user's field of view;

[1976] A means for the terminal to acquire the user's voice and transmit it to a server;

[1977] A means for the server to convert the user's voice into text data and translate it;

[1978] A means for the server to convert the translated text data into voice data and transmit the voice data to the terminal;

[1979] A system including means for playing audio data received by a terminal.

[1980] (Claim 2)

[1981] 2. The system according to claim 1, further comprising means for performing sentiment analysis based on the text data and video data acquired by the server.

[1982] (Claim 3)

[1983] 2. The system according to claim 1, further comprising means for performing speech recognition based on the speech data acquired by the server and converting the speech data into text data.

[1984] "Example 1"

[1985] (Claim 1)

[1986] a device that acquires camera images and transmits the image data to a server;

[1987] A device in which the server extracts a person's voice from the video data, performs voice recognition, and converts it into text data;

[1988] A device in which a server analyzes the text data and video data to analyze the emotions and personality of the other party;

[1989] A device in which the server translates the text data and transmits the translated text data and emotion / personality information to the terminal;

[1990] A device that displays the translation results and emotion / personality information received by the terminal in the user's field of vision;

[1991] A device for the terminal to acquire the user's voice and transmit it to a server;

[1992] A device in which the server converts the user's voice into text data and translates it;

[1993] A device in which the server converts the translated text data into voice data and transmits the voice data to the terminal;

[1994] A system including a device for playing audio data received by a terminal.

[1995] (Claim 2)

[1996] The system according to claim 1, further comprising means for performing emotion analysis and personality analysis based on the text data and video data acquired by the server.

[1997] (Claim 3)

[1998] 2. The system according to claim 1, further comprising a device that performs speech recognition based on the speech data acquired by the server and converts it into text data.

[1999] "Application Example 1"

[2000] Described based on the format

[2001] (Claim 1)

[2002] means for acquiring camera images and transmitting the image data to a server;

[2003] A means for the server to extract the voice of a person from the video data, perform voice recognition, and convert it into text data;

[2004] A means for the server to analyze the text data and video data and analyze the emotions and personality of the other party;

[2005] A server translates text data and transmits the translated text data and emotion / personality information to a terminal, on the assumption that the data will be used in an autonomous driving vehicle;

[2006] A means for displaying the translation result and emotion / personality information received by the terminal in the user's field of view;

[2007] A means for the terminal to acquire the user's voice and transmit it to a server;

[2008] A means for the server to convert the user's voice into text data and translate it;

[2009] A means for the server to convert the translated text data into voice data and transmit the voice data to the terminal;

[2010] A system including means for playing audio data received by a terminal.

[2011] (Claim 2)

[2012] 2. The system according to claim 1, further comprising means for performing sentiment analysis based on the text data and video data acquired by the server.

[2013] (Claim 3)

[2014] 2. The system according to claim 1, further comprising means for performing speech recognition based on the speech data acquired by the server and converting the speech data into text data.

[2015] "Example 2: Combining Emotion Engines"

[2016] (Claim 1)

[2017] means for acquiring camera images and transmitting the image data to a server;

[2018] A means for the server to extract the voice of a person from the video data, perform voice recognition, and convert it into text data;

[2019] A means for the server to analyze the text data and video data and analyze the emotions and persona of the other party;

[2020] A means for the server to translate the text data and transmit the translated text data and emotion / persona information to the terminal;

[2021] A means for displaying the translation results and emotion / persona information received by the terminal in the user's field of view;

[2022] A means for the terminal to acquire the user's voice and transmit it to a server;

[2023] A means for the server to convert the user's voice into text data and translate it;

[2024] A means for the server to convert the translated text data into voice data and transmit the voice data to the terminal;

[2025] means for playing back audio data received by the terminal;

[2026] A means for the terminal to analyze the user's emotions and transmit the results to a server;

[2027] A means for the server to provide real-time emotional feedback based on the analyzed emotional data of the user;

[2028] A system including:

[2029] (Claim 2)

[2030] 2. The system of claim 1, wherein the server includes means for analyzing emotions and personas based on the text data and video data.

[2031] (Claim 3)

[2032] 2. The system according to claim 1, wherein the server includes means for converting the translated text data into audio data and transmitting the audio data to the terminal.

[2033] "Application example 2 when combining emotion engines"

[2034] (Claim 1)

[2035] means for acquiring camera images and transmitting the image data to a server;

[2036] A means for the server to extract target audio from the video data, perform audio recognition, and convert it into text data;

[2037] A means for the server to analyze the text data and video data and analyze the emotions and personality of the other party;

[2038] A means for the server to translate the text data and transmit the translated text data and emotion / personality information to the terminal;

[2039] A means for displaying the translation result and emotion / character information received by the terminal in the user's field of view;

[2040] A means for the terminal to acquire the user's voice and transmit it to a server;

[2041] A means for the server to convert the user's voice into text data and translate it;

[2042] A means for the server to convert the translated text data into voice data and transmit the voice data to the terminal;

[2043] means for playing back audio data received by the terminal;

[2044] a means for the terminal to provide translation and emotion information to customers of the store;

[2045] A means for the terminal to analyze and display customer interest and satisfaction with the product;

[2046] A system including:

[2047] (Claim 2)

[2048] 2. The system according to claim 1, further comprising means for performing sentiment analysis based on the text data and video data acquired by the server.

[2049] (Claim 3)

[2050] 2. The system according to claim 1, further comprising means for performing speech recognition based on the speech data acquired by the server and converting the speech data into text data. [Explanation of symbols]

[2051] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for acquiring camera images and transmitting the image data to a server; A means for the server to extract the voice of a person from the video data, perform voice recognition, and convert it into text data; A means for the server to analyze the text data and video data and analyze the emotions and persona of the other party; A means for the server to translate the text data and transmit the translated text data and emotion / persona information to the terminal; A means for displaying the translation result and emotion / persona information received by the terminal in the user's field of view; A means for the terminal to acquire the user's voice and transmit it to a server; A means for the server to convert the user's voice into text data and translate it; A means for the server to convert the translated text data into voice data and transmit the voice data to the terminal; A system including means for playing audio data received by a terminal.

2. The system according to claim 1, further comprising means for performing sentiment analysis based on the text data and video data acquired by the server.

3. 2. The system according to claim 1, further comprising means for performing speech recognition based on the speech data acquired by the server and converting the speech data into text data.

Images