System

Abstract

A system is provided.SOLUTION: A system comprising: means for acquiring voice data; means for converting the acquired voice data into text data; means for specifying a new word or a foreign word included in the text data and translating the new word or the foreign word into an easy-to-understand form; means for converting the translated text data into voice data and providing the voice data to a user; and means for analyzing an emotional state of the user based on the acquired biological information and converting the emotional state into an appropriate wording based on an analysis result.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, new words and foreign words are becoming more prevalent, posing a challenge for the elderly, who find it difficult to understand them. Furthermore, as technology advances, the elderly are increasingly forced to use digital technology, but they find it difficult to adapt to this. Furthermore, this disrupts communication and reduces the quality of life. There is a need for technology that can solve these issues and make it easier for the elderly to understand new words and foreign words in their daily lives. [Means for solving the problem]

[0005] The present invention provides a system including: means for acquiring voice data; means for converting the acquired voice data into text data; means for identifying new words and foreign words contained in the text data and translating them into an easily understandable form; means for converting the translated text data into voice data and providing it to a user; means for acquiring biometric information from the user; and means for analyzing the user's emotional state based on the acquired biometric information and converting the text data into appropriate phrases based on the analysis results. This allows elderly people to understand new words and foreign words in real time and enables appropriate communication based on their emotional state. Specifically, the system includes transmitting voice data to a server, which converts the voice data into text data, and analyzing the user's pulse based on the acquired biometric information to estimate the user's emotional state. This allows elderly people to become familiar with new information and facilitates everyday communication.

[0006] "Voice data" refers to data obtained by converting voice information uttered by a user into digital form.

[0007] The "acquisition means" refers to a device or function for capturing voice or biometric information using an input device such as a microphone.

[0008] "Text data" refers to data obtained by converting voice data into text information using a voice recognition algorithm.

[0009] "Translation tools" are algorithms or programs that convert identified new words or foreign words into a form that is easy for seniors to understand.

[0010] "Biometric information" refers to physical data such as the user's pulse and heart rate.

[0011] The "emotional state" indicates the user's psychological state, such as tension, relaxation, or confusion, that is analyzed based on biological information.

[0012] "Means of analysis" refers to algorithms or programs for evaluating and estimating emotional states based on acquired biometric information.

[0013] A "server" is a computer system that processes voice data and biometric information, and converts and analyzes the data. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] MODE FOR CARRYING OUT THE INVENTION

[0036] The present invention is an AI earphone system that translates new words and foreign words used in everyday life into a form that is easy for elderly people to understand in real time. Specific embodiments of this system are described below.

[0037] System configuration and operation

[0038] This system consists of three main elements: a terminal for acquiring voice data, a server for analyzing the acquired voice data and biometric information, and earphones for providing information to the user.

[0039] Acquiring and sending audio data

[0040] The user puts on the earphones and engages in normal conversation. The microphone in the device captures the user's voice in real time and transmits the voice data to the server. The voice data is transmitted to the server using a secure communication protocol, ensuring data security.

[0041] Speech data analysis and translation

[0042] The server then runs the received voice data through a speech recognition algorithm and converts it into text data. It then identifies new words and foreign words from this text data and translates them into words and expressions that are easy for seniors to understand. For example, if the word "influencer" is detected, it will be translated as "influential person."

[0043] Real-time audio output

[0044] The translated text data is converted back into audio data on the server side and sent to the user's earphones in real time, allowing the user to receive the translated information in a natural voice.

[0045] Collection and analysis of biological information

[0046] The device is equipped with a biometric information acquisition device such as a pulse sensor that acquires the user's biometric information in real time. This data is sent to a server, where an emotion analysis algorithm evaluates the user's emotional state. For example, a high pulse rate may indicate tension.

[0047] Providing information according to emotions

[0048] Based on the analysis results, the server prepares appropriate phrases and additional information. For example, if it detects that the user is confused by a particular word, it will provide additional explanation for that word or convert it into a more familiar expression. This converted information is also sent to the user's earphones in real time and delivered to the user's ears.

[0049] Specific examples

[0050] Example 1:

[0051] In the scenario where a user asks "What is an influencer?", the following happens:

[0052] Users encounter the new term "influencer" in everyday conversations.

[0053] The device captures this audio and sends it to the server.

[0054] The server identifies "influencer" and translates it to "influential person."

[0055] The translated information is converted back into audio data and played in real time through the user's earphones.

[0056] Example 2:

[0057] In the scenario where a user asks "How do I use a smartphone?", the following happens:

[0058] Pulse data shows that users are confused by the use of the foreign word "smartphone."

[0059] The device captures the audio and sends it to the server.

[0060] The server identifies "smartphone" and translates it into "a mobile phone with many functions."

[0061] The translated information is converted back into audio data and played in real time through the user's earphones.

[0062] Additionally, the server detects the user's confusion and provides additional information such as, "If you want to know more about the basic operations of your smartphone, it's best to learn by touching it directly."

[0063] In this way, the present invention is a system that helps elderly people become familiar with new words and information and realizes smooth communication in their daily lives.

[0064] The processing flow will be explained below.

[0065] Specific processing steps of the program

[0066] Step 1: Acquire audio

[0067] The user puts on the earphones and engages in normal conversation.

[0068] A microphone built into the device captures the user's voice in real time.

[0069] Step 2: Sending audio data

[0070] The device then packetizes the captured audio data and sends it to the server using a secure protocol.

[0071] Step 3: Voice Recognition

[0072] The server analyzes the received voice data and converts it into text data using a voice recognition algorithm.

[0073] Step 4: Identifying new words

[0074] The server applies natural language processing to identify new and foreign words contained in the text data.

[0075] Step 5: Translation process

[0076] The server translates each identified new or foreign word into a word or phrase that is easy for seniors to understand, for example, converting "influencer" into "influential person."

[0077] Step 6: Audio translation

[0078] The server reconverts the translated text data into audio data using a speech synthesis algorithm.

[0079] Step 7: Submit your translation data

[0080] The server packetizes the generated voice data and transmits it to the terminal.

[0081] Step 8: Real-time output

[0082] The terminal plays back the received audio data in real time and provides it to the user.

[0083] Step 9: Acquire biometric information

[0084] A pulse sensor installed in the device acquires the user's pulse data.

[0085] Step 10: Sending biometric information

[0086] The terminal transmits the acquired pulse data to the server.

[0087] Step 11: Sentiment Analysis

[0088] The server analyzes the received pulse data and estimates the user's emotional state. For example, if the pulse rate is high, it is determined that the user is nervous.

[0089] Step 12: Generate additional information

[0090] The server then translates the message based on the results of the sentiment analysis, providing appropriate phrasing and additional explanations as needed. For example, if the user is confused, a more detailed explanation is added.

[0091] Step 13: Vocalize additional information

[0092] The server converts the generated additional text data into voice data using a voice synthesis algorithm.

[0093] Step 14: Submit additional information

[0094] The server transmits the additional information in the form of voice to the terminal.

[0095] Step 15: Play additional audio

[0096] The terminal reproduces the received audio data of the additional information in real time and provides it to the user.

[0097] These are the specific processing steps of "Generation Communicator." Through this processing flow, elderly people can easily understand new words and foreign words, realizing comfortable communication.

[0098] Example 1

[0099] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0100] In the past, it was often difficult for elderly people to understand new words and foreign words used in daily life. Furthermore, there was no system that could analyze and translate speech data in real time and provide appropriate information according to the user's emotional state. This made it difficult for elderly people to communicate smoothly when they encountered new words or concepts. The purpose of this invention is to solve this problem.

[0101] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0102] In this invention, the server includes a means for converting voice data into text data, a means for translating new words and foreign words contained in the text data into a form that is easy for seniors to understand, and a means for reconverting the translated text data into voice data. This enables seniors to understand new words and foreign words used in everyday conversation in real time, enabling smooth communication. Furthermore, by analyzing the user's emotional state based on biometric information and providing appropriate phrases and additional information based on the analysis results, a system that is more user-friendly and easy to use can be provided. This allows seniors to communicate smoothly without being confused by new words and information.

[0103] "Audio data" refers to data in the form of sound waves, such as user conversations or environmental sounds.

[0104] A "terminal" is a device for acquiring voice data and biometric information and transmitting that data to a server.

[0105] A "server" is a computer system that analyzes, transforms, and presents information to users.

[0106] "Text data" refers to data obtained by converting voice data into character information.

[0107] "New words and foreign words" are words that have been introduced into the user's native language in a relatively short period of time or words that have been adopted from a foreign country.

[0108] "Translation" is the act of converting something from one language into another language or into an understandable form.

[0109] "Biometric information" refers to physical data such as the user's pulse rate and body temperature.

[0110] "Emotional state" refers to a psychological state estimated based on the user's pulse rate and other biometric information.

[0111] An "emotion analysis algorithm" is a computational method for analyzing biometric information and assessing a user's emotional state.

[0112] "Additional information" refers to supplemental information or explanations provided depending on the user's emotional state.

[0113] A "secure communication protocol" is a communication protocol for safely sending and receiving data.

[0114] A "generative AI model" is an artificial intelligence model that learns patterns from large amounts of data and generates new information.

[0115] The present invention is a system that acquires, analyzes, and translates voice data in real time to provide users with easy-to-understand information about new words and foreign words. A specific embodiment of the present invention will be described. The system comprises a terminal for acquiring voice data, a server for analyzing the acquired voice data and biometric information, and earphones for providing translated information to the user.

[0116] Acquiring audio data

[0117] The device captures the user's voice in real time using a built-in microphone when they are having a normal conversation. The voice data is then temporarily stored on the device. For example, if a user says, "Young people these days are only chasing influencers," the voice data will be captured by the device.

[0118] Sending audio data

[0119] The terminal transmits the captured audio data to the server using a secure communication protocol (for example, HTTPS or TLS).

[0120] Analysis of audio data

[0121] The server converts the received voice data into text data using a speech recognition API such as Google Cloud Speech-to-Text or Amazon Transcribe. For example, the generated text data would be, "Young people these days are only chasing influencers."

[0122] Identifying and translating new and foreign words

[0123] The server analyzes the generated text data and identifies new words and foreign words (e.g., "influencer") using natural language processing (NLP) technology. It then translates these new words and foreign words into a form that is easy for seniors to understand (e.g., "influencer" becomes "influential person") using a pre-trained generative AI model or machine translation engine.

[0124] Converting translation text to audio data

[0125] The server reconverts the translated text data into audio data using a speech synthesis API such as Google Cloud Text-to-Speech or Amazon Polly.

[0126] Audio output to the user

[0127] The translated voice data is then sent to the device again using a secure communication protocol and finally output to the user's earphones. The user can listen to the translated voice in real time. For example, the translated voice may provide the user with information such as, "Young people these days only chase after influential people."

[0128] Collection and analysis of biological information

[0129] The device is equipped with a biometric information acquisition device such as a pulse sensor, which acquires the user's biometric information (e.g., pulse) in real time and temporarily stores the data. This biometric information is also transmitted to a server using a secure communication protocol, and the server evaluates the user's emotional state (e.g., tension) using an emotion analysis algorithm.

[0130] Providing information according to emotions

[0131] The server generates appropriate phrases and additional information based on the analysis results. For example, if it determines that the user is confused by the foreign word "smartphone," it generates advice such as, "If you want to learn more about the basic operations of a smartphone, it's a good idea to learn by touching it directly," and then converts this into audio data using Google Cloud Text-to-Speech and provides it to the user.

[0132] In this way, this system helps elderly people become familiar with new words and information, enabling smooth communication in their daily lives.

[0133] Examples of prompt statements

[0134] The following are examples of prompt sentences:

[0135] Capture conversations that include new foreign words or technical terms, translate them in real time to make them understandable for seniors, and provide appropriate phrasing or additional information based on the user's emotional state.

[0136] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0137] The flow of this system's program processing

[0138] Step 1: Acquire audio data

[0139] The device uses a built-in microphone to capture the user's voice in real time. The input is the user's voice, and the output is the captured voice data. This voice data is immediately and temporarily stored on the device. For example, when a user says, "What is an influencer?", the voice is captured.

[0140] Step 2: Sending audio data

[0141] The device sends the captured audio data to the server using a secure communication protocol (HTTPS or TLS). The input is the captured audio data, and the output is the audio data sent to the server. The transfer of the audio data makes it possible to proceed to the next analysis step.

[0142] Step 3: Analyzing the audio data

[0143] The server converts the received voice data into text data by running it through a speech recognition API such as Google Cloud Speech-to-Text or Amazon Transcribe. The input is the transmitted voice data, and the output is the generated text data. For example, the voice data "What is an influencer?" is converted into the text data "What is an influencer?"

[0144] Step 4: Identifying new words and loan words

[0145] The server analyzes the generated text data using natural language processing (NLP) techniques to identify new words and loan words. The input is the text data, and the output is a list of identified new words and loan words. For example, the word "influencer" is identified.

[0146] Step 5: Translation process

[0147] The server translates the identified new words and foreign words into a form that is easy for seniors to understand. It uses a generative AI model and a machine translation engine. The input is the identified new words and foreign words, and the output is the translated sentence. For example, the word "influencer" is translated into "influential person."

[0148] Step 6: Converting translation text to audio data

[0149] The server reconverts the translated text data into audio data using Google Cloud Text-to-Speech or Amazon Polly. The input is the translated text data, and the output is the reconverted audio data. For example, the text "influential person" is converted into audio data.

[0150] Step 7: Audio output to the user

[0151] The server sends the re-encoded voice data to the device using a secure communication protocol, and the device transmits the voice data to the user through earphones. The input is the re-encoded voice data, and the output is the voice the user hears in real time. The user hears the voice answer "Influential person" in response to the question "What is an influencer?"

[0152] Step 8: Collect biometric data

[0153] The device acquires the user's biometric information (e.g., pulse) in real time using a pulse sensor or the like. The input is a biometric signal such as the user's pulse, and the output is the acquired biometric data. For example, data on the user's pulse rate is collected.

[0154] Step 9: Sending biometric information

[0155] The device transmits the acquired biometric information to the server via a secure communication protocol. The input is the acquired biometric data, and the output is the biometric data transmitted to the server. The transfer of the biometric data enables the next analysis step.

[0156] Step 10: Analysis of biological information

[0157] The server then applies the received biometric data to an emotion analysis algorithm to evaluate the user's emotional state. The input is the transmitted biometric data, and the output is the analyzed emotional state. For example, a high pulse rate is evaluated as a state of tension.

[0158] Step 11: Provide emotionally relevant information

[0159] The server generates appropriate phrases and additional information based on the results of the emotion analysis, converts them into voice data, and provides them to the user. A generative AI model is used. The input is the analyzed emotional state and a prompt to generate appropriate phrases, and the output is the appropriate phrases and additional information converted into voice data. If the user is confused about a smartphone, advice such as "If you want to know more about the basic operations of a smartphone, it's best to learn by touching it directly" is provided as voice data.

[0160] (Application example 1)

[0161] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0162] The problem is that elderly people are confused by new words and foreign words, especially security-related terms, and have difficulty understanding them. Furthermore, when it comes to emergency responses, accurate and quick understanding is required, but elderly people are prone to feeling stressed and confused at times. In such situations, appropriate responses may not be possible, which could increase security risks.

[0163] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0164] In this invention, the server includes means for acquiring voice data, means for converting the acquired voice data into text data, means for identifying new words and foreign words contained in the text data and translating them into an easy-to-understand form, means for converting the translated text data into voice data and providing it to the user, means for acquiring the user's biometric information, means for analyzing the user's emotional state based on the acquired biometric information and converting the translated text data into appropriate phrases based on the analysis results, means for translating security-related terms and emergency response instructions into easy-to-understand forms and providing them to the user, and means for providing additional information to reduce the user's stress if the user's emotional state exceeds a reference value. This makes it easier for elderly people to understand security-related terms and emergency response instructions, enabling them to take necessary measures quickly and accurately.

[0165] The "means for acquiring voice data" refers to a device for capturing the user's voice in real time and acquiring the data.

[0166] "Means for converting acquired voice data into text data" refers to a device or software that uses voice recognition technology to convert voice data into text information.

[0167] "Means for identifying new words and foreign words contained in the text data and translating them into an easily understandable form" refers to a device or software that detects new or difficult terms from the converted text data and replaces them with words that are easy for seniors to understand.

[0168] "Means for converting translated text data into audio data and providing it to the user" refers to a device or software that plays back the translated text data as audio.

[0169] "Means for acquiring user's biometric information" refers to a device that acquires biometric data such as the user's pulse and skin potential in real time.

[0170] "Means for analyzing the user's emotional state based on the acquired biometric information and converting it into appropriate phrases based on the analysis results" refers to a device or software that analyzes biometric information to evaluate the user's emotional state and translates it into more appropriate expressions based on the results.

[0171] "Means for translating security-related terms and emergency response instructions into easy-to-understand language and providing them to users" refers to a device or software that translates difficult security-related terms and instructions into simple language and provides them to users as audio.

[0172] "Means for providing additional information to reduce stress in a user when the emotional state exceeds a reference value" refers to a device or software that provides supplemental information to reduce stress in a user when the analyzed emotional state exceeds a reference value.

[0173] System configuration and operation

[0174] The system consists of the following elements:

[0175] 1. Terminal: A device that acquires voice data and collects the user's biometric information.

[0176] 2. Server: A device that analyzes and translates voice data and generates additional information based on the translation results.

[0177] 3. Earphones: A device that provides translated text data as audio.

[0178] Acquiring and sending audio data

[0179] The user puts on the earphones and starts a normal conversation. The microphone in the device captures the user's voice in real time and transmits the voice data to the server. A secure communication protocol is used for this transmission, ensuring the safety of the data.

[0180] Speech data analysis and translation

[0181] The server then runs the received voice data through a speech recognition algorithm and converts it into text. It then identifies new or foreign words from this text and translates them into language that is easy for seniors to understand. For example, if the term "phishing attack" is detected, it will be translated into "a method of deceiving personal information."

[0182] Real-time audio output

[0183] The translated text data is converted back into audio data on the server side and sent to the user's earphones in real time, allowing the user to receive the translated information in a natural voice.

[0184] Collection and analysis of biological information

[0185] The device is equipped with a biometric information acquisition device, such as a pulse sensor, that acquires the user's biometric information in real time. This data is sent to a server, where an emotion analysis algorithm evaluates the user's emotional state. For example, a high pulse rate may indicate tension.

[0186] Providing information according to emotions

[0187] Based on the analysis results, the server prepares appropriate phrases and additional information. For example, if it detects that the user is confused by a particular word, it will provide additional explanations or convert the word into a more user-friendly expression. This converted information is also sent to the user's earphones in real time and delivered to the user's ears.

[0188] Specific examples

[0189] Example 1: In this scenario, if a user asks "What is a phishing attack?", the following will happen:

[0190] Users encounter a new term in security conversations: "phishing attack."

[0191] The device captures this audio and sends it to the server.

[0192] The server identifies "phishing attacks" and translates them into "tricks to trick people into revealing personal information."

[0193] The translated information is converted back into audio data and played in real time through the user's earphones.

[0194] At the same time, if the user's biometric information indicates a state of tension, additional information and advice is provided, such as "When it comes to phishing attacks, it's best to ignore the links and emails."

[0195] Example prompts to input to a generative AI model:

[0196] "How are phishing attacks used online?"

[0197] Hardware and software used

[0198] Hardware:

[0199] Devices (smartphones, smart glasses, head-mounted displays)

[0200] Biometric information acquisition device (pulse sensor)

[0201] software:

[0202] Speech Recognition Library: speech_recognition

[0203] Biosensor library: some_bio_sensor_library

[0204] Text-to-speech (TTS) library: pyttsx3

[0205] HTTP request library:requests

[0206] Server-side processing: NLP models, sentiment analysis algorithms, speech conversion engines

[0207] In this way, the present invention is a system that helps seniors become familiar with new words and information and improves their understanding and ability to respond to security-related issues.

[0208] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0209] Step 1:

[0210] The device acquires the user's voice data. Specifically, the device's microphone is used to capture the user's voice in real time. The acquired voice data is saved as raw voice waveform data.

[0211] Step 2:

[0212] The terminal transmits the acquired voice data to the server. Specifically, the terminal transfers the voice data to the server using a secure communication protocol. This communication ensures the security of the data.

[0213] Step 3:

[0214] The server runs the received voice data through a speech recognition algorithm and converts it into text data. Specifically, it uses a speech recognition library (e.g., speech_recognition) to convert the voice waveform data into corresponding text data. This conversion process uses an acoustic model and a language model.

[0215] Step 4:

[0216] The server identifies new words and foreign words contained in the converted text data. Specifically, it uses a generative AI model to analyze the text data and detect new words and foreign words. NLP technology is applied in this analysis.

[0217] Step 5:

[0218] The server translates the identified new words and foreign words into a form that is easy for seniors to understand. Specifically, the generative AI model refers to a translation dictionary and translates, for example, "phishing attack" into "a method of defrauding personal information." The translation results are saved as text data.

[0219] Step 6:

[0220] The server converts the translated text back into audio using a text-to-speech (TTS) library (e.g., pyttsx3). This audio is then given natural pronunciation and intonation.

[0221] Step 7:

[0222] The server transmits the re-converted audio data to the device in real time using a secure communication protocol. The device receives the audio data and immediately provides it to the user through earphones.

[0223] Step 8:

[0224] The device acquires biometric information. Specifically, it uses a pulse sensor built into the device to acquire biometric data such as the user's pulse and skin potential in real time. This data is saved as numerical data.

[0225] Step 9:

[0226] The terminal transmits the acquired biometric information to the server. Specifically, the biometric data is transferred to the server using a secure communication protocol. This communication ensures the security of the data.

[0227] Step 10:

[0228] The server analyzes the user's emotional state based on the acquired biometric information. Specifically, it uses a biometric analysis algorithm to evaluate, for example, whether the user is nervous. The evaluation results are saved as numerical data.

[0229] Step 11:

[0230] The server prepares appropriate phrases and additional information based on the analysis results. Specifically, the generative AI model uses the sentiment analysis results to generate additional advice, such as "Please ignore phishing attacks." This additional information is saved as text data.

[0231] Step 12:

[0232] The server converts the prepared additional information into audio data and sends it to the device. Specifically, it uses a text-to-speech (TTS) library to convert the text data of the additional information into audio data and transmits it to the device using a secure communication protocol. The device receives this audio data and provides it to the user through earphones.

[0233] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0234] MODE FOR CARRYING OUT THE INVENTION

[0235] The present invention is an AI earphone system that translates new words and foreign words in real time into a form that is easy for elderly people to understand, and also recognizes the user's emotions and provides information according to their state. Specific embodiments of this system are described below.

[0236] System configuration and operation

[0237] This system consists of a terminal that acquires voice data, a server that analyzes the acquired voice data and biometric information, earphones that provide information to the user, and an emotion engine that performs emotion recognition.

[0238] Acquiring and sending audio data

[0239] The user puts on the earphones and engages in normal conversation. The microphone in the device captures the user's voice in real time and transmits the voice data to the server. The voice data is transmitted to the server using a secure communication protocol, ensuring data security.

[0240] Speech data analysis and translation

[0241] The server converts the received voice data into text data using a speech recognition algorithm. New words and foreign words are identified from this text data and translated into words and expressions that are easy for seniors to understand. For example, if the word "influencer" is detected, it will be translated into "influential person."

[0242] Real-time audio output

[0243] The translated text data is converted back into audio data on the server side and sent to the user's earphones in real time, allowing the user to receive the translated information in a natural voice.

[0244] Biometric data collection and emotion analysis

[0245] A pulse sensor installed in the device captures the user's pulse data in real time. This data is sent to a server, where an emotion analysis algorithm evaluates the user's emotional state. The emotion engine then analyzes this data and adjusts the tone and intonation of the information according to the user's emotional state.

[0246] Providing information according to emotions

[0247] The emotion engine prepares appropriate phrases and additional information based on the user's emotional state. For example, if it senses that the user is confused by a particular word, it will provide additional explanation or convert that word into a more friendly expression. This converted information is also sent to the user's earphones in real time and delivered to the user's ears.

[0248] Specific examples

[0249] Example 1:

[0250] In the scenario where a user asks "What is an influencer?", the following happens:

[0251] Users encounter the new term "influencer" in everyday conversations.

[0252] The device captures this audio and sends it to the server.

[0253] The server identifies "influencer" and translates it to "influential person."

[0254] The translated information is converted back into audio data and played in real time through the user's earphones.

[0255] The emotion engine analyzes the user's pulse and adjusts the tone and intonation to make it easier for the user to understand.

[0256] Example 2:

[0257] In the scenario where a user asks "How do I use a smartphone?", the following happens:

[0258] Pulse data shows that users are confused by the use of the foreign word "smartphone."

[0259] The device captures the audio and sends it to the server.

[0260] The server identifies "smartphone" and translates it into "a mobile phone with many functions."

[0261] The translated information is converted back into audio data and played in real time through the user's earphones.

[0262] The emotion engine detects the user's confusion and provides additional information such as, "If you want to know more about the basic operations of your smartphone, it's best to learn by touching it."

[0263] In this way, the present invention is a system that helps elderly people become familiar with new words and information and realizes smooth communication in their daily lives. Furthermore, the introduction of an emotion engine makes it possible to provide more personalized information according to the user's emotional state, providing a comfortable user experience.

[0264] The processing flow will be explained below.

[0265] MODE FOR CARRYING OUT THE INVENTION

[0266] The present invention is an AI earphone system that translates new words and foreign words into a form that is easy for seniors to understand in real time, and also recognizes the user's emotions and provides information according to their state. A specific embodiment of this system will be described in detail below, with each processing step explained.

[0267] Step 1: Acquire audio

[0268] The user puts on the earphones and engages in normal conversation.

[0269] A microphone in the device captures the user's voice in real time.

[0270] Step 2: Sending audio data

[0271] The terminal packetizes the captured audio data and transmits it to the server.

[0272] The audio data is sent to the server using a secure communication protocol, ensuring data security.

[0273] Step 3: Voice Recognition

[0274] The server analyzes the received voice data and converts it into text data using a voice recognition algorithm.

[0275] For example, the spoken word "influencer" is converted into text data.

[0276] Step 4: Identifying new words

[0277] The server applies natural language processing to identify new and foreign words contained in the text data.

[0278] For example, "influencers" are identified.

[0279] Step 5: Translation process

[0280] The server translates each identified new word or foreign word into words or expressions that are easy for seniors to understand.

[0281] For example, convert "influencer" to "influencer."

[0282] Step 6: Audio translation

[0283] The server reconverts the translated text data into audio data using a speech synthesis algorithm.

[0284] Step 7: Submit your translation data

[0285] The server packetizes the generated voice data and transmits it to the terminal.

[0286] Step 8: Real-time output

[0287] The terminal plays back the received audio data in real time and provides it to the user.

[0288] Step 9: Acquire biometric information

[0289] The pulse sensor installed in the device collects the user's pulse data in real time.

[0290] Step 10: Sending biometric information

[0291] The terminal transmits the acquired pulse data to the server.

[0292] Step 11: Sentiment Analysis

[0293] The server analyzes the received pulse data and estimates the user's emotional state.

[0294] For example, if your pulse rate is high, it is determined that you are nervous.

[0295] Step 12: Adjusting Tone and Intonation

[0296] The server's emotional engine adjusts the tone and intonation of the translated words based on the user's emotional state.

[0297] For example, if the user is nervous, the tone is changed to a gentler tone.

[0298] Step 13: Generate additional information

[0299] The server's emotion engine translates appropriate phrases and additional explanations as needed based on the results of emotion analysis.

[0300] For example, if the user is confused, provide a more detailed explanation.

[0301] Step 14: Vocalize additional information

[0302] The server converts the generated additional text data into voice data using a voice synthesis algorithm.

[0303] Step 15: Submit additional information

[0304] The server transmits the additional information in the form of voice to the terminal.

[0305] Step 16: Playing Additional Audio

[0306] The terminal reproduces the received audio data of the additional information in real time and provides it to the user.

[0307] Through these processing steps, users can deepen their understanding of new words and foreign words and receive personalized information in real time according to their emotional state. This system will facilitate communication among the elderly.

[0308] Example 2

[0309] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0310] In today's information society, new words and foreign words are constantly appearing, and they are often difficult for the elderly to understand. There is also a lack of appropriate support to help them understand this new information. Furthermore, communication can sometimes be hindered by a lack of information provided in accordance with the user's emotional state. It is important to resolve these issues and provide appropriate information that makes it easier for the elderly to understand new words and information, and that responds to their emotional state.

[0311] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0312] In this invention, the server includes a means for converting voice data into text data, a means for identifying new words and foreign words contained in the text data and translating them into an easily understandable form, and a means for converting the translated text data into voice data and providing it to the user. This makes it possible to convert new words and foreign words into a form that is easy for elderly people to understand in real time, and to provide information according to the user's emotional state.

[0313] "Voice data" refers to a digital representation of spoken information uttered by a user.

[0314] "Text data" is voice data converted into a string of characters.

[0315] "New words" are recently introduced terms or phrases that are unfamiliar to a particular user.

[0316] "Foreign words" refer to words introduced from foreign countries that are difficult for certain users to understand.

[0317] "Translation means" refers to techniques or methods that convert identified new words or foreign words into a form that is easy for users to understand.

[0318] "Biometric information" is data obtained from the user's body, and includes pulse rate, heart rate, and the like.

[0319] "Emotional state" refers to the user's state of mind, and includes happiness, sadness, confusion, and the like.

[0320] "Emotion analysis" is the process of assessing a user's emotional state based on acquired biometric information.

[0321] A "voice recognition algorithm" is a technology that converts acquired voice data into text data in the form of a string.

[0322] A "speech synthesis service" is a technology or platform for converting text data into voice data.

[0323] A "server" is a computer system that analyzes voice data and biometric information and processes the data.

[0324] A "terminal" is a device that acquires voice data and biometric information from a user and transmits it to a server.

[0325] "Translated data" refers to text data that has been converted from identified new words or foreign words into an easily understandable form.

[0326] A "secure communications protocol" is a communications procedure used to ensure the security of information when transmitting and receiving data.

[0327] A "pulse sensor" is a device for measuring a user's pulse and acquiring biometric information.

[0328] The "emotion engine" is a system that analyzes acquired biometric information and provides information according to the user's emotional state.

[0329] This system translates new words and foreign words into a form that is easy for the elderly to understand in real time, and provides information according to the user's emotional state. This system consists of a terminal that acquires voice data, a server that analyzes the voice data and biometric information, earphones that provide information to the user, and an emotion engine that recognizes emotions.

[0330] Acquiring and sending audio data

[0331] The user puts on the earphones and engages in normal conversation. The microphone in the device captures the user's voice in real time and sends the voice data to the server. The data is sent using a secure communication protocol (e.g., HTTPS), ensuring data safety.

[0332] Speech data analysis and translation

[0333] The server converts the received voice data into text data using a speech recognition algorithm (e.g., Google Cloud Speech-to-Text API). New words and foreign words are identified from this text data and translated into words and expressions that are easy for seniors to understand. For example, if the word "influencer" is detected, it will be translated as "influential person."

[0334] Real-time audio output

[0335] The translated text data is converted back into audio data on the server side. This process uses a speech synthesis service (e.g., Amazon Polly). The converted audio data is sent to the user's earphones in real time, allowing the user to receive the translated information in a natural voice.

[0336] Biometric data collection and emotion analysis

[0337] A pulse sensor installed in the device captures the user's pulse data in real time. This data is sent to a server, where an emotion analysis algorithm evaluates the user's emotional state. The emotion engine then analyzes this data and adjusts the tone and intonation of the information according to the user's emotional state.

[0338] Providing information according to emotions

[0339] The emotion engine prepares appropriate phrases and additional information based on the user's emotional state. For example, if it senses that the user is confused by a particular word, it will provide additional explanation or convert that word into a more friendly expression. This converted information is also sent to the user's earphones in real time and delivered to the user's ears.

[0340] Specific examples

[0341] Example 1: "What is an influencer?"

[0342] Users encounter the new term "influencer" in everyday conversation.

[0343] The device captures this audio and sends it to the server.

[0344] The server identifies "influencer" and translates it to "influential person."

[0345] The translated information is converted back into audio data and played in real time through the user's earphones.

[0346] The emotion engine analyzes the user's pulse and adjusts the tone and intonation to make it easier for the user to understand.

[0347] Example 2: "How do you use a smartphone?"

[0348] Pulse data shows that users are confused by the use of the foreign word "smartphone."

[0349] The device captures the audio and sends it to the server.

[0350] The server identifies "smartphone" and translates it into "a mobile phone with many functions."

[0351] The translated information is converted back into audio data and played in real time through the user's earphones.

[0352] The emotion engine detects the user's confusion and provides additional information, such as, "If you want to know more about the basic operations of your smartphone, it's best to learn by touching it."

[0353] Prompt Sentence Examples

[0354] Please explain what kind of hardware and software you will use for the specific process of "acquiring and transmitting voice data."

[0355]

[0356] Regarding "providing information based on emotions," please explain what algorithms or methods are used to analyze the user's emotions and provide information.

[0357] This system is designed to help elderly people become familiar with new words and information, enabling smooth communication. The introduction of an emotion engine makes it possible to provide more personalized information according to the user's emotional state, providing a comfortable user experience.

[0358] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0359] Step 1: Acquire audio data

[0360] The device captures the user's voice, and the microphone in the earphone collects ambient sounds and the user's speech in real time.

[0361] Input: User's speech

[0362] Data processing: Apply noise reduction algorithm to convert into clear audio data.

[0363] Output: Digital audio data

[0364] What it does: A microphone in the device captures an audio signal and converts it into a digital format.

[0365] Step 2: Sending audio data

[0366] The device sends the captured audio data to the server, where it encrypts the data using a secure communication protocol (e.g., HTTPS) to ensure the data is secure during transmission.

[0367] Input: Digital audio data

[0368] Data processing: Audio data is divided into packets and encrypted using HTTPS.

[0369] Output: Audio data sent to the server

[0370] Specific operation: The device sends audio data to the server via HTTPS.

[0371] Step 3: Convert audio data to text

[0372] The server converts the received audio data into text using a speech recognition algorithm (e.g., a generative AI model). At this stage, the temporal attributes of the audio are also preserved.

[0373] Input: Audio data sent to the server

[0374] Data calculation: Applying a speech recognition algorithm to generate text data.

[0375] Output: Text data

[0376] Specific operation: The server analyzes the audio data and converts it into text data in string format.

[0377] Step 4: Identifying and translating new or foreign words

[0378] The server identifies new words and foreign words from the converted text data, and these words are translated into words and expressions that are easy for seniors to understand using a translation dictionary in the database.

[0379] Input: Text data

[0380] Data calculation: Identifying new words and foreign words, and matching them with translation dictionaries

[0381] Output: Translated text data

[0382] How it works: The server analyzes the text data and matches the relevant terms with a database to obtain the translation.

[0383] Step 5: Re-speech of text data

[0384] The server converts the translated text data back into voice data using a speech synthesis service (e.g., speech synthesis API). This voice is adjusted to have a natural and friendly tone.

[0385] Input: Translated text data

[0386] Data calculation: Apply a voice synthesis algorithm to generate voice data.

[0387] Output: Audio data

[0388] Specific operation: The server sends text data to the speech synthesis API and obtains the synthesized speech data.

[0389] Step 6: Transmit translation information to earphone

[0390] The server then transmits the re-audio data to the user's earphones in real time, again using a secure communication protocol to ensure the data is safe.

[0391] Input: Revoiced audio data

[0392] Data processing: Audio data is divided into packets and encrypted using HTTPS.

[0393] Output: The audio data that reaches the user

[0394] Specific operation: The server sends audio data to the device using HTTPS, and the device outputs it to the earphones.

[0395] Step 7: Acquire biometric information

[0396] The device uses a built-in pulse sensor to acquire the user's pulse data in real time, and this data is sent to a server.

[0397] Input: User's pulse signal

[0398] Data processing: Convert the signal into digital pulse data.

[0399] Output: Digital pulse data

[0400] How it works: A sensor inside the device captures pulse signals and converts them into digital data.

[0401] Step 8: Sentiment Analysis

[0402] The server analyzes the user's emotional state using the received pulse data, and classifies the user's emotions based on the data using an emotion analysis algorithm.

[0403] Input: Pulse data

[0404] Data Computation: Apply sentiment analysis algorithms to assess emotional states.

[0405] Output: Emotional state data

[0406] Specific operation: The server analyzes the pulse data and evaluates the emotional state using a probabilistic model.

[0407] Step 9: Provide emotionally relevant information

[0408] The emotion engine prepares appropriate phrases and additional information based on the analysis results, and the converted information is also sent to the user's earphones as audio data in real time.

[0409] Input: Emotional state data, text data

[0410] Data calculation: Text data generation, speech synthesis

[0411] Output: Audio data

[0412] Specific operation: The emotion engine generates text data, and the server converts it into audio data using a speech synthesis API and sends it to the earphones.

[0413] These are the processing steps of this system, which makes it easier for elderly people to understand new terms and provides information that is appropriate for their emotional state.

[0414] (Application example 2)

[0415] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0416] Elderly people often have difficulty understanding the content of explanations of new products or services in brick-and-mortar stores. New words and foreign words are particularly difficult for elderly people to understand. Furthermore, depending on the elderly person's emotional state, information may not be provided at the appropriate time or in the appropriate tone, making it difficult for them to make appropriate decisions. This invention aims to make it easier for such elderly people to understand product explanations in brick-and-mortar stores and to provide information that adapts to the elderly person's emotional state.

[0417] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0418] In this invention, the server includes means for acquiring voice data, means for converting the acquired voice data into text data, means for identifying new words and foreign words contained in the text data and translating them into an easy-to-understand form, means for converting the translated text data into voice data and providing it to the user, means for acquiring biometric information of the user, means for analyzing the emotional state of the user based on the acquired biometric information and converting the text data into appropriate wording based on the analysis results, and means for translating and recognizing emotions when the user is receiving an explanation of new products or services in a physical store and providing the user with information in real time. This makes it possible to provide elderly people with information that is easy to understand and appropriate to their emotional state when receiving an explanation of new products or services in a physical store.

[0419] "Speech data" refers to data that records sound information collected from the user's speech and the surrounding sound environment.

[0420] "Text data" is character information obtained by processing voice data for recognition.

[0421] "New words and foreign words" are words that are difficult for users to understand or that they do not normally use, or words that have been imported from other languages.

[0422] "Translation" is the process of converting identified text data into a form that is easy for elderly people to understand.

[0423] "Speech recognition means" refers to a device or software for converting voice data into text data.

[0424] "Biometric information" refers to information including the user's pulse rate and other physiological data.

[0425] "Emotional state" refers to a user's current psychological state or feelings.

[0426] The "emotion recognition means" is a device or software for analyzing the user's emotional state from collected biometric information.

[0427] "Real-time" means acting immediately, without delay.

[0428] A "brick and mortar store" is a place that offers goods or services in a physical location.

[0429] A "user" is a person who uses this system.

[0430] This invention is a system that helps elderly people understand the content of new products and services in real time when they are explained in a physical store. The system collects and translates voice data, collects biometric information from users, and analyzes their emotions, and provides appropriate information based on this information.

[0431] composition:

[0432] The system consists of the following elements:

[0433] 1. Device that receives audio data:

[0434] A microphone in the device captures the user's voice in real time, for example, smart glasses or a smartphone.

[0435] 2. Server that analyzes voice data and biometric information:

[0436] The server converts the received voice data into text data using a speech recognition algorithm (e.g., Google Speech-to-Text API), while simultaneously analyzing biometric information.

[0437] 3. Earphones that provide information to the user:

[0438] Earphones to convert the translated text data back into audio data and provide it to the user in real time. The built-in speakers of smart glasses can also be used.

[0439] 4. Emotion engine for emotion recognition:

[0440] The device's built-in pulse sensor captures the user's pulse data in real time and sends it to a server. Based on this data, an emotion analysis algorithm evaluates the user's emotional state.

[0441] Operation:

[0442] 1. Audio data capture and transmission:

[0443] When a user asks a question about a product or receives an explanation in a store, the device's microphone captures the voice and transmits it to the server. The voice data is transmitted using a secure communication protocol.

[0444] 2. Speech data analysis and translation:

[0445] The server uses a speech recognition algorithm to convert the received voice data into text data, and identifies new or foreign words from this text data. For example, if the phrase "detox water" is included, it translates it into an easy-to-understand form (e.g., "water that flushes toxins from the body").

[0446] 3. Real-time audio output:

[0447] The translated text data is converted back into audio data on the server side and sent to the user's earphones in real time.

[0448] 4. Biometric data collection and emotion analysis:

[0449] The user's pulse sensor captures pulse data in real time and sends it to a server, where an emotion recognition algorithm analyzes it and provides appropriate information according to the user's emotional state.

[0450] Examples:

[0451] Example 1:

[0452] When seniors ask about "detox water" and new products.

[0453] Salesperson: "This is the latest detox water. It has detoxifying effects."

[0454] Seniors: "What's a detox?"

[0455] Example prompt sentence:

[0456] "What is detox water?"

[0457] "Tell me more about this new product."

[0458] Please explain the ingredients

[0459] In this situation, the system captures the elderly person's question, translates the new term "detox" into "something that has the function of expelling toxins from the body," and provides this information in audio. If the system senses that the elderly person is confused, it provides additional information such as, "We can also explain in detail the effects of detox water here."

[0460] Recommended hardware and software:

[0461] Speech Recognition: Google Speech-to-Text API

[0462] Translation: Dedicated translation server (e.g. Google Cloud Translation API)

[0463] Emotion Recognition: Dedicated emotion analysis algorithms (e.g., SomeEmotionRecognitionLibrary)

[0464] Audio output: pyttsx3 library (Python)

[0465] This means that when elderly people are explained about new products or services in a physical store, they can receive information in an easy-to-understand format immediately, helping them make appropriate decisions.

[0466] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0467] Step 1:

[0468] The terminal captures the user's voice data.

[0469] Input: User's voice

[0470] Output: Audio data (analog)

[0471] Processing: A microphone on the device picks up the user's voice in real time.

[0472] Specific operation: The words spoken by the user are captured by a microphone and stored as audio data on the device.

[0473] Step 2:

[0474] The device transmits the captured audio data to the server.

[0475] Input: Audio data (analog)

[0476] Output: Audio data (digital)

[0477] Processing: The audio data is converted into a digital format and sent to a server using a secure communication protocol.

[0478] Specific operation: The voice data in the device is converted into a digital format and sent to a server via the Internet.

[0479] Step 3:

[0480] The server converts the voice data into text data.

[0481] Input: Audio data (digital)

[0482] Output: Text data

[0483] Processing: Convert the audio data into text using a speech recognition algorithm (e.g., Google Speech-to-Text API).

[0484] Specific operation: The server analyzes the voice data and converts the spoken content into text information.

[0485] Step 4:

[0486] The server identifies new words and foreign words and translates them into a form that is easy for seniors to understand.

[0487] Input: Text data

[0488] Output: Translated text data

[0489] Processing: Identified new or foreign words are translated using a translation algorithm (e.g., Google Cloud Translation API).

[0490] Specific action: Translate the word "detox" as "the function of expelling toxins from the body."

[0491] Step 5:

[0492] The server converts the translated text data back into audio data.

[0493] Input: Translated text data

[0494] Output: Regenerated audio data

[0495] Processing: Convert the text data into audio data using a text-to-speech (TTS) algorithm (e.g., the pyttsx3 library).

[0496] Specific behavior: Regenerate text information written in different languages ​​as natural-sounding speech.

[0497] Step 6:

[0498] The terminal transmits the regenerated audio data to the earphone.

[0499] Input: Regenerated audio data

[0500] Output: Audio data delivered to earphones

[0501] Processing: The regenerated audio data is sent from the device to the earphones via a secure communication protocol.

[0502] Specific operation: Audio data sent from the server is delivered to the earphones in real time, and the user listens to the audio.

[0503] Step 7:

[0504] The device acquires the user's biometric information (such as pulse rate) and sends it to the server.

[0505] Input: Biometric information (pulse data, etc.)

[0506] Output: Biometric information (digital data)

[0507] Processing: Biometric data is acquired from the sensor device and sent to the server.

[0508] Specific operation: Acquires biometric data such as pulse rate and electrodermal activity (EDA) and sends it to a server via the Internet.

[0509] Step 8:

[0510] The server analyzes the emotional state based on biometric information and adds appropriate phrases.

[0511] Input: Biometric information (digital data)

[0512] Output: Sentiment analysis results and properly adjusted translation data

[0513] Processing: Emotion recognition algorithms analyze biometric data to assess the user's emotional state, and generate appropriate tones and additional supportive information based on the results.

[0514] What it does: If the user is confused, add a message to the translated information saying "Please let us know if you need further clarification."

[0515] In this way, the specific flow of implementing the invention based on the claims has been explained by dividing it into processing steps.

[0516] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0517] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0518] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0519] [Second embodiment]

[0520] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0521] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0522] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0523] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0524] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0525] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0526] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0527] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0528] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0529] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0530] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0531] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0532] MODE FOR CARRYING OUT THE INVENTION

[0533] The present invention is an AI earphone system that translates new words and foreign words used in everyday life into a form that is easy for elderly people to understand in real time. Specific embodiments of this system are described below.

[0534] System configuration and operation

[0535] This system consists of three main elements: a terminal for acquiring voice data, a server for analyzing the acquired voice data and biometric information, and earphones for providing information to the user.

[0536] Acquiring and sending audio data

[0537] The user puts on the earphones and engages in normal conversation. The microphone in the device captures the user's voice in real time and transmits the voice data to the server. The voice data is transmitted to the server using a secure communication protocol, ensuring data security.

[0538] Speech data analysis and translation

[0539] The server then runs the received voice data through a speech recognition algorithm and converts it into text data. It then identifies new words and foreign words from this text data and translates them into words and expressions that are easy for seniors to understand. For example, if the word "influencer" is detected, it will be translated as "influential person."

[0540] Real-time audio output

[0541] The translated text data is converted back into audio data on the server side and sent to the user's earphones in real time, allowing the user to receive the translated information in a natural voice.

[0542] Collection and analysis of biological information

[0543] The device is equipped with a biometric information acquisition device such as a pulse sensor that acquires the user's biometric information in real time. This data is sent to a server, where an emotion analysis algorithm evaluates the user's emotional state. For example, a high pulse rate may indicate tension.

[0544] Providing information according to emotions

[0545] Based on the analysis results, the server prepares appropriate phrases and additional information. For example, if it detects that the user is confused by a particular word, it will provide additional explanation for that word or convert it into a more familiar expression. This converted information is also sent to the user's earphones in real time and delivered to the user's ears.

[0546] Specific examples

[0547] Example 1:

[0548] In the scenario where a user asks "What is an influencer?", the following happens:

[0549] Users encounter the new term "influencer" in everyday conversations.

[0550] The device captures this audio and sends it to the server.

[0551] The server identifies "influencer" and translates it to "influential person."

[0552] The translated information is converted back into audio data and played in real time through the user's earphones.

[0553] Example 2:

[0554] In the scenario where a user asks "How do I use a smartphone?", the following happens:

[0555] Pulse data shows that users are confused by the use of the foreign word "smartphone."

[0556] The device captures the audio and sends it to the server.

[0557] The server identifies "smartphone" and translates it into "a mobile phone with many functions."

[0558] The translated information is converted back into audio data and played in real time through the user's earphones.

[0559] Additionally, the server detects the user's confusion and provides additional information such as, "If you want to know more about the basic operations of your smartphone, it's best to learn by touching it directly."

[0560] In this way, the present invention is a system that helps elderly people become familiar with new words and information and realizes smooth communication in their daily lives.

[0561] The processing flow will be explained below.

[0562] Specific processing steps of the program

[0563] Step 1: Acquire audio

[0564] The user puts on the earphones and engages in normal conversation.

[0565] A microphone built into the device captures the user's voice in real time.

[0566] Step 2: Sending audio data

[0567] The device then packetizes the captured audio data and sends it to the server using a secure protocol.

[0568] Step 3: Voice Recognition

[0569] The server analyzes the received voice data and converts it into text data using a voice recognition algorithm.

[0570] Step 4: Identifying new words

[0571] The server applies natural language processing to identify new and foreign words contained in the text data.

[0572] Step 5: Translation process

[0573] The server translates each identified new or foreign word into a word or phrase that is easy for seniors to understand, for example, converting "influencer" into "influential person."

[0574] Step 6: Audio translation

[0575] The server reconverts the translated text data into audio data using a speech synthesis algorithm.

[0576] Step 7: Submit your translation data

[0577] The server packetizes the generated voice data and transmits it to the terminal.

[0578] Step 8: Real-time output

[0579] The terminal plays back the received audio data in real time and provides it to the user.

[0580] Step 9: Acquire biometric information

[0581] A pulse sensor installed in the device acquires the user's pulse data.

[0582] Step 10: Sending biometric information

[0583] The terminal transmits the acquired pulse data to the server.

[0584] Step 11: Sentiment Analysis

[0585] The server analyzes the received pulse data and estimates the user's emotional state. For example, if the pulse rate is high, it is determined that the user is nervous.

[0586] Step 12: Generate additional information

[0587] The server then translates the message based on the results of the sentiment analysis, providing appropriate phrasing and additional explanations as needed. For example, if the user is confused, a more detailed explanation is added.

[0588] Step 13: Vocalize additional information

[0589] The server converts the generated additional text data into voice data using a voice synthesis algorithm.

[0590] Step 14: Submit additional information

[0591] The server transmits the additional information in the form of voice to the terminal.

[0592] Step 15: Play additional audio

[0593] The terminal reproduces the received audio data of the additional information in real time and provides it to the user.

[0594] These are the specific processing steps of "Generation Communicator." Through this processing flow, elderly people can easily understand new words and foreign words, realizing comfortable communication.

[0595] Example 1

[0596] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0597] In the past, it was often difficult for elderly people to understand new words and foreign words used in daily life. Furthermore, there was no system that could analyze and translate speech data in real time and provide appropriate information according to the user's emotional state. This made it difficult for elderly people to communicate smoothly when they encountered new words or concepts. The purpose of this invention is to solve this problem.

[0598] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0599] In this invention, the server includes a means for converting voice data into text data, a means for translating new words and foreign words contained in the text data into a form that is easy for seniors to understand, and a means for reconverting the translated text data into voice data. This enables seniors to understand new words and foreign words used in everyday conversation in real time, enabling smooth communication. Furthermore, by analyzing the user's emotional state based on biometric information and providing appropriate phrases and additional information based on the analysis results, a system that is more user-friendly and easy to use can be provided. This allows seniors to communicate smoothly without being confused by new words and information.

[0600] "Audio data" refers to data in the form of sound waves, such as user conversations or environmental sounds.

[0601] A "terminal" is a device for acquiring voice data and biometric information and transmitting that data to a server.

[0602] A "server" is a computer system that analyzes, transforms, and presents information to users.

[0603] "Text data" refers to data obtained by converting voice data into character information.

[0604] "New words and foreign words" are words that have been introduced into the user's native language in a relatively short period of time or words that have been adopted from a foreign country.

[0605] "Translation" is the act of converting something from one language into another language or into an understandable form.

[0606] "Biometric information" refers to physical data such as the user's pulse rate and body temperature.

[0607] "Emotional state" refers to a psychological state estimated based on the user's pulse rate and other biometric information.

[0608] An "emotion analysis algorithm" is a computational method for analyzing biometric information and assessing a user's emotional state.

[0609] "Additional information" refers to supplemental information or explanations provided depending on the user's emotional state.

[0610] A "secure communication protocol" is a communication protocol for safely sending and receiving data.

[0611] A "generative AI model" is an artificial intelligence model that learns patterns from large amounts of data and generates new information.

[0612] The present invention is a system that acquires, analyzes, and translates voice data in real time to provide users with easy-to-understand information about new words and foreign words. A specific embodiment of the present invention will be described. The system comprises a terminal for acquiring voice data, a server for analyzing the acquired voice data and biometric information, and earphones for providing translated information to the user.

[0613] Acquiring audio data

[0614] The device captures the user's voice in real time using a built-in microphone when they are having a normal conversation. The voice data is then temporarily stored on the device. For example, if a user says, "Young people these days are only chasing influencers," the voice data will be captured by the device.

[0615] Sending audio data

[0616] The terminal transmits the captured audio data to the server using a secure communication protocol (for example, HTTPS or TLS).

[0617] Analysis of audio data

[0618] The server converts the received voice data into text data using a speech recognition API such as Google Cloud Speech-to-Text or Amazon Transcribe. For example, the generated text data would be, "Young people these days are only chasing influencers."

[0619] Identifying and translating new and foreign words

[0620] The server analyzes the generated text data and identifies new words and foreign words (e.g., "influencer") using natural language processing (NLP) technology. It then translates these new words and foreign words into a form that is easy for seniors to understand (e.g., "influencer" becomes "influential person") using a pre-trained generative AI model or machine translation engine.

[0621] Converting translation text to audio data

[0622] The server reconverts the translated text data into audio data using a speech synthesis API such as Google Cloud Text-to-Speech or Amazon Polly.

[0623] Audio output to the user

[0624] The translated voice data is then sent to the device again using a secure communication protocol and finally output to the user's earphones. The user can listen to the translated voice in real time. For example, the translated voice may provide the user with information such as, "Young people these days only chase after influential people."

[0625] Collection and analysis of biological information

[0626] The device is equipped with a biometric information acquisition device such as a pulse sensor, which acquires the user's biometric information (e.g., pulse) in real time and temporarily stores the data. This biometric information is also transmitted to a server using a secure communication protocol, and the server evaluates the user's emotional state (e.g., tension) using an emotion analysis algorithm.

[0627] Providing information according to emotions

[0628] The server generates appropriate phrases and additional information based on the analysis results. For example, if it determines that the user is confused by the foreign word "smartphone," it generates advice such as, "If you want to learn more about the basic operations of a smartphone, it's a good idea to learn by touching it directly," and then converts this into audio data using Google Cloud Text-to-Speech and provides it to the user.

[0629] In this way, this system helps elderly people become familiar with new words and information, enabling smooth communication in their daily lives.

[0630] Examples of prompt statements

[0631] The following are examples of prompt sentences:

[0632] Capture conversations that include new foreign words or technical terms, translate them in real time to make them understandable for seniors, and provide appropriate phrasing or additional information based on the user's emotional state.

[0633] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0634] The flow of this system's program processing

[0635] Step 1: Acquire audio data

[0636] The device uses a built-in microphone to capture the user's voice in real time. The input is the user's voice, and the output is the captured voice data. This voice data is immediately and temporarily stored on the device. For example, when a user says, "What is an influencer?", the voice is captured.

[0637] Step 2: Sending audio data

[0638] The device sends the captured audio data to the server using a secure communication protocol (HTTPS or TLS). The input is the captured audio data, and the output is the audio data sent to the server. The transfer of the audio data makes it possible to proceed to the next analysis step.

[0639] Step 3: Analyzing the audio data

[0640] The server converts the received voice data into text data by running it through a speech recognition API such as Google Cloud Speech-to-Text or Amazon Transcribe. The input is the transmitted voice data, and the output is the generated text data. For example, the voice data "What is an influencer?" is converted into the text data "What is an influencer?"

[0641] Step 4: Identifying new words and loan words

[0642] The server analyzes the generated text data using natural language processing (NLP) techniques to identify new words and loan words. The input is the text data, and the output is a list of identified new words and loan words. For example, the word "influencer" is identified.

[0643] Step 5: Translation process

[0644] The server translates the identified new words and foreign words into a form that is easy for seniors to understand. It uses a generative AI model and a machine translation engine. The input is the identified new words and foreign words, and the output is the translated sentence. For example, the word "influencer" is translated into "influential person."

[0645] Step 6: Converting translation text to audio data

[0646] The server reconverts the translated text data into audio data using Google Cloud Text-to-Speech or Amazon Polly. The input is the translated text data, and the output is the reconverted audio data. For example, the text "influential person" is converted into audio data.

[0647] Step 7: Audio output to the user

[0648] The server sends the re-encoded voice data to the device using a secure communication protocol, and the device transmits the voice data to the user through earphones. The input is the re-encoded voice data, and the output is the voice the user hears in real time. The user hears the voice answer "Influential person" in response to the question "What is an influencer?"

[0649] Step 8: Collect biometric data

[0650] The device acquires the user's biometric information (e.g., pulse) in real time using a pulse sensor or the like. The input is a biometric signal such as the user's pulse, and the output is the acquired biometric data. For example, data on the user's pulse rate is collected.

[0651] Step 9: Sending biometric information

[0652] The device transmits the acquired biometric information to the server via a secure communication protocol. The input is the acquired biometric data, and the output is the biometric data transmitted to the server. The transfer of the biometric data enables the next analysis step.

[0653] Step 10: Analysis of biological information

[0654] The server then applies the received biometric data to an emotion analysis algorithm to evaluate the user's emotional state. The input is the transmitted biometric data, and the output is the analyzed emotional state. For example, a high pulse rate is evaluated as a state of tension.

[0655] Step 11: Provide emotionally relevant information

[0656] The server generates appropriate phrases and additional information based on the results of the emotion analysis, converts them into voice data, and provides them to the user. A generative AI model is used. The input is the analyzed emotional state and a prompt to generate appropriate phrases, and the output is the appropriate phrases and additional information converted into voice data. If the user is confused about a smartphone, advice such as "If you want to know more about the basic operations of a smartphone, it's best to learn by touching it directly" is provided as voice data.

[0657] (Application example 1)

[0658] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0659] The problem is that elderly people are confused by new words and foreign words, especially security-related terms, and have difficulty understanding them. Furthermore, when it comes to emergency responses, accurate and quick understanding is required, but elderly people are prone to feeling stressed and confused at times. In such situations, appropriate responses may not be possible, which could increase security risks.

[0660] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0661] In this invention, the server includes means for acquiring voice data, means for converting the acquired voice data into text data, means for identifying new words and foreign words contained in the text data and translating them into an easy-to-understand form, means for converting the translated text data into voice data and providing it to the user, means for acquiring the user's biometric information, means for analyzing the user's emotional state based on the acquired biometric information and converting the translated text data into appropriate phrases based on the analysis results, means for translating security-related terms and emergency response instructions into easy-to-understand forms and providing them to the user, and means for providing additional information to reduce the user's stress if the user's emotional state exceeds a reference value. This makes it easier for elderly people to understand security-related terms and emergency response instructions, enabling them to take necessary measures quickly and accurately.

[0662] The "means for acquiring voice data" refers to a device for capturing the user's voice in real time and acquiring the data.

[0663] "Means for converting acquired voice data into text data" refers to a device or software that uses voice recognition technology to convert voice data into text information.

[0664] "Means for identifying new words and foreign words contained in the text data and translating them into an easily understandable form" refers to a device or software that detects new or difficult terms from the converted text data and replaces them with words that are easy for seniors to understand.

[0665] "Means for converting translated text data into audio data and providing it to the user" refers to a device or software that plays back the translated text data as audio.

[0666] "Means for acquiring user's biometric information" refers to a device that acquires biometric data such as the user's pulse and skin potential in real time.

[0667] "Means for analyzing the user's emotional state based on the acquired biometric information and converting it into appropriate phrases based on the analysis results" refers to a device or software that analyzes biometric information to evaluate the user's emotional state and translates it into more appropriate expressions based on the results.

[0668] "Means for translating security-related terms and emergency response instructions into easy-to-understand language and providing them to users" refers to a device or software that translates difficult security-related terms and instructions into simple language and provides them to users as audio.

[0669] "Means for providing additional information to reduce stress in a user when the emotional state exceeds a reference value" refers to a device or software that provides supplemental information to reduce stress in a user when the analyzed emotional state exceeds a reference value.

[0670] System configuration and operation

[0671] The system consists of the following elements:

[0672] 1. Terminal: A device that acquires voice data and collects the user's biometric information.

[0673] 2. Server: A device that analyzes and translates voice data and generates additional information based on the translation results.

[0674] 3. Earphones: A device that provides translated text data as audio.

[0675] Acquiring and sending audio data

[0676] The user puts on the earphones and starts a normal conversation. The microphone in the device captures the user's voice in real time and transmits the voice data to the server. A secure communication protocol is used for this transmission, ensuring the safety of the data.

[0677] Speech data analysis and translation

[0678] The server then runs the received voice data through a speech recognition algorithm and converts it into text. It then identifies new or foreign words from this text and translates them into language that is easy for seniors to understand. For example, if the term "phishing attack" is detected, it will be translated into "a method of deceiving personal information."

[0679] Real-time audio output

[0680] The translated text data is converted back into audio data on the server side and sent to the user's earphones in real time, allowing the user to receive the translated information in a natural voice.

[0681] Collection and analysis of biological information

[0682] The device is equipped with a biometric information acquisition device, such as a pulse sensor, that acquires the user's biometric information in real time. This data is sent to a server, where an emotion analysis algorithm evaluates the user's emotional state. For example, a high pulse rate may indicate tension.

[0683] Providing information according to emotions

[0684] Based on the analysis results, the server prepares appropriate phrases and additional information. For example, if it detects that the user is confused by a particular word, it will provide additional explanations or convert the word into a more user-friendly expression. This converted information is also sent to the user's earphones in real time and delivered to the user's ears.

[0685] Specific examples

[0686] Example 1: In this scenario, if a user asks "What is a phishing attack?", the following will happen:

[0687] Users encounter a new term in security conversations: "phishing attack."

[0688] The device captures this audio and sends it to the server.

[0689] The server identifies "phishing attacks" and translates them into "tricks to trick people into revealing personal information."

[0690] The translated information is converted back into audio data and played in real time through the user's earphones.

[0691] At the same time, if the user's biometric information indicates a state of tension, additional information and advice is provided, such as "When it comes to phishing attacks, it's best to ignore the links and emails."

[0692] Example prompts to input to a generative AI model:

[0693] "How are phishing attacks used online?"

[0694] Hardware and software used

[0695] Hardware:

[0696] Devices (smartphones, smart glasses, head-mounted displays)

[0697] Biometric information acquisition device (pulse sensor)

[0698] software:

[0699] Speech Recognition Library: speech_recognition

[0700] Biosensor library: some_bio_sensor_library

[0701] Text-to-speech (TTS) library: pyttsx3

[0702] HTTP request library:requests

[0703] Server-side processing: NLP models, sentiment analysis algorithms, speech conversion engines

[0704] In this way, the present invention is a system that helps seniors become familiar with new words and information and improves their understanding and ability to respond to security-related issues.

[0705] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0706] Step 1:

[0707] The device acquires the user's voice data. Specifically, the device's microphone is used to capture the user's voice in real time. The acquired voice data is saved as raw voice waveform data.

[0708] Step 2:

[0709] The terminal transmits the acquired voice data to the server. Specifically, the terminal transfers the voice data to the server using a secure communication protocol. This communication ensures the security of the data.

[0710] Step 3:

[0711] The server runs the received voice data through a speech recognition algorithm and converts it into text data. Specifically, it uses a speech recognition library (e.g., speech_recognition) to convert the voice waveform data into corresponding text data. This conversion process uses an acoustic model and a language model.

[0712] Step 4:

[0713] The server identifies new words and foreign words contained in the converted text data. Specifically, it uses a generative AI model to analyze the text data and detect new words and foreign words. NLP technology is applied in this analysis.

[0714] Step 5:

[0715] The server translates the identified new words and foreign words into a form that is easy for seniors to understand. Specifically, the generative AI model refers to a translation dictionary and translates, for example, "phishing attack" into "a method of defrauding personal information." The translation results are saved as text data.

[0716] Step 6:

[0717] The server converts the translated text back into audio using a text-to-speech (TTS) library (e.g., pyttsx3). This audio is then given natural pronunciation and intonation.

[0718] Step 7:

[0719] The server transmits the re-converted audio data to the device in real time using a secure communication protocol. The device receives the audio data and immediately provides it to the user through earphones.

[0720] Step 8:

[0721] The device acquires biometric information. Specifically, it uses a pulse sensor built into the device to acquire biometric data such as the user's pulse and skin potential in real time. This data is saved as numerical data.

[0722] Step 9:

[0723] The terminal transmits the acquired biometric information to the server. Specifically, the biometric data is transferred to the server using a secure communication protocol. This communication ensures the security of the data.

[0724] Step 10:

[0725] The server analyzes the user's emotional state based on the acquired biometric information. Specifically, it uses a biometric analysis algorithm to evaluate, for example, whether the user is nervous. The evaluation results are saved as numerical data.

[0726] Step 11:

[0727] The server prepares appropriate phrases and additional information based on the analysis results. Specifically, the generative AI model uses the sentiment analysis results to generate additional advice, such as "Please ignore phishing attacks." This additional information is saved as text data.

[0728] Step 12:

[0729] The server converts the prepared additional information into audio data and sends it to the device. Specifically, it uses a text-to-speech (TTS) library to convert the text data of the additional information into audio data and transmits it to the device using a secure communication protocol. The device receives this audio data and provides it to the user through earphones.

[0730] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0731] MODE FOR CARRYING OUT THE INVENTION

[0732] The present invention is an AI earphone system that translates new words and foreign words in real time into a form that is easy for elderly people to understand, and also recognizes the user's emotions and provides information according to their state. Specific embodiments of this system are described below.

[0733] System configuration and operation

[0734] This system consists of a terminal that acquires voice data, a server that analyzes the acquired voice data and biometric information, earphones that provide information to the user, and an emotion engine that performs emotion recognition.

[0735] Acquiring and sending audio data

[0736] The user puts on the earphones and engages in normal conversation. The microphone in the device captures the user's voice in real time and transmits the voice data to the server. The voice data is transmitted to the server using a secure communication protocol, ensuring data security.

[0737] Speech data analysis and translation

[0738] The server converts the received voice data into text data using a speech recognition algorithm. New words and foreign words are identified from this text data and translated into words and expressions that are easy for seniors to understand. For example, if the word "influencer" is detected, it will be translated into "influential person."

[0739] Real-time audio output

[0740] The translated text data is converted back into audio data on the server side and sent to the user's earphones in real time, allowing the user to receive the translated information in a natural voice.

[0741] Biometric data collection and emotion analysis

[0742] A pulse sensor installed in the device captures the user's pulse data in real time. This data is sent to a server, where an emotion analysis algorithm evaluates the user's emotional state. The emotion engine then analyzes this data and adjusts the tone and intonation of the information according to the user's emotional state.

[0743] Providing information according to emotions

[0744] The emotion engine prepares appropriate phrases and additional information based on the user's emotional state. For example, if it senses that the user is confused by a particular word, it will provide additional explanation or convert that word into a more friendly expression. This converted information is also sent to the user's earphones in real time and delivered to the user's ears.

[0745] Specific examples

[0746] Example 1:

[0747] In the scenario where a user asks "What is an influencer?", the following happens:

[0748] Users encounter the new term "influencer" in everyday conversations.

[0749] The device captures this audio and sends it to the server.

[0750] The server identifies "influencer" and translates it to "influential person."

[0751] The translated information is converted back into audio data and played in real time through the user's earphones.

[0752] The emotion engine analyzes the user's pulse and adjusts the tone and intonation to make it easier for the user to understand.

[0753] Example 2:

[0754] In the scenario where a user asks "How do I use a smartphone?", the following happens:

[0755] Pulse data shows that users are confused by the use of the foreign word "smartphone."

[0756] The device captures the audio and sends it to the server.

[0757] The server identifies "smartphone" and translates it into "a mobile phone with many functions."

[0758] The translated information is converted back into audio data and played in real time through the user's earphones.

[0759] The emotion engine detects the user's confusion and provides additional information such as, "If you want to know more about the basic operations of your smartphone, it's best to learn by touching it."

[0760] In this way, the present invention is a system that helps elderly people become familiar with new words and information and realizes smooth communication in their daily lives. Furthermore, the introduction of an emotion engine makes it possible to provide more personalized information according to the user's emotional state, providing a comfortable user experience.

[0761] The processing flow will be explained below.

[0762] MODE FOR CARRYING OUT THE INVENTION

[0763] The present invention is an AI earphone system that translates new words and foreign words into a form that is easy for seniors to understand in real time, and also recognizes the user's emotions and provides information according to their state. A specific embodiment of this system will be described in detail below, with each processing step explained.

[0764] Step 1: Acquire audio

[0765] The user puts on the earphones and engages in normal conversation.

[0766] A microphone in the device captures the user's voice in real time.

[0767] Step 2: Sending audio data

[0768] The terminal packetizes the captured audio data and transmits it to the server.

[0769] The audio data is sent to the server using a secure communication protocol, ensuring data security.

[0770] Step 3: Voice Recognition

[0771] The server analyzes the received voice data and converts it into text data using a voice recognition algorithm.

[0772] For example, the spoken word "influencer" is converted into text data.

[0773] Step 4: Identifying new words

[0774] The server applies natural language processing to identify new and foreign words contained in the text data.

[0775] For example, "influencers" are identified.

[0776] Step 5: Translation process

[0777] The server translates each identified new word or foreign word into words or expressions that are easy for seniors to understand.

[0778] For example, convert "influencer" to "influencer."

[0779] Step 6: Audio translation

[0780] The server reconverts the translated text data into audio data using a speech synthesis algorithm.

[0781] Step 7: Submit your translation data

[0782] The server packetizes the generated voice data and transmits it to the terminal.

[0783] Step 8: Real-time output

[0784] The terminal plays back the received audio data in real time and provides it to the user.

[0785] Step 9: Acquire biometric information

[0786] The pulse sensor installed in the device collects the user's pulse data in real time.

[0787] Step 10: Sending biometric information

[0788] The terminal transmits the acquired pulse data to the server.

[0789] Step 11: Sentiment Analysis

[0790] The server analyzes the received pulse data and estimates the user's emotional state.

[0791] For example, if your pulse rate is high, it is determined that you are nervous.

[0792] Step 12: Adjusting Tone and Intonation

[0793] The server's emotional engine adjusts the tone and intonation of the translated words based on the user's emotional state.

[0794] For example, if the user is nervous, the tone is changed to a gentler tone.

[0795] Step 13: Generate additional information

[0796] The server's emotion engine translates appropriate phrases and additional explanations as needed based on the results of emotion analysis.

[0797] For example, if the user is confused, provide a more detailed explanation.

[0798] Step 14: Vocalize additional information

[0799] The server converts the generated additional text data into voice data using a voice synthesis algorithm.

[0800] Step 15: Submit additional information

[0801] The server transmits the additional information in the form of voice to the terminal.

[0802] Step 16: Playing Additional Audio

[0803] The terminal reproduces the received audio data of the additional information in real time and provides it to the user.

[0804] Through these processing steps, users can deepen their understanding of new words and foreign words and receive personalized information in real time according to their emotional state. This system will facilitate communication among the elderly.

[0805] Example 2

[0806] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0807] In today's information society, new words and foreign words are constantly appearing, and they are often difficult for the elderly to understand. There is also a lack of appropriate support to help them understand this new information. Furthermore, communication can sometimes be hindered by a lack of information provided in accordance with the user's emotional state. It is important to resolve these issues and provide appropriate information that makes it easier for the elderly to understand new words and information, and that responds to their emotional state.

[0808] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0809] In this invention, the server includes a means for converting voice data into text data, a means for identifying new words and foreign words contained in the text data and translating them into an easily understandable form, and a means for converting the translated text data into voice data and providing it to the user. This makes it possible to convert new words and foreign words into a form that is easy for elderly people to understand in real time, and to provide information according to the user's emotional state.

[0810] "Voice data" refers to a digital representation of spoken information uttered by a user.

[0811] "Text data" is voice data converted into a string of characters.

[0812] "New words" are recently introduced terms or phrases that are unfamiliar to a particular user.

[0813] "Foreign words" refer to words introduced from foreign countries that are difficult for certain users to understand.

[0814] "Translation means" refers to techniques or methods that convert identified new words or foreign words into a form that is easy for users to understand.

[0815] "Biometric information" is data obtained from the user's body, and includes pulse rate, heart rate, and the like.

[0816] "Emotional state" refers to the user's state of mind, and includes happiness, sadness, confusion, and the like.

[0817] "Emotion analysis" is the process of assessing a user's emotional state based on acquired biometric information.

[0818] A "voice recognition algorithm" is a technology that converts acquired voice data into text data in the form of a string.

[0819] A "speech synthesis service" is a technology or platform for converting text data into voice data.

[0820] A "server" is a computer system that analyzes voice data and biometric information and processes the data.

[0821] A "terminal" is a device that acquires voice data and biometric information from a user and transmits it to a server.

[0822] "Translated data" refers to text data that has been converted from identified new words or foreign words into an easily understandable form.

[0823] A "secure communications protocol" is a communications procedure used to ensure the security of information when transmitting and receiving data.

[0824] A "pulse sensor" is a device for measuring a user's pulse and acquiring biometric information.

[0825] The "emotion engine" is a system that analyzes acquired biometric information and provides information according to the user's emotional state.

[0826] This system translates new words and foreign words into a form that is easy for the elderly to understand in real time, and provides information according to the user's emotional state. This system consists of a terminal that acquires voice data, a server that analyzes the voice data and biometric information, earphones that provide information to the user, and an emotion engine that recognizes emotions.

[0827] Acquiring and sending audio data

[0828] The user puts on the earphones and engages in normal conversation. The microphone in the device captures the user's voice in real time and sends the voice data to the server. The data is sent using a secure communication protocol (e.g., HTTPS), ensuring data safety.

[0829] Speech data analysis and translation

[0830] The server converts the received voice data into text data using a speech recognition algorithm (e.g., Google Cloud Speech-to-Text API). New words and foreign words are identified from this text data and translated into words and expressions that are easy for seniors to understand. For example, if the word "influencer" is detected, it will be translated as "influential person."

[0831] Real-time audio output

[0832] The translated text data is converted back into audio data on the server side. This process uses a speech synthesis service (e.g., Amazon Polly). The converted audio data is sent to the user's earphones in real time, allowing the user to receive the translated information in a natural voice.

[0833] Biometric data collection and emotion analysis

[0834] A pulse sensor installed in the device captures the user's pulse data in real time. This data is sent to a server, where an emotion analysis algorithm evaluates the user's emotional state. The emotion engine then analyzes this data and adjusts the tone and intonation of the information according to the user's emotional state.

[0835] Providing information according to emotions

[0836] The emotion engine prepares appropriate phrases and additional information based on the user's emotional state. For example, if it senses that the user is confused by a particular word, it will provide additional explanation or convert that word into a more friendly expression. This converted information is also sent to the user's earphones in real time and delivered to the user's ears.

[0837] Specific examples

[0838] Example 1: "What is an influencer?"

[0839] Users encounter the new term "influencer" in everyday conversation.

[0840] The device captures this audio and sends it to the server.

[0841] The server identifies "influencer" and translates it to "influential person."

[0842] The translated information is converted back into audio data and played in real time through the user's earphones.

[0843] The emotion engine analyzes the user's pulse and adjusts the tone and intonation to make it easier for the user to understand.

[0844] Example 2: "How do you use a smartphone?"

[0845] Pulse data shows that users are confused by the use of the foreign word "smartphone."

[0846] The device captures the audio and sends it to the server.

[0847] The server identifies "smartphone" and translates it into "a mobile phone with many functions."

[0848] The translated information is converted back into audio data and played in real time through the user's earphones.

[0849] The emotion engine detects the user's confusion and provides additional information, such as, "If you want to know more about the basic operations of your smartphone, it's best to learn by touching it."

[0850] Prompt Sentence Examples

[0851] Please explain what kind of hardware and software you will use for the specific process of "acquiring and transmitting voice data."

[0852]

[0853] Regarding "providing information based on emotions," please explain what algorithms or methods are used to analyze the user's emotions and provide information.

[0854] This system is designed to help elderly people become familiar with new words and information, enabling smooth communication. The introduction of an emotion engine makes it possible to provide more personalized information according to the user's emotional state, providing a comfortable user experience.

[0855] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0856] Step 1: Acquire audio data

[0857] The device captures the user's voice, and the microphone in the earphone collects ambient sounds and the user's speech in real time.

[0858] Input: User's speech

[0859] Data processing: Apply noise reduction algorithm to convert into clear audio data.

[0860] Output: Digital audio data

[0861] What it does: A microphone in the device captures an audio signal and converts it into a digital format.

[0862] Step 2: Sending audio data

[0863] The device sends the captured audio data to the server, where it encrypts the data using a secure communication protocol (e.g., HTTPS) to ensure the data is secure during transmission.

[0864] Input: Digital audio data

[0865] Data processing: Audio data is divided into packets and encrypted using HTTPS.

[0866] Output: Audio data sent to the server

[0867] Specific operation: The device sends audio data to the server via HTTPS.

[0868] Step 3: Convert audio data to text

[0869] The server converts the received audio data into text using a speech recognition algorithm (e.g., a generative AI model). At this stage, the temporal attributes of the audio are also preserved.

[0870] Input: Audio data sent to the server

[0871] Data calculation: Applying a speech recognition algorithm to generate text data.

[0872] Output: Text data

[0873] Specific operation: The server analyzes the audio data and converts it into text data in string format.

[0874] Step 4: Identifying and translating new or foreign words

[0875] The server identifies new words and foreign words from the converted text data, and these words are translated into words and expressions that are easy for seniors to understand using a translation dictionary in the database.

[0876] Input: Text data

[0877] Data calculation: Identifying new words and foreign words, and matching them with translation dictionaries

[0878] Output: Translated text data

[0879] How it works: The server analyzes the text data and matches the relevant terms with a database to obtain the translation.

[0880] Step 5: Re-speech of text data

[0881] The server converts the translated text data back into voice data using a speech synthesis service (e.g., speech synthesis API). This voice is adjusted to have a natural and friendly tone.

[0882] Input: Translated text data

[0883] Data calculation: Apply a voice synthesis algorithm to generate voice data.

[0884] Output: Audio data

[0885] Specific operation: The server sends text data to the speech synthesis API and obtains the synthesized speech data.

[0886] Step 6: Transmit translation information to earphone

[0887] The server then transmits the re-audio data to the user's earphones in real time, again using a secure communication protocol to ensure the data is safe.

[0888] Input: Revoiced audio data

[0889] Data processing: Audio data is divided into packets and encrypted using HTTPS.

[0890] Output: The audio data that reaches the user

[0891] Specific operation: The server sends audio data to the device using HTTPS, and the device outputs it to the earphones.

[0892] Step 7: Acquire biometric information

[0893] The device uses a built-in pulse sensor to acquire the user's pulse data in real time, and this data is sent to a server.

[0894] Input: User's pulse signal

[0895] Data processing: Convert the signal into digital pulse data.

[0896] Output: Digital pulse data

[0897] How it works: A sensor inside the device captures pulse signals and converts them into digital data.

[0898] Step 8: Sentiment Analysis

[0899] The server analyzes the user's emotional state using the received pulse data, and classifies the user's emotions based on the data using an emotion analysis algorithm.

[0900] Input: Pulse data

[0901] Data Computation: Apply sentiment analysis algorithms to assess emotional states.

[0902] Output: Emotional state data

[0903] Specific operation: The server analyzes the pulse data and evaluates the emotional state using a probabilistic model.

[0904] Step 9: Provide emotionally relevant information

[0905] The emotion engine prepares appropriate phrases and additional information based on the analysis results, and the converted information is also sent to the user's earphones as audio data in real time.

[0906] Input: Emotional state data, text data

[0907] Data calculation: Text data generation, speech synthesis

[0908] Output: Audio data

[0909] Specific operation: The emotion engine generates text data, and the server converts it into audio data using a speech synthesis API and sends it to the earphones.

[0910] These are the processing steps of this system, which makes it easier for elderly people to understand new terms and provides information that is appropriate for their emotional state.

[0911] (Application example 2)

[0912] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0913] Elderly people often have difficulty understanding the content of explanations of new products or services in brick-and-mortar stores. New words and foreign words are particularly difficult for elderly people to understand. Furthermore, depending on the elderly person's emotional state, information may not be provided at the appropriate time or in the appropriate tone, making it difficult for them to make appropriate decisions. This invention aims to make it easier for such elderly people to understand product explanations in brick-and-mortar stores and to provide information that adapts to the elderly person's emotional state.

[0914] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0915] In this invention, the server includes means for acquiring voice data, means for converting the acquired voice data into text data, means for identifying new words and foreign words contained in the text data and translating them into an easy-to-understand form, means for converting the translated text data into voice data and providing it to the user, means for acquiring biometric information of the user, means for analyzing the emotional state of the user based on the acquired biometric information and converting the text data into appropriate wording based on the analysis results, and means for translating and recognizing emotions when the user is receiving an explanation of new products or services in a physical store and providing the user with information in real time. This makes it possible to provide elderly people with information that is easy to understand and appropriate to their emotional state when receiving an explanation of new products or services in a physical store.

[0916] "Speech data" refers to data that records sound information collected from the user's speech and the surrounding sound environment.

[0917] "Text data" is character information obtained by processing voice data for recognition.

[0918] "New words and foreign words" are words that are difficult for users to understand or that they do not normally use, or words that have been imported from other languages.

[0919] "Translation" is the process of converting identified text data into a form that is easy for elderly people to understand.

[0920] "Speech recognition means" refers to a device or software for converting voice data into text data.

[0921] "Biometric information" refers to information including the user's pulse rate and other physiological data.

[0922] "Emotional state" refers to a user's current psychological state or feelings.

[0923] The "emotion recognition means" is a device or software for analyzing the user's emotional state from collected biometric information.

[0924] "Real-time" means acting immediately, without delay.

[0925] A "brick and mortar store" is a place that offers goods or services in a physical location.

[0926] A "user" is a person who uses this system.

[0927] This invention is a system that helps elderly people understand the content of new products and services in real time when they are explained in a physical store. The system collects and translates voice data, collects biometric information from users, and analyzes their emotions, and provides appropriate information based on this information.

[0928] composition:

[0929] The system consists of the following elements:

[0930] 1. Device that receives audio data:

[0931] A microphone in the device captures the user's voice in real time, for example, smart glasses or a smartphone.

[0932] 2. Server that analyzes voice data and biometric information:

[0933] The server converts the received voice data into text data using a speech recognition algorithm (e.g., Google Speech-to-Text API), while simultaneously analyzing biometric information.

[0934] 3. Earphones that provide information to the user:

[0935] Earphones to convert the translated text data back into audio data and provide it to the user in real time. The built-in speakers of smart glasses can also be used.

[0936] 4. Emotion engine for emotion recognition:

[0937] The device's built-in pulse sensor captures the user's pulse data in real time and sends it to a server. Based on this data, an emotion analysis algorithm evaluates the user's emotional state.

[0938] Operation:

[0939] 1. Audio data capture and transmission:

[0940] When a user asks a question about a product or receives an explanation in a store, the device's microphone captures the voice and transmits it to the server. The voice data is transmitted using a secure communication protocol.

[0941] 2. Speech data analysis and translation:

[0942] The server uses a speech recognition algorithm to convert the received voice data into text data, and identifies new or foreign words from this text data. For example, if the phrase "detox water" is included, it translates it into an easy-to-understand form (e.g., "water that flushes toxins from the body").

[0943] 3. Real-time audio output:

[0944] The translated text data is converted back into audio data on the server side and sent to the user's earphones in real time.

[0945] 4. Biometric data collection and emotion analysis:

[0946] The user's pulse sensor captures pulse data in real time and sends it to a server, where an emotion recognition algorithm analyzes it and provides appropriate information according to the user's emotional state.

[0947] Examples:

[0948] Example 1:

[0949] When seniors ask about "detox water" and new products.

[0950] Salesperson: "This is the latest detox water. It has detoxifying effects."

[0951] Seniors: "What's a detox?"

[0952] Example prompt sentence:

[0953] "What is detox water?"

[0954] "Tell me more about this new product."

[0955] Please explain the ingredients

[0956] In this situation, the system captures the elderly person's question, translates the new term "detox" into "something that has the function of expelling toxins from the body," and provides this information in audio. If the system senses that the elderly person is confused, it provides additional information such as, "We can also explain in detail the effects of detox water here."

[0957] Recommended hardware and software:

[0958] Speech Recognition: Google Speech-to-Text API

[0959] Translation: Dedicated translation server (e.g. Google Cloud Translation API)

[0960] Emotion Recognition: Dedicated emotion analysis algorithms (e.g., SomeEmotionRecognitionLibrary)

[0961] Audio output: pyttsx3 library (Python)

[0962] This means that when elderly people are explained about new products or services in a physical store, they can receive information in an easy-to-understand format immediately, helping them make appropriate decisions.

[0963] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0964] Step 1:

[0965] The terminal captures the user's voice data.

[0966] Input: User's voice

[0967] Output: Audio data (analog)

[0968] Processing: A microphone on the device picks up the user's voice in real time.

[0969] Specific operation: The words spoken by the user are captured by a microphone and stored as audio data on the device.

[0970] Step 2:

[0971] The device transmits the captured audio data to the server.

[0972] Input: Audio data (analog)

[0973] Output: Audio data (digital)

[0974] Processing: The audio data is converted into a digital format and sent to a server using a secure communication protocol.

[0975] Specific operation: The voice data in the device is converted into a digital format and sent to a server via the Internet.

[0976] Step 3:

[0977] The server converts the voice data into text data.

[0978] Input: Audio data (digital)

[0979] Output: Text data

[0980] Processing: Convert the audio data into text using a speech recognition algorithm (e.g., Google Speech-to-Text API).

[0981] Specific operation: The server analyzes the voice data and converts the spoken content into text information.

[0982] Step 4:

[0983] The server identifies new words and foreign words and translates them into a form that is easy for seniors to understand.

[0984] Input: Text data

[0985] Output: Translated text data

[0986] Processing: Identified new or foreign words are translated using a translation algorithm (e.g., Google Cloud Translation API).

[0987] Specific action: Translate the word "detox" as "the function of expelling toxins from the body."

[0988] Step 5:

[0989] The server converts the translated text data back into audio data.

[0990] Input: Translated text data

[0991] Output: Regenerated audio data

[0992] Processing: Convert the text data into audio data using a text-to-speech (TTS) algorithm (e.g., the pyttsx3 library).

[0993] Specific behavior: Regenerate text information written in different languages ​​as natural-sounding speech.

[0994] Step 6:

[0995] The terminal transmits the regenerated audio data to the earphone.

[0996] Input: Regenerated audio data

[0997] Output: Audio data delivered to earphones

[0998] Processing: The regenerated audio data is sent from the device to the earphones via a secure communication protocol.

[0999] Specific operation: Audio data sent from the server is delivered to the earphones in real time, and the user listens to the audio.

[1000] Step 7:

[1001] The device acquires the user's biometric information (such as pulse rate) and sends it to the server.

[1002] Input: Biometric information (pulse data, etc.)

[1003] Output: Biometric information (digital data)

[1004] Processing: Biometric data is acquired from the sensor device and sent to the server.

[1005] Specific operation: Acquires biometric data such as pulse rate and electrodermal activity (EDA) and sends it to a server via the Internet.

[1006] Step 8:

[1007] The server analyzes the emotional state based on biometric information and adds appropriate phrases.

[1008] Input: Biometric information (digital data)

[1009] Output: Sentiment analysis results and properly adjusted translation data

[1010] Processing: Emotion recognition algorithms analyze biometric data to assess the user's emotional state, and generate appropriate tones and additional supportive information based on the results.

[1011] What it does: If the user is confused, add a message to the translated information saying "Please let us know if you need further clarification."

[1012] In this way, the specific flow of implementing the invention based on the claims has been explained by dividing it into processing steps.

[1013] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1014] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1015] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1016] [Third embodiment]

[1017] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1018] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1019] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1020] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1021] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1022] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1023] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1024] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1025] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1026] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1027] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1028] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1029] MODE FOR CARRYING OUT THE INVENTION

[1030] The present invention is an AI earphone system that translates new words and foreign words used in everyday life into a form that is easy for elderly people to understand in real time. Specific embodiments of this system are described below.

[1031] System configuration and operation

[1032] This system consists of three main elements: a terminal for acquiring voice data, a server for analyzing the acquired voice data and biometric information, and earphones for providing information to the user.

[1033] Acquiring and sending audio data

[1034] The user puts on the earphones and engages in normal conversation. The microphone in the device captures the user's voice in real time and transmits the voice data to the server. The voice data is transmitted to the server using a secure communication protocol, ensuring data security.

[1035] Speech data analysis and translation

[1036] The server then runs the received voice data through a speech recognition algorithm and converts it into text data. It then identifies new words and foreign words from this text data and translates them into words and expressions that are easy for seniors to understand. For example, if the word "influencer" is detected, it will be translated as "influential person."

[1037] Real-time audio output

[1038] The translated text data is converted back into audio data on the server side and sent to the user's earphones in real time, allowing the user to receive the translated information in a natural voice.

[1039] Collection and analysis of biological information

[1040] The device is equipped with a biometric information acquisition device such as a pulse sensor that acquires the user's biometric information in real time. This data is sent to a server, where an emotion analysis algorithm evaluates the user's emotional state. For example, a high pulse rate may indicate tension.

[1041] Providing information according to emotions

[1042] Based on the analysis results, the server prepares appropriate phrases and additional information. For example, if it detects that the user is confused by a particular word, it will provide additional explanation for that word or convert it into a more familiar expression. This converted information is also sent to the user's earphones in real time and delivered to the user's ears.

[1043] Specific examples

[1044] Example 1:

[1045] In the scenario where a user asks "What is an influencer?", the following happens:

[1046] Users encounter the new term "influencer" in everyday conversations.

[1047] The device captures this audio and sends it to the server.

[1048] The server identifies "influencer" and translates it to "influential person."

[1049] The translated information is converted back into audio data and played in real time through the user's earphones.

[1050] Example 2:

[1051] In the scenario where a user asks "How do I use a smartphone?", the following happens:

[1052] Pulse data shows that users are confused by the use of the foreign word "smartphone."

[1053] The device captures the audio and sends it to the server.

[1054] The server identifies "smartphone" and translates it into "a mobile phone with many functions."

[1055] The translated information is converted back into audio data and played in real time through the user's earphones.

[1056] Additionally, the server detects the user's confusion and provides additional information such as, "If you want to know more about the basic operations of your smartphone, it's best to learn by touching it directly."

[1057] In this way, the present invention is a system that helps elderly people become familiar with new words and information and realizes smooth communication in their daily lives.

[1058] The processing flow will be explained below.

[1059] Specific processing steps of the program

[1060] Step 1: Acquire audio

[1061] The user puts on the earphones and engages in normal conversation.

[1062] A microphone built into the device captures the user's voice in real time.

[1063] Step 2: Sending audio data

[1064] The device then packetizes the captured audio data and sends it to the server using a secure protocol.

[1065] Step 3: Voice Recognition

[1066] The server analyzes the received voice data and converts it into text data using a voice recognition algorithm.

[1067] Step 4: Identifying new words

[1068] The server applies natural language processing to identify new and foreign words contained in the text data.

[1069] Step 5: Translation process

[1070] The server translates each identified new or foreign word into a word or phrase that is easy for seniors to understand, for example, converting "influencer" into "influential person."

[1071] Step 6: Audio translation

[1072] The server reconverts the translated text data into audio data using a speech synthesis algorithm.

[1073] Step 7: Submit your translation data

[1074] The server packetizes the generated voice data and transmits it to the terminal.

[1075] Step 8: Real-time output

[1076] The terminal plays back the received audio data in real time and provides it to the user.

[1077] Step 9: Acquire biometric information

[1078] A pulse sensor installed in the device acquires the user's pulse data.

[1079] Step 10: Sending biometric information

[1080] The terminal transmits the acquired pulse data to the server.

[1081] Step 11: Sentiment Analysis

[1082] The server analyzes the received pulse data and estimates the user's emotional state. For example, if the pulse rate is high, it is determined that the user is nervous.

[1083] Step 12: Generate additional information

[1084] The server then translates the message based on the results of the sentiment analysis, providing appropriate phrasing and additional explanations as needed. For example, if the user is confused, a more detailed explanation is added.

[1085] Step 13: Vocalize additional information

[1086] The server converts the generated additional text data into voice data using a voice synthesis algorithm.

[1087] Step 14: Submit additional information

[1088] The server transmits the additional information in the form of voice to the terminal.

[1089] Step 15: Play additional audio

[1090] The terminal reproduces the received audio data of the additional information in real time and provides it to the user.

[1091] These are the specific processing steps of "Generation Communicator." Through this processing flow, elderly people can easily understand new words and foreign words, realizing comfortable communication.

[1092] Example 1

[1093] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1094] In the past, it was often difficult for elderly people to understand new words and foreign words used in daily life. Furthermore, there was no system that could analyze and translate speech data in real time and provide appropriate information according to the user's emotional state. This made it difficult for elderly people to communicate smoothly when they encountered new words or concepts. The purpose of this invention is to solve this problem.

[1095] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1096] In this invention, the server includes a means for converting voice data into text data, a means for translating new words and foreign words contained in the text data into a form that is easy for seniors to understand, and a means for reconverting the translated text data into voice data. This enables seniors to understand new words and foreign words used in everyday conversation in real time, enabling smooth communication. Furthermore, by analyzing the user's emotional state based on biometric information and providing appropriate phrases and additional information based on the analysis results, a system that is more user-friendly and easy to use can be provided. This allows seniors to communicate smoothly without being confused by new words and information.

[1097] "Audio data" refers to data in the form of sound waves, such as user conversations or environmental sounds.

[1098] A "terminal" is a device for acquiring voice data and biometric information and transmitting that data to a server.

[1099] A "server" is a computer system that analyzes, transforms, and presents information to users.

[1100] "Text data" refers to data obtained by converting voice data into character information.

[1101] "New words and foreign words" are words that have been introduced into the user's native language in a relatively short period of time or words that have been adopted from a foreign country.

[1102] "Translation" is the act of converting something from one language into another language or into an understandable form.

[1103] "Biometric information" refers to physical data such as the user's pulse rate and body temperature.

[1104] "Emotional state" refers to a psychological state estimated based on the user's pulse rate and other biometric information.

[1105] An "emotion analysis algorithm" is a computational method for analyzing biometric information and assessing a user's emotional state.

[1106] "Additional information" refers to supplemental information or explanations provided depending on the user's emotional state.

[1107] A "secure communication protocol" is a communication protocol for safely sending and receiving data.

[1108] A "generative AI model" is an artificial intelligence model that learns patterns from large amounts of data and generates new information.

[1109] The present invention is a system that acquires, analyzes, and translates voice data in real time to provide users with easy-to-understand information about new words and foreign words. A specific embodiment of the present invention will be described. The system comprises a terminal for acquiring voice data, a server for analyzing the acquired voice data and biometric information, and earphones for providing translated information to the user.

[1110] Acquiring audio data

[1111] The device captures the user's voice in real time using a built-in microphone when they are having a normal conversation. The voice data is then temporarily stored on the device. For example, if a user says, "Young people these days are only chasing influencers," the voice data will be captured by the device.

[1112] Sending audio data

[1113] The terminal transmits the captured audio data to the server using a secure communication protocol (for example, HTTPS or TLS).

[1114] Analysis of audio data

[1115] The server converts the received voice data into text data using a speech recognition API such as Google Cloud Speech-to-Text or Amazon Transcribe. For example, the generated text data would be, "Young people these days are only chasing influencers."

[1116] Identifying and translating new and foreign words

[1117] The server analyzes the generated text data and identifies new words and foreign words (e.g., "influencer") using natural language processing (NLP) technology. It then translates these new words and foreign words into a form that is easy for seniors to understand (e.g., "influencer" becomes "influential person") using a pre-trained generative AI model or machine translation engine.

[1118] Converting translation text to audio data

[1119] The server reconverts the translated text data into audio data using a speech synthesis API such as Google Cloud Text-to-Speech or Amazon Polly.

[1120] Audio output to the user

[1121] The translated voice data is then sent to the device again using a secure communication protocol and finally output to the user's earphones. The user can listen to the translated voice in real time. For example, the translated voice may provide the user with information such as, "Young people these days only chase after influential people."

[1122] Collection and analysis of biological information

[1123] The device is equipped with a biometric information acquisition device such as a pulse sensor, which acquires the user's biometric information (e.g., pulse) in real time and temporarily stores the data. This biometric information is also transmitted to a server using a secure communication protocol, and the server evaluates the user's emotional state (e.g., tension) using an emotion analysis algorithm.

[1124] Providing information according to emotions

[1125] The server generates appropriate phrases and additional information based on the analysis results. For example, if it determines that the user is confused by the foreign word "smartphone," it generates advice such as, "If you want to learn more about the basic operations of a smartphone, it's a good idea to learn by touching it directly," and then converts this into audio data using Google Cloud Text-to-Speech and provides it to the user.

[1126] In this way, this system helps elderly people become familiar with new words and information, enabling smooth communication in their daily lives.

[1127] Examples of prompt statements

[1128] The following are examples of prompt sentences:

[1129] Capture conversations that include new foreign words or technical terms, translate them in real time to make them understandable for seniors, and provide appropriate phrasing or additional information based on the user's emotional state.

[1130] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1131] The flow of this system's program processing

[1132] Step 1: Acquire audio data

[1133] The device uses a built-in microphone to capture the user's voice in real time. The input is the user's voice, and the output is the captured voice data. This voice data is immediately and temporarily stored on the device. For example, when a user says, "What is an influencer?", the voice is captured.

[1134] Step 2: Sending audio data

[1135] The device sends the captured audio data to the server using a secure communication protocol (HTTPS or TLS). The input is the captured audio data, and the output is the audio data sent to the server. The transfer of the audio data makes it possible to proceed to the next analysis step.

[1136] Step 3: Analyzing the audio data

[1137] The server converts the received voice data into text data by running it through a speech recognition API such as Google Cloud Speech-to-Text or Amazon Transcribe. The input is the transmitted voice data, and the output is the generated text data. For example, the voice data "What is an influencer?" is converted into the text data "What is an influencer?"

[1138] Step 4: Identifying new words and loan words

[1139] The server analyzes the generated text data using natural language processing (NLP) techniques to identify new words and loan words. The input is the text data, and the output is a list of identified new words and loan words. For example, the word "influencer" is identified.

[1140] Step 5: Translation process

[1141] The server translates the identified new words and foreign words into a form that is easy for seniors to understand. It uses a generative AI model and a machine translation engine. The input is the identified new words and foreign words, and the output is the translated sentence. For example, the word "influencer" is translated into "influential person."

[1142] Step 6: Converting translation text to audio data

[1143] The server reconverts the translated text data into audio data using Google Cloud Text-to-Speech or Amazon Polly. The input is the translated text data, and the output is the reconverted audio data. For example, the text "influential person" is converted into audio data.

[1144] Step 7: Audio output to the user

[1145] The server sends the re-encoded voice data to the device using a secure communication protocol, and the device transmits the voice data to the user through earphones. The input is the re-encoded voice data, and the output is the voice the user hears in real time. The user hears the voice answer "Influential person" in response to the question "What is an influencer?"

[1146] Step 8: Collect biometric data

[1147] The device acquires the user's biometric information (e.g., pulse) in real time using a pulse sensor or the like. The input is a biometric signal such as the user's pulse, and the output is the acquired biometric data. For example, data on the user's pulse rate is collected.

[1148] Step 9: Sending biometric information

[1149] The device transmits the acquired biometric information to the server via a secure communication protocol. The input is the acquired biometric data, and the output is the biometric data transmitted to the server. The transfer of the biometric data enables the next analysis step.

[1150] Step 10: Analysis of biological information

[1151] The server then applies the received biometric data to an emotion analysis algorithm to evaluate the user's emotional state. The input is the transmitted biometric data, and the output is the analyzed emotional state. For example, a high pulse rate is evaluated as a state of tension.

[1152] Step 11: Provide emotionally relevant information

[1153] The server generates appropriate phrases and additional information based on the results of the emotion analysis, converts them into voice data, and provides them to the user. A generative AI model is used. The input is the analyzed emotional state and a prompt to generate appropriate phrases, and the output is the appropriate phrases and additional information converted into voice data. If the user is confused about a smartphone, advice such as "If you want to know more about the basic operations of a smartphone, it's best to learn by touching it directly" is provided as voice data.

[1154] (Application example 1)

[1155] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1156] The problem is that elderly people are confused by new words and foreign words, especially security-related terms, and have difficulty understanding them. Furthermore, when it comes to emergency responses, accurate and quick understanding is required, but elderly people are prone to feeling stressed and confused at times. In such situations, appropriate responses may not be possible, which could increase security risks.

[1157] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1158] In this invention, the server includes means for acquiring voice data, means for converting the acquired voice data into text data, means for identifying new words and foreign words contained in the text data and translating them into an easy-to-understand form, means for converting the translated text data into voice data and providing it to the user, means for acquiring the user's biometric information, means for analyzing the user's emotional state based on the acquired biometric information and converting the translated text data into appropriate phrases based on the analysis results, means for translating security-related terms and emergency response instructions into easy-to-understand forms and providing them to the user, and means for providing additional information to reduce the user's stress if the user's emotional state exceeds a reference value. This makes it easier for elderly people to understand security-related terms and emergency response instructions, enabling them to take necessary measures quickly and accurately.

[1159] The "means for acquiring voice data" refers to a device for capturing the user's voice in real time and acquiring the data.

[1160] "Means for converting acquired voice data into text data" refers to a device or software that uses voice recognition technology to convert voice data into text information.

[1161] "Means for identifying new words and foreign words contained in the text data and translating them into an easily understandable form" refers to a device or software that detects new or difficult terms from the converted text data and replaces them with words that are easy for seniors to understand.

[1162] "Means for converting translated text data into audio data and providing it to the user" refers to a device or software that plays back the translated text data as audio.

[1163] "Means for acquiring user's biometric information" refers to a device that acquires biometric data such as the user's pulse and skin potential in real time.

[1164] "Means for analyzing the user's emotional state based on the acquired biometric information and converting it into appropriate phrases based on the analysis results" refers to a device or software that analyzes biometric information to evaluate the user's emotional state and translates it into more appropriate expressions based on the results.

[1165] "Means for translating security-related terms and emergency response instructions into easy-to-understand language and providing them to users" refers to a device or software that translates difficult security-related terms and instructions into simple language and provides them to users as audio.

[1166] "Means for providing additional information to reduce stress in a user when the emotional state exceeds a reference value" refers to a device or software that provides supplemental information to reduce stress in a user when the analyzed emotional state exceeds a reference value.

[1167] System configuration and operation

[1168] The system consists of the following elements:

[1169] 1. Terminal: A device that acquires voice data and collects the user's biometric information.

[1170] 2. Server: A device that analyzes and translates voice data and generates additional information based on the translation results.

[1171] 3. Earphones: A device that provides translated text data as audio.

[1172] Acquiring and sending audio data

[1173] The user puts on the earphones and starts a normal conversation. The microphone in the device captures the user's voice in real time and transmits the voice data to the server. A secure communication protocol is used for this transmission, ensuring the safety of the data.

[1174] Speech data analysis and translation

[1175] The server then runs the received voice data through a speech recognition algorithm and converts it into text. It then identifies new or foreign words from this text and translates them into language that is easy for seniors to understand. For example, if the term "phishing attack" is detected, it will be translated into "a method of deceiving personal information."

[1176] Real-time audio output

[1177] The translated text data is converted back into audio data on the server side and sent to the user's earphones in real time, allowing the user to receive the translated information in a natural voice.

[1178] Collection and analysis of biological information

[1179] The device is equipped with a biometric information acquisition device, such as a pulse sensor, that acquires the user's biometric information in real time. This data is sent to a server, where an emotion analysis algorithm evaluates the user's emotional state. For example, a high pulse rate may indicate tension.

[1180] Providing information according to emotions

[1181] Based on the analysis results, the server prepares appropriate phrases and additional information. For example, if it detects that the user is confused by a particular word, it will provide additional explanations or convert the word into a more user-friendly expression. This converted information is also sent to the user's earphones in real time and delivered to the user's ears.

[1182] Specific examples

[1183] Example 1: In this scenario, if a user asks "What is a phishing attack?", the following will happen:

[1184] Users encounter a new term in security conversations: "phishing attack."

[1185] The device captures this audio and sends it to the server.

[1186] The server identifies "phishing attacks" and translates them into "tricks to trick people into revealing personal information."

[1187] The translated information is converted back into audio data and played in real time through the user's earphones.

[1188] At the same time, if the user's biometric information indicates a state of tension, additional information and advice is provided, such as "When it comes to phishing attacks, it's best to ignore the links and emails."

[1189] Example prompts to input to a generative AI model:

[1190] "How are phishing attacks used online?"

[1191] Hardware and software used

[1192] Hardware:

[1193] Devices (smartphones, smart glasses, head-mounted displays)

[1194] Biometric information acquisition device (pulse sensor)

[1195] software:

[1196] Speech Recognition Library: speech_recognition

[1197] Biosensor library: some_bio_sensor_library

[1198] Text-to-speech (TTS) library: pyttsx3

[1199] HTTP request library:requests

[1200] Server-side processing: NLP models, sentiment analysis algorithms, speech conversion engines

[1201] In this way, the present invention is a system that helps seniors become familiar with new words and information and improves their understanding and ability to respond to security-related issues.

[1202] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1203] Step 1:

[1204] The device acquires the user's voice data. Specifically, the device's microphone is used to capture the user's voice in real time. The acquired voice data is saved as raw voice waveform data.

[1205] Step 2:

[1206] The terminal transmits the acquired voice data to the server. Specifically, the terminal transfers the voice data to the server using a secure communication protocol. This communication ensures the security of the data.

[1207] Step 3:

[1208] The server runs the received voice data through a speech recognition algorithm and converts it into text data. Specifically, it uses a speech recognition library (e.g., speech_recognition) to convert the voice waveform data into corresponding text data. This conversion process uses an acoustic model and a language model.

[1209] Step 4:

[1210] The server identifies new words and foreign words contained in the converted text data. Specifically, it uses a generative AI model to analyze the text data and detect new words and foreign words. NLP technology is applied in this analysis.

[1211] Step 5:

[1212] The server translates the identified new words and foreign words into a form that is easy for seniors to understand. Specifically, the generative AI model refers to a translation dictionary and translates, for example, "phishing attack" into "a method of defrauding personal information." The translation results are saved as text data.

[1213] Step 6:

[1214] The server converts the translated text back into audio using a text-to-speech (TTS) library (e.g., pyttsx3). This audio is then given natural pronunciation and intonation.

[1215] Step 7:

[1216] The server transmits the re-converted audio data to the device in real time using a secure communication protocol. The device receives the audio data and immediately provides it to the user through earphones.

[1217] Step 8:

[1218] The device acquires biometric information. Specifically, it uses a pulse sensor built into the device to acquire biometric data such as the user's pulse and skin potential in real time. This data is saved as numerical data.

[1219] Step 9:

[1220] The terminal transmits the acquired biometric information to the server. Specifically, the biometric data is transferred to the server using a secure communication protocol. This communication ensures the security of the data.

[1221] Step 10:

[1222] The server analyzes the user's emotional state based on the acquired biometric information. Specifically, it uses a biometric analysis algorithm to evaluate, for example, whether the user is nervous. The evaluation results are saved as numerical data.

[1223] Step 11:

[1224] The server prepares appropriate phrases and additional information based on the analysis results. Specifically, the generative AI model uses the sentiment analysis results to generate additional advice, such as "Please ignore phishing attacks." This additional information is saved as text data.

[1225] Step 12:

[1226] The server converts the prepared additional information into audio data and sends it to the device. Specifically, it uses a text-to-speech (TTS) library to convert the text data of the additional information into audio data and transmits it to the device using a secure communication protocol. The device receives this audio data and provides it to the user through earphones.

[1227] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1228] MODE FOR CARRYING OUT THE INVENTION

[1229] The present invention is an AI earphone system that translates new words and foreign words in real time into a form that is easy for elderly people to understand, and also recognizes the user's emotions and provides information according to their state. Specific embodiments of this system are described below.

[1230] System configuration and operation

[1231] This system consists of a terminal that acquires voice data, a server that analyzes the acquired voice data and biometric information, earphones that provide information to the user, and an emotion engine that performs emotion recognition.

[1232] Acquiring and sending audio data

[1233] The user puts on the earphones and engages in normal conversation. The microphone in the device captures the user's voice in real time and transmits the voice data to the server. The voice data is transmitted to the server using a secure communication protocol, ensuring data security.

[1234] Speech data analysis and translation

[1235] The server converts the received voice data into text data using a speech recognition algorithm. New words and foreign words are identified from this text data and translated into words and expressions that are easy for seniors to understand. For example, if the word "influencer" is detected, it will be translated into "influential person."

[1236] Real-time audio output

[1237] The translated text data is converted back into audio data on the server side and sent to the user's earphones in real time, allowing the user to receive the translated information in a natural voice.

[1238] Biometric data collection and emotion analysis

[1239] A pulse sensor installed in the device captures the user's pulse data in real time. This data is sent to a server, where an emotion analysis algorithm evaluates the user's emotional state. The emotion engine then analyzes this data and adjusts the tone and intonation of the information according to the user's emotional state.

[1240] Providing information according to emotions

[1241] The emotion engine prepares appropriate phrases and additional information based on the user's emotional state. For example, if it senses that the user is confused by a particular word, it will provide additional explanation or convert that word into a more friendly expression. This converted information is also sent to the user's earphones in real time and delivered to the user's ears.

[1242] Specific examples

[1243] Example 1:

[1244] In the scenario where a user asks "What is an influencer?", the following happens:

[1245] Users encounter the new term "influencer" in everyday conversations.

[1246] The device captures this audio and sends it to the server.

[1247] The server identifies "influencer" and translates it to "influential person."

[1248] The translated information is converted back into audio data and played in real time through the user's earphones.

[1249] The emotion engine analyzes the user's pulse and adjusts the tone and intonation to make it easier for the user to understand.

[1250] Example 2:

[1251] In the scenario where a user asks "How do I use a smartphone?", the following happens:

[1252] Pulse data shows that users are confused by the use of the foreign word "smartphone."

[1253] The device captures the audio and sends it to the server.

[1254] The server identifies "smartphone" and translates it into "a mobile phone with many functions."

[1255] The translated information is converted back into audio data and played in real time through the user's earphones.

[1256] The emotion engine detects the user's confusion and provides additional information such as, "If you want to know more about the basic operations of your smartphone, it's best to learn by touching it."

[1257] In this way, the present invention is a system that helps elderly people become familiar with new words and information and realizes smooth communication in their daily lives. Furthermore, the introduction of an emotion engine makes it possible to provide more personalized information according to the user's emotional state, providing a comfortable user experience.

[1258] The processing flow will be explained below.

[1259] MODE FOR CARRYING OUT THE INVENTION

[1260] The present invention is an AI earphone system that translates new words and foreign words into a form that is easy for seniors to understand in real time, and also recognizes the user's emotions and provides information according to their state. A specific embodiment of this system will be described in detail below, with each processing step explained.

[1261] Step 1: Acquire audio

[1262] The user puts on the earphones and engages in normal conversation.

[1263] A microphone in the device captures the user's voice in real time.

[1264] Step 2: Sending audio data

[1265] The terminal packetizes the captured audio data and transmits it to the server.

[1266] The audio data is sent to the server using a secure communication protocol, ensuring data security.

[1267] Step 3: Voice Recognition

[1268] The server analyzes the received voice data and converts it into text data using a voice recognition algorithm.

[1269] For example, the spoken word "influencer" is converted into text data.

[1270] Step 4: Identifying new words

[1271] The server applies natural language processing to identify new and foreign words contained in the text data.

[1272] For example, "influencers" are identified.

[1273] Step 5: Translation process

[1274] The server translates each identified new word or foreign word into words or expressions that are easy for seniors to understand.

[1275] For example, convert "influencer" to "influencer."

[1276] Step 6: Audio translation

[1277] The server reconverts the translated text data into audio data using a speech synthesis algorithm.

[1278] Step 7: Submit your translation data

[1279] The server packetizes the generated voice data and transmits it to the terminal.

[1280] Step 8: Real-time output

[1281] The terminal plays back the received audio data in real time and provides it to the user.

[1282] Step 9: Acquire biometric information

[1283] The pulse sensor installed in the device collects the user's pulse data in real time.

[1284] Step 10: Sending biometric information

[1285] The terminal transmits the acquired pulse data to the server.

[1286] Step 11: Sentiment Analysis

[1287] The server analyzes the received pulse data and estimates the user's emotional state.

[1288] For example, if your pulse rate is high, it is determined that you are nervous.

[1289] Step 12: Adjusting Tone and Intonation

[1290] The server's emotional engine adjusts the tone and intonation of the translated words based on the user's emotional state.

[1291] For example, if the user is nervous, the tone is changed to a gentler tone.

[1292] Step 13: Generate additional information

[1293] The server's emotion engine translates appropriate phrases and additional explanations as needed based on the results of emotion analysis.

[1294] For example, if the user is confused, provide a more detailed explanation.

[1295] Step 14: Vocalize additional information

[1296] The server converts the generated additional text data into voice data using a voice synthesis algorithm.

[1297] Step 15: Submit additional information

[1298] The server transmits the additional information in the form of voice to the terminal.

[1299] Step 16: Playing Additional Audio

[1300] The terminal reproduces the received audio data of the additional information in real time and provides it to the user.

[1301] Through these processing steps, users can deepen their understanding of new words and foreign words and receive personalized information in real time according to their emotional state. This system will facilitate communication among the elderly.

[1302] Example 2

[1303] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1304] In today's information society, new words and foreign words are constantly appearing, and they are often difficult for the elderly to understand. There is also a lack of appropriate support to help them understand this new information. Furthermore, communication can sometimes be hindered by a lack of information provided in accordance with the user's emotional state. It is important to resolve these issues and provide appropriate information that makes it easier for the elderly to understand new words and information, and that responds to their emotional state.

[1305] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1306] In this invention, the server includes a means for converting voice data into text data, a means for identifying new words and foreign words contained in the text data and translating them into an easily understandable form, and a means for converting the translated text data into voice data and providing it to the user. This makes it possible to convert new words and foreign words into a form that is easy for elderly people to understand in real time, and to provide information according to the user's emotional state.

[1307] "Voice data" refers to a digital representation of spoken information uttered by a user.

[1308] "Text data" is voice data converted into a string of characters.

[1309] "New words" are recently introduced terms or phrases that are unfamiliar to a particular user.

[1310] "Foreign words" refer to words introduced from foreign countries that are difficult for certain users to understand.

[1311] "Translation means" refers to techniques or methods that convert identified new words or foreign words into a form that is easy for users to understand.

[1312] "Biometric information" is data obtained from the user's body, and includes pulse rate, heart rate, and the like.

[1313] "Emotional state" refers to the user's state of mind, and includes happiness, sadness, confusion, and the like.

[1314] "Emotion analysis" is the process of assessing a user's emotional state based on acquired biometric information.

[1315] A "voice recognition algorithm" is a technology that converts acquired voice data into text data in the form of a string.

[1316] A "speech synthesis service" is a technology or platform for converting text data into voice data.

[1317] A "server" is a computer system that analyzes voice data and biometric information and processes the data.

[1318] A "terminal" is a device that acquires voice data and biometric information from a user and transmits it to a server.

[1319] "Translated data" refers to text data that has been converted from identified new words or foreign words into an easily understandable form.

[1320] A "secure communications protocol" is a communications procedure used to ensure the security of information when transmitting and receiving data.

[1321] A "pulse sensor" is a device for measuring a user's pulse and acquiring biometric information.

[1322] The "emotion engine" is a system that analyzes acquired biometric information and provides information according to the user's emotional state.

[1323] This system translates new words and foreign words into a form that is easy for the elderly to understand in real time, and provides information according to the user's emotional state. This system consists of a terminal that acquires voice data, a server that analyzes the voice data and biometric information, earphones that provide information to the user, and an emotion engine that recognizes emotions.

[1324] Acquiring and sending audio data

[1325] The user puts on the earphones and engages in normal conversation. The microphone in the device captures the user's voice in real time and sends the voice data to the server. The data is sent using a secure communication protocol (e.g., HTTPS), ensuring data safety.

[1326] Speech data analysis and translation

[1327] The server converts the received voice data into text data using a speech recognition algorithm (e.g., Google Cloud Speech-to-Text API). New words and foreign words are identified from this text data and translated into words and expressions that are easy for seniors to understand. For example, if the word "influencer" is detected, it will be translated as "influential person."

[1328] Real-time audio output

[1329] The translated text data is converted back into audio data on the server side. This process uses a speech synthesis service (e.g., Amazon Polly). The converted audio data is sent to the user's earphones in real time, allowing the user to receive the translated information in a natural voice.

[1330] Biometric data collection and emotion analysis

[1331] A pulse sensor installed in the device captures the user's pulse data in real time. This data is sent to a server, where an emotion analysis algorithm evaluates the user's emotional state. The emotion engine then analyzes this data and adjusts the tone and intonation of the information according to the user's emotional state.

[1332] Providing information according to emotions

[1333] The emotion engine prepares appropriate phrases and additional information based on the user's emotional state. For example, if it senses that the user is confused by a particular word, it will provide additional explanation or convert that word into a more friendly expression. This converted information is also sent to the user's earphones in real time and delivered to the user's ears.

[1334] Specific examples

[1335] Example 1: "What is an influencer?"

[1336] Users encounter the new term "influencer" in everyday conversation.

[1337] The device captures this audio and sends it to the server.

[1338] The server identifies "influencer" and translates it to "influential person."

[1339] The translated information is converted back into audio data and played in real time through the user's earphones.

[1340] The emotion engine analyzes the user's pulse and adjusts the tone and intonation to make it easier for the user to understand.

[1341] Example 2: "How do you use a smartphone?"

[1342] Pulse data shows that users are confused by the use of the foreign word "smartphone."

[1343] The device captures the audio and sends it to the server.

[1344] The server identifies "smartphone" and translates it into "a mobile phone with many functions."

[1345] The translated information is converted back into audio data and played in real time through the user's earphones.

[1346] The emotion engine detects the user's confusion and provides additional information, such as, "If you want to know more about the basic operations of your smartphone, it's best to learn by touching it."

[1347] Prompt Sentence Examples

[1348] Please explain what kind of hardware and software you will use for the specific process of "acquiring and transmitting voice data."

[1349]

[1350] Regarding "providing information based on emotions," please explain what algorithms or methods are used to analyze the user's emotions and provide information.

[1351] This system is designed to help elderly people become familiar with new words and information, enabling smooth communication. The introduction of an emotion engine makes it possible to provide more personalized information according to the user's emotional state, providing a comfortable user experience.

[1352] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1353] Step 1: Acquire audio data

[1354] The device captures the user's voice, and the microphone in the earphone collects ambient sounds and the user's speech in real time.

[1355] Input: User's speech

[1356] Data processing: Apply noise reduction algorithm to convert into clear audio data.

[1357] Output: Digital audio data

[1358] What it does: A microphone in the device captures an audio signal and converts it into a digital format.

[1359] Step 2: Sending audio data

[1360] The device sends the captured audio data to the server, where it encrypts the data using a secure communication protocol (e.g., HTTPS) to ensure the data is secure during transmission.

[1361] Input: Digital audio data

[1362] Data processing: Audio data is divided into packets and encrypted using HTTPS.

[1363] Output: Audio data sent to the server

[1364] Specific operation: The device sends audio data to the server via HTTPS.

[1365] Step 3: Convert audio data to text

[1366] The server converts the received audio data into text using a speech recognition algorithm (e.g., a generative AI model). At this stage, the temporal attributes of the audio are also preserved.

[1367] Input: Audio data sent to the server

[1368] Data calculation: Applying a speech recognition algorithm to generate text data.

[1369] Output: Text data

[1370] Specific operation: The server analyzes the audio data and converts it into text data in string format.

[1371] Step 4: Identifying and translating new or foreign words

[1372] The server identifies new words and foreign words from the converted text data, and these words are translated into words and expressions that are easy for seniors to understand using a translation dictionary in the database.

[1373] Input: Text data

[1374] Data calculation: Identifying new words and foreign words, and matching them with translation dictionaries

[1375] Output: Translated text data

[1376] How it works: The server analyzes the text data and matches the relevant terms with a database to obtain the translation.

[1377] Step 5: Re-speech of text data

[1378] The server converts the translated text data back into voice data using a speech synthesis service (e.g., speech synthesis API). This voice is adjusted to have a natural and friendly tone.

[1379] Input: Translated text data

[1380] Data calculation: Apply a voice synthesis algorithm to generate voice data.

[1381] Output: Audio data

[1382] Specific operation: The server sends text data to the speech synthesis API and obtains the synthesized speech data.

[1383] Step 6: Transmit translation information to earphone

[1384] The server then transmits the re-audio data to the user's earphones in real time, again using a secure communication protocol to ensure the data is safe.

[1385] Input: Revoiced audio data

[1386] Data processing: Audio data is divided into packets and encrypted using HTTPS.

[1387] Output: The audio data that reaches the user

[1388] Specific operation: The server sends audio data to the device using HTTPS, and the device outputs it to the earphones.

[1389] Step 7: Acquire biometric information

[1390] The device uses a built-in pulse sensor to acquire the user's pulse data in real time, and this data is sent to a server.

[1391] Input: User's pulse signal

[1392] Data processing: Convert the signal into digital pulse data.

[1393] Output: Digital pulse data

[1394] How it works: A sensor inside the device captures pulse signals and converts them into digital data.

[1395] Step 8: Sentiment Analysis

[1396] The server analyzes the user's emotional state using the received pulse data, and classifies the user's emotions based on the data using an emotion analysis algorithm.

[1397] Input: Pulse data

[1398] Data Computation: Apply sentiment analysis algorithms to assess emotional states.

[1399] Output: Emotional state data

[1400] Specific operation: The server analyzes the pulse data and evaluates the emotional state using a probabilistic model.

[1401] Step 9: Provide emotionally relevant information

[1402] The emotion engine prepares appropriate phrases and additional information based on the analysis results, and the converted information is also sent to the user's earphones as audio data in real time.

[1403] Input: Emotional state data, text data

[1404] Data calculation: Text data generation, speech synthesis

[1405] Output: Audio data

[1406] Specific operation: The emotion engine generates text data, and the server converts it into audio data using a speech synthesis API and sends it to the earphones.

[1407] These are the processing steps of this system, which makes it easier for elderly people to understand new terms and provides information that is appropriate for their emotional state.

[1408] (Application example 2)

[1409] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1410] Elderly people often have difficulty understanding the content of explanations of new products or services in brick-and-mortar stores. New words and foreign words are particularly difficult for elderly people to understand. Furthermore, depending on the elderly person's emotional state, information may not be provided at the appropriate time or in the appropriate tone, making it difficult for them to make appropriate decisions. This invention aims to make it easier for such elderly people to understand product explanations in brick-and-mortar stores and to provide information that adapts to the elderly person's emotional state.

[1411] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1412] In this invention, the server includes means for acquiring voice data, means for converting the acquired voice data into text data, means for identifying new words and foreign words contained in the text data and translating them into an easy-to-understand form, means for converting the translated text data into voice data and providing it to the user, means for acquiring biometric information of the user, means for analyzing the emotional state of the user based on the acquired biometric information and converting the text data into appropriate wording based on the analysis results, and means for translating and recognizing emotions when the user is receiving an explanation of new products or services in a physical store and providing the user with information in real time. This makes it possible to provide elderly people with information that is easy to understand and appropriate to their emotional state when receiving an explanation of new products or services in a physical store.

[1413] "Speech data" refers to data that records sound information collected from the user's speech and the surrounding sound environment.

[1414] "Text data" is character information obtained by processing voice data for recognition.

[1415] "New words and foreign words" are words that are difficult for users to understand or that they do not normally use, or words that have been imported from other languages.

[1416] "Translation" is the process of converting identified text data into a form that is easy for elderly people to understand.

[1417] "Speech recognition means" refers to a device or software for converting voice data into text data.

[1418] "Biometric information" refers to information including the user's pulse rate and other physiological data.

[1419] "Emotional state" refers to a user's current psychological state or feelings.

[1420] The "emotion recognition means" is a device or software for analyzing the user's emotional state from collected biometric information.

[1421] "Real-time" means acting immediately, without delay.

[1422] A "brick and mortar store" is a place that offers goods or services in a physical location.

[1423] A "user" is a person who uses this system.

[1424] This invention is a system that helps elderly people understand the content of new products and services in real time when they are explained in a physical store. The system collects and translates voice data, collects biometric information from users, and analyzes their emotions, and provides appropriate information based on this information.

[1425] composition:

[1426] The system consists of the following elements:

[1427] 1. Device that receives audio data:

[1428] A microphone in the device captures the user's voice in real time, for example, smart glasses or a smartphone.

[1429] 2. Server that analyzes voice data and biometric information:

[1430] The server converts the received voice data into text data using a speech recognition algorithm (e.g., Google Speech-to-Text API), while simultaneously analyzing biometric information.

[1431] 3. Earphones that provide information to the user:

[1432] Earphones to convert the translated text data back into audio data and provide it to the user in real time. The built-in speakers of smart glasses can also be used.

[1433] 4. Emotion engine for emotion recognition:

[1434] The device's built-in pulse sensor captures the user's pulse data in real time and sends it to a server. Based on this data, an emotion analysis algorithm evaluates the user's emotional state.

[1435] Operation:

[1436] 1. Audio data capture and transmission:

[1437] When a user asks a question about a product or receives an explanation in a store, the device's microphone captures the voice and transmits it to the server. The voice data is transmitted using a secure communication protocol.

[1438] 2. Speech data analysis and translation:

[1439] The server uses a speech recognition algorithm to convert the received voice data into text data, and identifies new or foreign words from this text data. For example, if the phrase "detox water" is included, it translates it into an easy-to-understand form (e.g., "water that flushes toxins from the body").

[1440] 3. Real-time audio output:

[1441] The translated text data is converted back into audio data on the server side and sent to the user's earphones in real time.

[1442] 4. Biometric data collection and emotion analysis:

[1443] The user's pulse sensor captures pulse data in real time and sends it to a server, where an emotion recognition algorithm analyzes it and provides appropriate information according to the user's emotional state.

[1444] Examples:

[1445] Example 1:

[1446] When seniors ask about "detox water" and new products.

[1447] Salesperson: "This is the latest detox water. It has detoxifying effects."

[1448] Seniors: "What's a detox?"

[1449] Example prompt sentence:

[1450] "What is detox water?"

[1451] "Tell me more about this new product."

[1452] Please explain the ingredients

[1453] In this situation, the system captures the elderly person's question, translates the new term "detox" into "something that has the function of expelling toxins from the body," and provides this information in audio. If the system senses that the elderly person is confused, it provides additional information such as, "We can also explain in detail the effects of detox water here."

[1454] Recommended hardware and software:

[1455] Speech Recognition: Google Speech-to-Text API

[1456] Translation: Dedicated translation server (e.g. Google Cloud Translation API)

[1457] Emotion Recognition: Dedicated emotion analysis algorithms (e.g., SomeEmotionRecognitionLibrary)

[1458] Audio output: pyttsx3 library (Python)

[1459] This means that when elderly people are explained about new products or services in a physical store, they can receive information in an easy-to-understand format immediately, helping them make appropriate decisions.

[1460] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1461] Step 1:

[1462] The terminal captures the user's voice data.

[1463] Input: User's voice

[1464] Output: Audio data (analog)

[1465] Processing: A microphone on the device picks up the user's voice in real time.

[1466] Specific operation: The words spoken by the user are captured by a microphone and stored as audio data on the device.

[1467] Step 2:

[1468] The device transmits the captured audio data to the server.

[1469] Input: Audio data (analog)

[1470] Output: Audio data (digital)

[1471] Processing: The audio data is converted into a digital format and sent to a server using a secure communication protocol.

[1472] Specific operation: The voice data in the device is converted into a digital format and sent to a server via the Internet.

[1473] Step 3:

[1474] The server converts the voice data into text data.

[1475] Input: Audio data (digital)

[1476] Output: Text data

[1477] Processing: Convert the audio data into text using a speech recognition algorithm (e.g., Google Speech-to-Text API).

[1478] Specific operation: The server analyzes the voice data and converts the spoken content into text information.

[1479] Step 4:

[1480] The server identifies new words and foreign words and translates them into a form that is easy for seniors to understand.

[1481] Input: Text data

[1482] Output: Translated text data

[1483] Processing: Identified new or foreign words are translated using a translation algorithm (e.g., Google Cloud Translation API).

[1484] Specific action: Translate the word "detox" as "the function of expelling toxins from the body."

[1485] Step 5:

[1486] The server converts the translated text data back into audio data.

[1487] Input: Translated text data

[1488] Output: Regenerated audio data

[1489] Processing: Convert the text data into audio data using a text-to-speech (TTS) algorithm (e.g., the pyttsx3 library).

[1490] Specific behavior: Regenerate text information written in different languages ​​as natural-sounding speech.

[1491] Step 6:

[1492] The terminal transmits the regenerated audio data to the earphone.

[1493] Input: Regenerated audio data

[1494] Output: Audio data delivered to earphones

[1495] Processing: The regenerated audio data is sent from the device to the earphones via a secure communication protocol.

[1496] Specific operation: Audio data sent from the server is delivered to the earphones in real time, and the user listens to the audio.

[1497] Step 7:

[1498] The device acquires the user's biometric information (such as pulse rate) and sends it to the server.

[1499] Input: Biometric information (pulse data, etc.)

[1500] Output: Biometric information (digital data)

[1501] Processing: Biometric data is acquired from the sensor device and sent to the server.

[1502] Specific operation: Acquires biometric data such as pulse rate and electrodermal activity (EDA) and sends it to a server via the Internet.

[1503] Step 8:

[1504] The server analyzes the emotional state based on biometric information and adds appropriate phrases.

[1505] Input: Biometric information (digital data)

[1506] Output: Sentiment analysis results and properly adjusted translation data

[1507] Processing: Emotion recognition algorithms analyze biometric data to assess the user's emotional state, and generate appropriate tones and additional supportive information based on the results.

[1508] What it does: If the user is confused, add a message to the translated information saying "Please let us know if you need further clarification."

[1509] In this way, the specific flow of implementing the invention based on the claims has been explained by dividing it into processing steps.

[1510] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1511] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1512] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1513] [Fourth embodiment]

[1514] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1515] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1516] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1517] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1518] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1519] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1520] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1521] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1522] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1523] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1524] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1525] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1526] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1527] MODE FOR CARRYING OUT THE INVENTION

[1528] The present invention is an AI earphone system that translates new words and foreign words used in everyday life into a form that is easy for elderly people to understand in real time. Specific embodiments of this system are described below.

[1529] System configuration and operation

[1530] This system consists of three main elements: a terminal for acquiring voice data, a server for analyzing the acquired voice data and biometric information, and earphones for providing information to the user.

[1531] Acquiring and sending audio data

[1532] The user puts on the earphones and engages in normal conversation. The microphone in the device captures the user's voice in real time and transmits the voice data to the server. The voice data is transmitted to the server using a secure communication protocol, ensuring data security.

[1533] Speech data analysis and translation

[1534] The server then runs the received voice data through a speech recognition algorithm and converts it into text data. It then identifies new words and foreign words from this text data and translates them into words and expressions that are easy for seniors to understand. For example, if the word "influencer" is detected, it will be translated as "influential person."

[1535] Real-time audio output

[1536] The translated text data is converted back into audio data on the server side and sent to the user's earphones in real time, allowing the user to receive the translated information in a natural voice.

[1537] Collection and analysis of biological information

[1538] The device is equipped with a biometric information acquisition device such as a pulse sensor that acquires the user's biometric information in real time. This data is sent to a server, where an emotion analysis algorithm evaluates the user's emotional state. For example, a high pulse rate may indicate tension.

[1539] Providing information according to emotions

[1540] Based on the analysis results, the server prepares appropriate phrases and additional information. For example, if it detects that the user is confused by a particular word, it will provide additional explanation for that word or convert it into a more familiar expression. This converted information is also sent to the user's earphones in real time and delivered to the user's ears.

[1541] Specific examples

[1542] Example 1:

[1543] In the scenario where a user asks "What is an influencer?", the following happens:

[1544] Users encounter the new term "influencer" in everyday conversations.

[1545] The device captures this audio and sends it to the server.

[1546] The server identifies "influencer" and translates it to "influential person."

[1547] The translated information is converted back into audio data and played in real time through the user's earphones.

[1548] Example 2:

[1549] In the scenario where a user asks "How do I use a smartphone?", the following happens:

[1550] Pulse data shows that users are confused by the use of the foreign word "smartphone."

[1551] The device captures the audio and sends it to the server.

[1552] The server identifies "smartphone" and translates it into "a mobile phone with many functions."

[1553] The translated information is converted back into audio data and played in real time through the user's earphones.

[1554] Additionally, the server detects the user's confusion and provides additional information such as, "If you want to know more about the basic operations of your smartphone, it's best to learn by touching it directly."

[1555] In this way, the present invention is a system that helps elderly people become familiar with new words and information and realizes smooth communication in their daily lives.

[1556] The processing flow will be explained below.

[1557] Specific processing steps of the program

[1558] Step 1: Acquire audio

[1559] The user puts on the earphones and engages in normal conversation.

[1560] A microphone built into the device captures the user's voice in real time.

[1561] Step 2: Sending audio data

[1562] The device then packetizes the captured audio data and sends it to the server using a secure protocol.

[1563] Step 3: Voice Recognition

[1564] The server analyzes the received voice data and converts it into text data using a voice recognition algorithm.

[1565] Step 4: Identifying new words

[1566] The server applies natural language processing to identify new and foreign words contained in the text data.

[1567] Step 5: Translation process

[1568] The server translates each identified new or foreign word into a word or phrase that is easy for seniors to understand, for example, converting "influencer" into "influential person."

[1569] Step 6: Audio translation

[1570] The server reconverts the translated text data into audio data using a speech synthesis algorithm.

[1571] Step 7: Submit your translation data

[1572] The server packetizes the generated voice data and transmits it to the terminal.

[1573] Step 8: Real-time output

[1574] The terminal plays back the received audio data in real time and provides it to the user.

[1575] Step 9: Acquire biometric information

[1576] A pulse sensor installed in the device acquires the user's pulse data.

[1577] Step 10: Sending biometric information

[1578] The terminal transmits the acquired pulse data to the server.

[1579] Step 11: Sentiment Analysis

[1580] The server analyzes the received pulse data and estimates the user's emotional state. For example, if the pulse rate is high, it is determined that the user is nervous.

[1581] Step 12: Generate additional information

[1582] The server then translates the message based on the results of the sentiment analysis, providing appropriate phrasing and additional explanations as needed. For example, if the user is confused, a more detailed explanation is added.

[1583] Step 13: Vocalize additional information

[1584] The server converts the generated additional text data into voice data using a voice synthesis algorithm.

[1585] Step 14: Submit additional information

[1586] The server transmits the additional information in the form of voice to the terminal.

[1587] Step 15: Play additional audio

[1588] The terminal reproduces the received audio data of the additional information in real time and provides it to the user.

[1589] These are the specific processing steps of "Generation Communicator." Through this processing flow, elderly people can easily understand new words and foreign words, realizing comfortable communication.

[1590] Example 1

[1591] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1592] In the past, it was often difficult for elderly people to understand new words and foreign words used in daily life. Furthermore, there was no system that could analyze and translate speech data in real time and provide appropriate information according to the user's emotional state. This made it difficult for elderly people to communicate smoothly when they encountered new words or concepts. The purpose of this invention is to solve this problem.

[1593] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1594] In this invention, the server includes a means for converting voice data into text data, a means for translating new words and foreign words contained in the text data into a form that is easy for seniors to understand, and a means for reconverting the translated text data into voice data. This enables seniors to understand new words and foreign words used in everyday conversation in real time, enabling smooth communication. Furthermore, by analyzing the user's emotional state based on biometric information and providing appropriate phrases and additional information based on the analysis results, a system that is more user-friendly and easy to use can be provided. This allows seniors to communicate smoothly without being confused by new words and information.

[1595] "Audio data" refers to data in the form of sound waves, such as user conversations or environmental sounds.

[1596] A "terminal" is a device for acquiring voice data and biometric information and transmitting that data to a server.

[1597] A "server" is a computer system that analyzes, transforms, and presents information to users.

[1598] "Text data" refers to data obtained by converting voice data into character information.

[1599] "New words and foreign words" are words that have been introduced into the user's native language in a relatively short period of time or words that have been adopted from a foreign country.

[1600] "Translation" is the act of converting something from one language into another language or into an understandable form.

[1601] "Biometric information" refers to physical data such as the user's pulse rate and body temperature.

[1602] "Emotional state" refers to a psychological state estimated based on the user's pulse rate and other biometric information.

[1603] An "emotion analysis algorithm" is a computational method for analyzing biometric information and assessing a user's emotional state.

[1604] "Additional information" refers to supplemental information or explanations provided depending on the user's emotional state.

[1605] A "secure communication protocol" is a communication protocol for safely sending and receiving data.

[1606] A "generative AI model" is an artificial intelligence model that learns patterns from large amounts of data and generates new information.

[1607] The present invention is a system that acquires, analyzes, and translates voice data in real time to provide users with easy-to-understand information about new words and foreign words. A specific embodiment of the present invention will be described. The system comprises a terminal for acquiring voice data, a server for analyzing the acquired voice data and biometric information, and earphones for providing translated information to the user.

[1608] Acquiring audio data

[1609] The device captures the user's voice in real time using a built-in microphone when they are having a normal conversation. The voice data is then temporarily stored on the device. For example, if a user says, "Young people these days are only chasing influencers," the voice data will be captured by the device.

[1610] Sending audio data

[1611] The terminal transmits the captured audio data to the server using a secure communication protocol (for example, HTTPS or TLS).

[1612] Analysis of audio data

[1613] The server converts the received voice data into text data using a speech recognition API such as Google Cloud Speech-to-Text or Amazon Transcribe. For example, the generated text data would be, "Young people these days are only chasing influencers."

[1614] Identifying and translating new and foreign words

[1615] The server analyzes the generated text data and identifies new words and foreign words (e.g., "influencer") using natural language processing (NLP) technology. It then translates these new words and foreign words into a form that is easy for seniors to understand (e.g., "influencer" becomes "influential person") using a pre-trained generative AI model or machine translation engine.

[1616] Converting translation text to audio data

[1617] The server reconverts the translated text data into audio data using a speech synthesis API such as Google Cloud Text-to-Speech or Amazon Polly.

[1618] Audio output to the user

[1619] The translated voice data is then sent to the device again using a secure communication protocol and finally output to the user's earphones. The user can listen to the translated voice in real time. For example, the translated voice may provide the user with information such as, "Young people these days only chase after influential people."

[1620] Collection and analysis of biological information

[1621] The device is equipped with a biometric information acquisition device such as a pulse sensor, which acquires the user's biometric information (e.g., pulse) in real time and temporarily stores the data. This biometric information is also transmitted to a server using a secure communication protocol, and the server evaluates the user's emotional state (e.g., tension) using an emotion analysis algorithm.

[1622] Providing information according to emotions

[1623] The server generates appropriate phrases and additional information based on the analysis results. For example, if it determines that the user is confused by the foreign word "smartphone," it generates advice such as, "If you want to learn more about the basic operations of a smartphone, it's a good idea to learn by touching it directly," and then converts this into audio data using Google Cloud Text-to-Speech and provides it to the user.

[1624] In this way, this system helps elderly people become familiar with new words and information, enabling smooth communication in their daily lives.

[1625] Examples of prompt statements

[1626] The following are examples of prompt sentences:

[1627] Capture conversations that include new foreign words or technical terms, translate them in real time to make them understandable for seniors, and provide appropriate phrasing or additional information based on the user's emotional state.

[1628] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1629] The flow of this system's program processing

[1630] Step 1: Acquire audio data

[1631] The device uses a built-in microphone to capture the user's voice in real time. The input is the user's voice, and the output is the captured voice data. This voice data is immediately and temporarily stored on the device. For example, when a user says, "What is an influencer?", the voice is captured.

[1632] Step 2: Sending audio data

[1633] The device sends the captured audio data to the server using a secure communication protocol (HTTPS or TLS). The input is the captured audio data, and the output is the audio data sent to the server. The transfer of the audio data makes it possible to proceed to the next analysis step.

[1634] Step 3: Analyzing the audio data

[1635] The server converts the received voice data into text data by running it through a speech recognition API such as Google Cloud Speech-to-Text or Amazon Transcribe. The input is the transmitted voice data, and the output is the generated text data. For example, the voice data "What is an influencer?" is converted into the text data "What is an influencer?"

[1636] Step 4: Identifying new words and loan words

[1637] The server analyzes the generated text data using natural language processing (NLP) techniques to identify new words and loan words. The input is the text data, and the output is a list of identified new words and loan words. For example, the word "influencer" is identified.

[1638] Step 5: Translation process

[1639] The server translates the identified new words and foreign words into a form that is easy for seniors to understand. It uses a generative AI model and a machine translation engine. The input is the identified new words and foreign words, and the output is the translated sentence. For example, the word "influencer" is translated into "influential person."

[1640] Step 6: Converting translation text to audio data

[1641] The server reconverts the translated text data into audio data using Google Cloud Text-to-Speech or Amazon Polly. The input is the translated text data, and the output is the reconverted audio data. For example, the text "influential person" is converted into audio data.

[1642] Step 7: Audio output to the user

[1643] The server sends the re-encoded voice data to the device using a secure communication protocol, and the device transmits the voice data to the user through earphones. The input is the re-encoded voice data, and the output is the voice the user hears in real time. The user hears the voice answer "Influential person" in response to the question "What is an influencer?"

[1644] Step 8: Collect biometric data

[1645] The device acquires the user's biometric information (e.g., pulse) in real time using a pulse sensor or the like. The input is a biometric signal such as the user's pulse, and the output is the acquired biometric data. For example, data on the user's pulse rate is collected.

[1646] Step 9: Sending biometric information

[1647] The device transmits the acquired biometric information to the server via a secure communication protocol. The input is the acquired biometric data, and the output is the biometric data transmitted to the server. The transfer of the biometric data enables the next analysis step.

[1648] Step 10: Analysis of biological information

[1649] The server then applies the received biometric data to an emotion analysis algorithm to evaluate the user's emotional state. The input is the transmitted biometric data, and the output is the analyzed emotional state. For example, a high pulse rate is evaluated as a state of tension.

[1650] Step 11: Provide emotionally relevant information

[1651] The server generates appropriate phrases and additional information based on the results of the emotion analysis, converts them into voice data, and provides them to the user. A generative AI model is used. The input is the analyzed emotional state and a prompt to generate appropriate phrases, and the output is the appropriate phrases and additional information converted into voice data. If the user is confused about a smartphone, advice such as "If you want to know more about the basic operations of a smartphone, it's best to learn by touching it directly" is provided as voice data.

[1652] (Application example 1)

[1653] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1654] The problem is that elderly people are confused by new words and foreign words, especially security-related terms, and have difficulty understanding them. Furthermore, when it comes to emergency responses, accurate and quick understanding is required, but elderly people are prone to feeling stressed and confused at times. In such situations, appropriate responses may not be possible, which could increase security risks.

[1655] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1656] In this invention, the server includes means for acquiring voice data, means for converting the acquired voice data into text data, means for identifying new words and foreign words contained in the text data and translating them into an easy-to-understand form, means for converting the translated text data into voice data and providing it to the user, means for acquiring the user's biometric information, means for analyzing the user's emotional state based on the acquired biometric information and converting the translated text data into appropriate phrases based on the analysis results, means for translating security-related terms and emergency response instructions into easy-to-understand forms and providing them to the user, and means for providing additional information to reduce the user's stress if the user's emotional state exceeds a reference value. This makes it easier for elderly people to understand security-related terms and emergency response instructions, enabling them to take necessary measures quickly and accurately.

[1657] The "means for acquiring voice data" refers to a device for capturing the user's voice in real time and acquiring the data.

[1658] "Means for converting acquired voice data into text data" refers to a device or software that uses voice recognition technology to convert voice data into text information.

[1659] "Means for identifying new words and foreign words contained in the text data and translating them into an easily understandable form" refers to a device or software that detects new or difficult terms from the converted text data and replaces them with words that are easy for seniors to understand.

[1660] "Means for converting translated text data into audio data and providing it to the user" refers to a device or software that plays back the translated text data as audio.

[1661] "Means for acquiring user's biometric information" refers to a device that acquires biometric data such as the user's pulse and skin potential in real time.

[1662] "Means for analyzing the user's emotional state based on the acquired biometric information and converting it into appropriate phrases based on the analysis results" refers to a device or software that analyzes biometric information to evaluate the user's emotional state and translates it into more appropriate expressions based on the results.

[1663] "Means for translating security-related terms and emergency response instructions into easy-to-understand language and providing them to users" refers to a device or software that translates difficult security-related terms and instructions into simple language and provides them to users as audio.

[1664] "Means for providing additional information to reduce stress in a user when the emotional state exceeds a reference value" refers to a device or software that provides supplemental information to reduce stress in a user when the analyzed emotional state exceeds a reference value.

[1665] System configuration and operation

[1666] The system consists of the following elements:

[1667] 1. Terminal: A device that acquires voice data and collects the user's biometric information.

[1668] 2. Server: A device that analyzes and translates voice data and generates additional information based on the translation results.

[1669] 3. Earphones: A device that provides translated text data as audio.

[1670] Acquiring and sending audio data

[1671] The user puts on the earphones and starts a normal conversation. The microphone in the device captures the user's voice in real time and transmits the voice data to the server. A secure communication protocol is used for this transmission, ensuring the safety of the data.

[1672] Speech data analysis and translation

[1673] The server then runs the received voice data through a speech recognition algorithm and converts it into text. It then identifies new or foreign words from this text and translates them into language that is easy for seniors to understand. For example, if the term "phishing attack" is detected, it will be translated into "a method of deceiving personal information."

[1674] Real-time audio output

[1675] The translated text data is converted back into audio data on the server side and sent to the user's earphones in real time, allowing the user to receive the translated information in a natural voice.

[1676] Collection and analysis of biological information

[1677] The device is equipped with a biometric information acquisition device, such as a pulse sensor, that acquires the user's biometric information in real time. This data is sent to a server, where an emotion analysis algorithm evaluates the user's emotional state. For example, a high pulse rate may indicate tension.

[1678] Providing information according to emotions

[1679] Based on the analysis results, the server prepares appropriate phrases and additional information. For example, if it detects that the user is confused by a particular word, it will provide additional explanations or convert the word into a more user-friendly expression. This converted information is also sent to the user's earphones in real time and delivered to the user's ears.

[1680] Specific examples

[1681] Example 1: In this scenario, if a user asks "What is a phishing attack?", the following will happen:

[1682] Users encounter a new term in security conversations: "phishing attack."

[1683] The device captures this audio and sends it to the server.

[1684] The server identifies "phishing attacks" and translates them into "tricks to trick people into revealing personal information."

[1685] The translated information is converted back into audio data and played in real time through the user's earphones.

[1686] At the same time, if the user's biometric information indicates a state of tension, additional information and advice is provided, such as "When it comes to phishing attacks, it's best to ignore the links and emails."

[1687] Example prompts to input to a generative AI model:

[1688] "How are phishing attacks used online?"

[1689] Hardware and software used

[1690] Hardware:

[1691] Devices (smartphones, smart glasses, head-mounted displays)

[1692] Biometric information acquisition device (pulse sensor)

[1693] software:

[1694] Speech Recognition Library: speech_recognition

[1695] Biosensor library: some_bio_sensor_library

[1696] Text-to-speech (TTS) library: pyttsx3

[1697] HTTP request library:requests

[1698] Server-side processing: NLP models, sentiment analysis algorithms, speech conversion engines

[1699] In this way, the present invention is a system that helps seniors become familiar with new words and information and improves their understanding and ability to respond to security-related issues.

[1700] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1701] Step 1:

[1702] The device acquires the user's voice data. Specifically, the device's microphone is used to capture the user's voice in real time. The acquired voice data is saved as raw voice waveform data.

[1703] Step 2:

[1704] The terminal transmits the acquired voice data to the server. Specifically, the terminal transfers the voice data to the server using a secure communication protocol. This communication ensures the security of the data.

[1705] Step 3:

[1706] The server runs the received voice data through a speech recognition algorithm and converts it into text data. Specifically, it uses a speech recognition library (e.g., speech_recognition) to convert the voice waveform data into corresponding text data. This conversion process uses an acoustic model and a language model.

[1707] Step 4:

[1708] The server identifies new words and foreign words contained in the converted text data. Specifically, it uses a generative AI model to analyze the text data and detect new words and foreign words. NLP technology is applied in this analysis.

[1709] Step 5:

[1710] The server translates the identified new words and foreign words into a form that is easy for seniors to understand. Specifically, the generative AI model refers to a translation dictionary and translates, for example, "phishing attack" into "a method of defrauding personal information." The translation results are saved as text data.

[1711] Step 6:

[1712] The server converts the translated text back into audio using a text-to-speech (TTS) library (e.g., pyttsx3). This audio is then given natural pronunciation and intonation.

[1713] Step 7:

[1714] The server transmits the re-converted audio data to the device in real time using a secure communication protocol. The device receives the audio data and immediately provides it to the user through earphones.

[1715] Step 8:

[1716] The device acquires biometric information. Specifically, it uses a pulse sensor built into the device to acquire biometric data such as the user's pulse and skin potential in real time. This data is saved as numerical data.

[1717] Step 9:

[1718] The terminal transmits the acquired biometric information to the server. Specifically, the biometric data is transferred to the server using a secure communication protocol. This communication ensures the security of the data.

[1719] Step 10:

[1720] The server analyzes the user's emotional state based on the acquired biometric information. Specifically, it uses a biometric analysis algorithm to evaluate, for example, whether the user is nervous. The evaluation results are saved as numerical data.

[1721] Step 11:

[1722] The server prepares appropriate phrases and additional information based on the analysis results. Specifically, the generative AI model uses the sentiment analysis results to generate additional advice, such as "Please ignore phishing attacks." This additional information is saved as text data.

[1723] Step 12:

[1724] The server converts the prepared additional information into audio data and sends it to the device. Specifically, it uses a text-to-speech (TTS) library to convert the text data of the additional information into audio data and transmits it to the device using a secure communication protocol. The device receives this audio data and provides it to the user through earphones.

[1725] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1726] MODE FOR CARRYING OUT THE INVENTION

[1727] The present invention is an AI earphone system that translates new words and foreign words in real time into a form that is easy for elderly people to understand, and also recognizes the user's emotions and provides information according to their state. Specific embodiments of this system are described below.

[1728] System configuration and operation

[1729] This system consists of a terminal that acquires voice data, a server that analyzes the acquired voice data and biometric information, earphones that provide information to the user, and an emotion engine that performs emotion recognition.

[1730] Acquiring and sending audio data

[1731] The user puts on the earphones and engages in normal conversation. The microphone in the device captures the user's voice in real time and transmits the voice data to the server. The voice data is transmitted to the server using a secure communication protocol, ensuring data security.

[1732] Speech data analysis and translation

[1733] The server converts the received voice data into text data using a speech recognition algorithm. New words and foreign words are identified from this text data and translated into words and expressions that are easy for seniors to understand. For example, if the word "influencer" is detected, it will be translated into "influential person."

[1734] Real-time audio output

[1735] The translated text data is converted back into audio data on the server side and sent to the user's earphones in real time, allowing the user to receive the translated information in a natural voice.

[1736] Biometric data collection and emotion analysis

[1737] A pulse sensor installed in the device captures the user's pulse data in real time. This data is sent to a server, where an emotion analysis algorithm evaluates the user's emotional state. The emotion engine then analyzes this data and adjusts the tone and intonation of the information according to the user's emotional state.

[1738] Providing information according to emotions

[1739] The emotion engine prepares appropriate phrases and additional information based on the user's emotional state. For example, if it senses that the user is confused by a particular word, it will provide additional explanation or convert that word into a more friendly expression. This converted information is also sent to the user's earphones in real time and delivered to the user's ears.

[1740] Specific examples

[1741] Example 1:

[1742] In the scenario where a user asks "What is an influencer?", the following happens:

[1743] Users encounter the new term "influencer" in everyday conversations.

[1744] The device captures this audio and sends it to the server.

[1745] The server identifies "influencer" and translates it to "influential person."

[1746] The translated information is converted back into audio data and played in real time through the user's earphones.

[1747] The emotion engine analyzes the user's pulse and adjusts the tone and intonation to make it easier for the user to understand.

[1748] Example 2:

[1749] In the scenario where a user asks "How do I use a smartphone?", the following happens:

[1750] Pulse data shows that users are confused by the use of the foreign word "smartphone."

[1751] The device captures the audio and sends it to the server.

[1752] The server identifies "smartphone" and translates it into "a mobile phone with many functions."

[1753] The translated information is converted back into audio data and played in real time through the user's earphones.

[1754] The emotion engine detects the user's confusion and provides additional information such as, "If you want to know more about the basic operations of your smartphone, it's best to learn by touching it."

[1755] In this way, the present invention is a system that helps elderly people become familiar with new words and information and realizes smooth communication in their daily lives. Furthermore, the introduction of an emotion engine makes it possible to provide more personalized information according to the user's emotional state, providing a comfortable user experience.

[1756] The processing flow will be explained below.

[1757] MODE FOR CARRYING OUT THE INVENTION

[1758] The present invention is an AI earphone system that translates new words and foreign words into a form that is easy for seniors to understand in real time, and also recognizes the user's emotions and provides information according to their state. A specific embodiment of this system will be described in detail below, with each processing step explained.

[1759] Step 1: Acquire audio

[1760] The user puts on the earphones and engages in normal conversation.

[1761] A microphone in the device captures the user's voice in real time.

[1762] Step 2: Sending audio data

[1763] The terminal packetizes the captured audio data and transmits it to the server.

[1764] The audio data is sent to the server using a secure communication protocol, ensuring data security.

[1765] Step 3: Voice Recognition

[1766] The server analyzes the received voice data and converts it into text data using a voice recognition algorithm.

[1767] For example, the spoken word "influencer" is converted into text data.

[1768] Step 4: Identifying new words

[1769] The server applies natural language processing to identify new and foreign words contained in the text data.

[1770] For example, "influencers" are identified.

[1771] Step 5: Translation process

[1772] The server translates each identified new word or foreign word into words or expressions that are easy for seniors to understand.

[1773] For example, convert "influencer" to "influencer."

[1774] Step 6: Audio translation

[1775] The server reconverts the translated text data into audio data using a speech synthesis algorithm.

[1776] Step 7: Submit your translation data

[1777] The server packetizes the generated voice data and transmits it to the terminal.

[1778] Step 8: Real-time output

[1779] The terminal plays back the received audio data in real time and provides it to the user.

[1780] Step 9: Acquire biometric information

[1781] The pulse sensor installed in the device collects the user's pulse data in real time.

[1782] Step 10: Sending biometric information

[1783] The terminal transmits the acquired pulse data to the server.

[1784] Step 11: Sentiment Analysis

[1785] The server analyzes the received pulse data and estimates the user's emotional state.

[1786] For example, if your pulse rate is high, it is determined that you are nervous.

[1787] Step 12: Adjusting Tone and Intonation

[1788] The server's emotional engine adjusts the tone and intonation of the translated words based on the user's emotional state.

[1789] For example, if the user is nervous, the tone is changed to a gentler tone.

[1790] Step 13: Generate additional information

[1791] The server's emotion engine translates appropriate phrases and additional explanations as needed based on the results of emotion analysis.

[1792] For example, if the user is confused, provide a more detailed explanation.

[1793] Step 14: Vocalize additional information

[1794] The server converts the generated additional text data into voice data using a voice synthesis algorithm.

[1795] Step 15: Submit additional information

[1796] The server transmits the additional information in the form of voice to the terminal.

[1797] Step 16: Playing Additional Audio

[1798] The terminal reproduces the received audio data of the additional information in real time and provides it to the user.

[1799] Through these processing steps, users can deepen their understanding of new words and foreign words and receive personalized information in real time according to their emotional state. This system will facilitate communication among the elderly.

[1800] Example 2

[1801] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1802] In today's information society, new words and foreign words are constantly appearing, and they are often difficult for the elderly to understand. There is also a lack of appropriate support to help them understand this new information. Furthermore, communication can sometimes be hindered by a lack of information provided in accordance with the user's emotional state. It is important to resolve these issues and provide appropriate information that makes it easier for the elderly to understand new words and information, and that responds to their emotional state.

[1803] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1804] In this invention, the server includes a means for converting voice data into text data, a means for identifying new words and foreign words contained in the text data and translating them into an easily understandable form, and a means for converting the translated text data into voice data and providing it to the user. This makes it possible to convert new words and foreign words into a form that is easy for elderly people to understand in real time, and to provide information according to the user's emotional state.

[1805] "Voice data" refers to a digital representation of spoken information uttered by a user.

[1806] "Text data" is voice data converted into a string of characters.

[1807] "New words" are recently introduced terms or phrases that are unfamiliar to a particular user.

[1808] "Foreign words" refer to words introduced from foreign countries that are difficult for certain users to understand.

[1809] "Translation means" refers to techniques or methods that convert identified new words or foreign words into a form that is easy for users to understand.

[1810] "Biometric information" is data obtained from the user's body, and includes pulse rate, heart rate, and the like.

[1811] "Emotional state" refers to the user's state of mind, and includes happiness, sadness, confusion, and the like.

[1812] "Emotion analysis" is the process of assessing a user's emotional state based on acquired biometric information.

[1813] A "voice recognition algorithm" is a technology that converts acquired voice data into text data in the form of a string.

[1814] A "speech synthesis service" is a technology or platform for converting text data into voice data.

[1815] A "server" is a computer system that analyzes voice data and biometric information and processes the data.

[1816] A "terminal" is a device that acquires voice data and biometric information from a user and transmits it to a server.

[1817] "Translated data" refers to text data that has been converted from identified new words or foreign words into an easily understandable form.

[1818] A "secure communications protocol" is a communications procedure used to ensure the security of information when transmitting and receiving data.

[1819] A "pulse sensor" is a device for measuring a user's pulse and acquiring biometric information.

[1820] The "emotion engine" is a system that analyzes acquired biometric information and provides information according to the user's emotional state.

[1821] This system translates new words and foreign words into a form that is easy for the elderly to understand in real time, and provides information according to the user's emotional state. This system consists of a terminal that acquires voice data, a server that analyzes the voice data and biometric information, earphones that provide information to the user, and an emotion engine that recognizes emotions.

[1822] Acquiring and sending audio data

[1823] The user puts on the earphones and engages in normal conversation. The microphone in the device captures the user's voice in real time and sends the voice data to the server. The data is sent using a secure communication protocol (e.g., HTTPS), ensuring data safety.

[1824] Speech data analysis and translation

[1825] The server converts the received voice data into text data using a speech recognition algorithm (e.g., Google Cloud Speech-to-Text API). New words and foreign words are identified from this text data and translated into words and expressions that are easy for seniors to understand. For example, if the word "influencer" is detected, it will be translated as "influential person."

[1826] Real-time audio output

[1827] The translated text data is converted back into audio data on the server side. This process uses a speech synthesis service (e.g., Amazon Polly). The converted audio data is sent to the user's earphones in real time, allowing the user to receive the translated information in a natural voice.

[1828] Biometric data collection and emotion analysis

[1829] A pulse sensor installed in the device captures the user's pulse data in real time. This data is sent to a server, where an emotion analysis algorithm evaluates the user's emotional state. The emotion engine then analyzes this data and adjusts the tone and intonation of the information according to the user's emotional state.

[1830] Providing information according to emotions

[1831] The emotion engine prepares appropriate phrases and additional information based on the user's emotional state. For example, if it senses that the user is confused by a particular word, it will provide additional explanation or convert that word into a more friendly expression. This converted information is also sent to the user's earphones in real time and delivered to the user's ears.

[1832] Specific examples

[1833] Example 1: "What is an influencer?"

[1834] Users encounter the new term "influencer" in everyday conversation.

[1835] The device captures this audio and sends it to the server.

[1836] The server identifies "influencer" and translates it to "influential person."

[1837] The translated information is converted back into audio data and played in real time through the user's earphones.

[1838] The emotion engine analyzes the user's pulse and adjusts the tone and intonation to make it easier for the user to understand.

[1839] Example 2: "How do you use a smartphone?"

[1840] Pulse data shows that users are confused by the use of the foreign word "smartphone."

[1841] The device captures the audio and sends it to the server.

[1842] The server identifies "smartphone" and translates it into "a mobile phone with many functions."

[1843] The translated information is converted back into audio data and played in real time through the user's earphones.

[1844] The emotion engine detects the user's confusion and provides additional information, such as, "If you want to know more about the basic operations of your smartphone, it's best to learn by touching it."

[1845] Prompt Sentence Examples

[1846] Please explain what kind of hardware and software you will use for the specific process of "acquiring and transmitting voice data."

[1847]

[1848] Regarding "providing information based on emotions," please explain what algorithms or methods are used to analyze the user's emotions and provide information.

[1849] This system is designed to help elderly people become familiar with new words and information, enabling smooth communication. The introduction of an emotion engine makes it possible to provide more personalized information according to the user's emotional state, providing a comfortable user experience.

[1850] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1851] Step 1: Acquire audio data

[1852] The device captures the user's voice, and the microphone in the earphone collects ambient sounds and the user's speech in real time.

[1853] Input: User's speech

[1854] Data processing: Apply noise reduction algorithm to convert into clear audio data.

[1855] Output: Digital audio data

[1856] What it does: A microphone in the device captures an audio signal and converts it into a digital format.

[1857] Step 2: Sending audio data

[1858] The device sends the captured audio data to the server, where it encrypts the data using a secure communication protocol (e.g., HTTPS) to ensure the data is secure during transmission.

[1859] Input: Digital audio data

[1860] Data processing: Audio data is divided into packets and encrypted using HTTPS.

[1861] Output: Audio data sent to the server

[1862] Specific operation: The device sends audio data to the server via HTTPS.

[1863] Step 3: Convert audio data to text

[1864] The server converts the received audio data into text using a speech recognition algorithm (e.g., a generative AI model). At this stage, the temporal attributes of the audio are also preserved.

[1865] Input: Audio data sent to the server

[1866] Data calculation: Applying a speech recognition algorithm to generate text data.

[1867] Output: Text data

[1868] Specific operation: The server analyzes the audio data and converts it into text data in string format.

[1869] Step 4: Identifying and translating new or foreign words

[1870] The server identifies new words and foreign words from the converted text data, and these words are translated into words and expressions that are easy for seniors to understand using a translation dictionary in the database.

[1871] Input: Text data

[1872] Data calculation: Identifying new words and foreign words, and matching them with translation dictionaries

[1873] Output: Translated text data

[1874] How it works: The server analyzes the text data and matches the relevant terms with a database to obtain the translation.

[1875] Step 5: Re-speech of text data

[1876] The server converts the translated text data back into voice data using a speech synthesis service (e.g., speech synthesis API). This voice is adjusted to have a natural and friendly tone.

[1877] Input: Translated text data

[1878] Data calculation: Apply a voice synthesis algorithm to generate voice data.

[1879] Output: Audio data

[1880] Specific operation: The server sends text data to the speech synthesis API and obtains the synthesized speech data.

[1881] Step 6: Transmit translation information to earphone

[1882] The server then transmits the re-audio data to the user's earphones in real time, again using a secure communication protocol to ensure the data is safe.

[1883] Input: Revoiced audio data

[1884] Data processing: Audio data is divided into packets and encrypted using HTTPS.

[1885] Output: The audio data that reaches the user

[1886] Specific operation: The server sends audio data to the device using HTTPS, and the device outputs it to the earphones.

[1887] Step 7: Acquire biometric information

[1888] The device uses a built-in pulse sensor to acquire the user's pulse data in real time, and this data is sent to a server.

[1889] Input: User's pulse signal

[1890] Data processing: Convert the signal into digital pulse data.

[1891] Output: Digital pulse data

[1892] How it works: A sensor inside the device captures pulse signals and converts them into digital data.

[1893] Step 8: Sentiment Analysis

[1894] The server analyzes the user's emotional state using the received pulse data, and classifies the user's emotions based on the data using an emotion analysis algorithm.

[1895] Input: Pulse data

[1896] Data Computation: Apply sentiment analysis algorithms to assess emotional states.

[1897] Output: Emotional state data

[1898] Specific operation: The server analyzes the pulse data and evaluates the emotional state using a probabilistic model.

[1899] Step 9: Provide emotionally relevant information

[1900] The emotion engine prepares appropriate phrases and additional information based on the analysis results, and the converted information is also sent to the user's earphones as audio data in real time.

[1901] Input: Emotional state data, text data

[1902] Data calculation: Text data generation, speech synthesis

[1903] Output: Audio data

[1904] Specific operation: The emotion engine generates text data, and the server converts it into audio data using a speech synthesis API and sends it to the earphones.

[1905] These are the processing steps of this system, which makes it easier for elderly people to understand new terms and provides information that is appropriate for their emotional state.

[1906] (Application example 2)

[1907] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1908] Elderly people often have difficulty understanding the content of explanations of new products or services in brick-and-mortar stores. New words and foreign words are particularly difficult for elderly people to understand. Furthermore, depending on the elderly person's emotional state, information may not be provided at the appropriate time or in the appropriate tone, making it difficult for them to make appropriate decisions. This invention aims to make it easier for such elderly people to understand product explanations in brick-and-mortar stores and to provide information that adapts to the elderly person's emotional state.

[1909] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1910] In this invention, the server includes means for acquiring voice data, means for converting the acquired voice data into text data, means for identifying new words and foreign words contained in the text data and translating them into an easy-to-understand form, means for converting the translated text data into voice data and providing it to the user, means for acquiring biometric information of the user, means for analyzing the emotional state of the user based on the acquired biometric information and converting the text data into appropriate wording based on the analysis results, and means for translating and recognizing emotions when the user is receiving an explanation of new products or services in a physical store and providing the user with information in real time. This makes it possible to provide elderly people with information that is easy to understand and appropriate to their emotional state when receiving an explanation of new products or services in a physical store.

[1911] "Speech data" refers to data that records sound information collected from the user's speech and the surrounding sound environment.

[1912] "Text data" is character information obtained by processing voice data for recognition.

[1913] "New words and foreign words" are words that are difficult for users to understand or that they do not normally use, or words that have been imported from other languages.

[1914] "Translation" is the process of converting identified text data into a form that is easy for elderly people to understand.

[1915] "Speech recognition means" refers to a device or software for converting voice data into text data.

[1916] "Biometric information" refers to information including the user's pulse rate and other physiological data.

[1917] "Emotional state" refers to a user's current psychological state or feelings.

[1918] The "emotion recognition means" is a device or software for analyzing the user's emotional state from collected biometric information.

[1919] "Real-time" means acting immediately, without delay.

[1920] A "brick and mortar store" is a place that offers goods or services in a physical location.

[1921] A "user" is a person who uses this system.

[1922] This invention is a system that helps elderly people understand the content of new products and services in real time when they are explained in a physical store. The system collects and translates voice data, collects biometric information from users, and analyzes their emotions, and provides appropriate information based on this information.

[1923] composition:

[1924] The system consists of the following elements:

[1925] 1. Device that receives audio data:

[1926] A microphone in the device captures the user's voice in real time, for example, smart glasses or a smartphone.

[1927] 2. Server that analyzes voice data and biometric information:

[1928] The server converts the received voice data into text data using a speech recognition algorithm (e.g., Google Speech-to-Text API), while simultaneously analyzing biometric information.

[1929] 3. Earphones that provide information to the user:

[1930] Earphones to convert the translated text data back into audio data and provide it to the user in real time. The built-in speakers of smart glasses can also be used.

[1931] 4. Emotion engine for emotion recognition:

[1932] The device's built-in pulse sensor captures the user's pulse data in real time and sends it to a server. Based on this data, an emotion analysis algorithm evaluates the user's emotional state.

[1933] Operation:

[1934] 1. Audio data capture and transmission:

[1935] When a user asks a question about a product or receives an explanation in a store, the device's microphone captures the voice and transmits it to the server. The voice data is transmitted using a secure communication protocol.

[1936] 2. Speech data analysis and translation:

[1937] The server uses a speech recognition algorithm to convert the received voice data into text data, and identifies new or foreign words from this text data. For example, if the phrase "detox water" is included, it translates it into an easy-to-understand form (e.g., "water that flushes toxins from the body").

[1938] 3. Real-time audio output:

[1939] The translated text data is converted back into audio data on the server side and sent to the user's earphones in real time.

[1940] 4. Biometric data collection and emotion analysis:

[1941] The user's pulse sensor captures pulse data in real time and sends it to a server, where an emotion recognition algorithm analyzes it and provides appropriate information according to the user's emotional state.

[1942] Examples:

[1943] Example 1:

[1944] When seniors ask about "detox water" and new products.

[1945] Salesperson: "This is the latest detox water. It has detoxifying effects."

[1946] Seniors: "What's a detox?"

[1947] Example prompt sentence:

[1948] "What is detox water?"

[1949] "Tell me more about this new product."

[1950] Please explain the ingredients

[1951] In this situation, the system captures the elderly person's question, translates the new term "detox" into "something that has the function of expelling toxins from the body," and provides this information in audio. If the system senses that the elderly person is confused, it provides additional information such as, "We can also explain in detail the effects of detox water here."

[1952] Recommended hardware and software:

[1953] Speech Recognition: Google Speech-to-Text API

[1954] Translation: Dedicated translation server (e.g. Google Cloud Translation API)

[1955] Emotion Recognition: Dedicated emotion analysis algorithms (e.g., SomeEmotionRecognitionLibrary)

[1956] Audio output: pyttsx3 library (Python)

[1957] This means that when elderly people are explained about new products or services in a physical store, they can receive information in an easy-to-understand format immediately, helping them make appropriate decisions.

[1958] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1959] Step 1:

[1960] The terminal captures the user's voice data.

[1961] Input: User's voice

[1962] Output: Audio data (analog)

[1963] Processing: A microphone on the device picks up the user's voice in real time.

[1964] Specific operation: The words spoken by the user are captured by a microphone and stored as audio data on the device.

[1965] Step 2:

[1966] The device transmits the captured audio data to the server.

[1967] Input: Audio data (analog)

[1968] Output: Audio data (digital)

[1969] Processing: The audio data is converted into a digital format and sent to a server using a secure communication protocol.

[1970] Specific operation: The voice data in the device is converted into a digital format and sent to a server via the Internet.

[1971] Step 3:

[1972] The server converts the voice data into text data.

[1973] Input: Audio data (digital)

[1974] Output: Text data

[1975] Processing: Convert the audio data into text using a speech recognition algorithm (e.g., Google Speech-to-Text API).

[1976] Specific operation: The server analyzes the voice data and converts the spoken content into text information.

[1977] Step 4:

[1978] The server identifies new words and foreign words and translates them into a form that is easy for seniors to understand.

[1979] Input: Text data

[1980] Output: Translated text data

[1981] Processing: Identified new or foreign words are translated using a translation algorithm (e.g., Google Cloud Translation API).

[1982] Specific action: Translate the word "detox" as "the function of expelling toxins from the body."

[1983] Step 5:

[1984] The server converts the translated text data back into audio data.

[1985] Input: Translated text data

[1986] Output: Regenerated audio data

[1987] Processing: Convert the text data into audio data using a text-to-speech (TTS) algorithm (e.g., the pyttsx3 library).

[1988] Specific behavior: Regenerate text information written in different languages ​​as natural-sounding speech.

[1989] Step 6:

[1990] The terminal transmits the regenerated audio data to the earphone.

[1991] Input: Regenerated audio data

[1992] Output: Audio data delivered to earphones

[1993] Processing: The regenerated audio data is sent from the device to the earphones via a secure communication protocol.

[1994] Specific operation: Audio data sent from the server is delivered to the earphones in real time, and the user listens to the audio.

[1995] Step 7:

[1996] The device acquires the user's biometric information (such as pulse rate) and sends it to the server.

[1997] Input: Biometric information (pulse data, etc.)

[1998] Output: Biometric information (digital data)

[1999] Processing: Biometric data is acquired from the sensor device and sent to the server.

[2000] Specific operation: Acquires biometric data such as pulse rate and electrodermal activity (EDA) and sends it to a server via the Internet.

[2001] Step 8:

[2002] The server analyzes the emotional state based on biometric information and adds appropriate phrases.

[2003] Input: Biometric information (digital data)

[2004] Output: Sentiment analysis results and properly adjusted translation data

[2005] Processing: Emotion recognition algorithms analyze biometric data to assess the user's emotional state, and generate appropriate tones and additional supportive information based on the results.

[2006] What it does: If the user is confused, add a message to the translated information saying "Please let us know if you need further clarification."

[2007] In this way, the specific flow of implementing the invention based on the claims has been explained by dividing it into processing steps.

[2008] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2009] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2010] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2011] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2012] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2013] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2014] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2015] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2016] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2017] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2018] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2019] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2020] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2021] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2022] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2023] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2024] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2025] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2026] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2027] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2028] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2029] The following is further disclosed regarding the above embodiment.

[2030] (Claim 1)

[2031] means for acquiring audio data;

[2032] A means for converting the acquired voice data into text data;

[2033] means for identifying new words and foreign words contained in said text data and translating them into an easily understandable form;

[2034] means for converting the translated text data into speech data and providing it to the user;

[2035] means for acquiring biometric information of a user;

[2036] A means for analyzing the emotional state of the user based on the acquired biometric information and converting the emotional state into an appropriate phrase based on the analysis result;

[2037] A system including:

[2038] (Claim 2)

[2039] 2. The system according to claim 1, wherein the acquired voice data is transmitted to a server, and the server converts the voice data into text data.

[2040] (Claim 3)

[2041] 2. The system according to claim 1, further comprising: analyzing the pulse of the user based on the acquired biological information.

[2042] "Example 1"

[2043] (Claim 1)

[2044] means for acquiring audio data;

[2045] means for transmitting the acquired voice data to a server;

[2046] A means for the server to convert the voice data into text data;

[2047] means for identifying new words and foreign words contained in said text data and translating them into an easily understandable form;

[2048] means for converting the translated text data back into speech data on the server side;

[2049] means for providing the translated speech data to the user;

[2050] means for acquiring biometric information of a user;

[2051] means for transmitting the acquired biometric information to a server;

[2052] means for analyzing the emotional state of the user based on the acquired biometric information;

[2053] A means for generating appropriate phrases and additional information based on the analysis results, converting them into voice data, and providing them to the user;

[2054] A system including:

[2055] (Claim 2)

[2056] 10. The system of claim 1, wherein the server performs analysis of the speech data and translation of new words and foreign words.

[2057] (Claim 3)

[2058] 10. The system according to claim 1, further comprising: analyzing the pulse of the user based on the acquired biological information.

[2059] "Application Example 1"

[2060] ...

Claims

1. means for acquiring audio data; A means for converting the acquired voice data into text data; means for identifying new words and foreign words contained in said text data and translating them into an easily understandable form; means for converting the translated text data into speech data and providing it to the user; means for acquiring biometric information of a user; A means for analyzing the emotional state of the user based on the acquired biometric information and converting the emotional state into an appropriate phrase based on the analysis result; A system including:

2. 2. The system according to claim 1, wherein the acquired voice data is transmitted to a server, and the server converts the voice data into text data.

3. The system according to claim 1 , further comprising: analyzing the pulse of the user based on the acquired biological information.

Images