System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving voice input from a user; means for converting the voice input into text; means for sending the converted text to a natural language processing engine to analyze the user's intent; means for searching a database based on the analyzed intent to obtain corresponding information; means for generating an answer based on the obtained information; means for converting the generated answer into voice data; and means for playing the voice data to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern times, people rely on books and online resources to learn about the thoughts and lives of great historical figures. However, these methods lack the ability to provide an interactive learning experience, limiting learning effectiveness. Furthermore, existing voice assistants provide poor responses to user questions, preventing a satisfactory learning experience. The present invention aims to solve these problems by providing a new form of educational experience in which users can learn through dialogue with great historical figures. [Means for solving the problem]

[0005] The present invention provides a system that includes a speech recognition unit that receives speech input from a user and converts it into text. It also includes a unit that sends the converted text to a natural language processing engine and analyzes the user's intent. It also includes a unit that searches a database based on the analyzed intent and obtains corresponding information. It also includes a unit that generates an answer based on the obtained information, a speech synthesis unit that converts the generated answer into audio data, and a unit that plays back the audio data to the user. This system allows a user to learn information about historical figures in an interactive format, providing a more interactive and effective learning experience than traditional learning methods.

[0006] A "user" is a person who uses the system to provide voice input to learn information about a great person.

[0007] "Voice input" is voice data that allows a user to communicate questions or instructions to the system.

[0008] "Speech recognition means" refers to a function or device for analyzing voice input and converting it into text data.

[0009] "Text conversion" is a process of converting voice input into data in the form of a string of characters using a voice recognition means.

[0010] A "natural language processing engine" is software or a service that analyzes user intent and content from acquired text data.

[0011] "Intent analysis" is the process by which a natural language processing engine understands a user's intent from text data.

[0012] A "database" is a data store that contains information, documents, quotes, etc. about historical figures.

[0013] "Database search" refers to retrieving appropriate information from a database based on the results of intent analysis.

[0014] The "answer generation means" is a function that generates a reply to the user in text format based on information obtained from the database.

[0015] "Audio data" is data obtained by converting text-format information into audio format.

[0016] "Speech synthesis means" refers to a technique or device for converting text data into speech data.

[0017] The "playback means" refers to a speaker or other audio device that allows the user to hear the audio data.

[0018] A "historical figure" is a famous person who had a significant influence in the past, and is also called a great person.

[0019] The term "system" refers to the entire device or software that includes all means and functions related to the present invention. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] System Overview

[0042] The present invention is a system that allows users to interactively learn information about a historical figure they specify. This system includes means for the user to input voice, convert the voice input into text data, and perform analysis, database search, answer generation, speech synthesis, and playback using a natural language processing engine. The specific processing of each subsystem is described below.

[0043] Process Overview

[0044] 1. User Input

[0045] Users can speak a question into their smartphone or smart speaker, for example, "Socrates, what is true knowledge?"

[0046] 2. Voice Recognition

[0047] The device captures the user's voice input with a microphone and converts the speech into text data. Cloud services such as Google® Speech-to-Text API and Amazon Transcribe can be used for speech recognition. The converted text data will read, "Socrates, what is true knowledge?"

[0048] 3. Text Processing

[0049] The device sends the text data obtained by speech recognition to a natural language processing (NLP) engine, which analyzes the user's intent. The NLP engine extracts keywords and contextual information from the text data and generates structured data. For example, it generates structured data such as "Socrates (great person)" and "true knowledge (question content)."

[0050] 4. Database Search

[0051] The server receives the structured data and searches a database containing information about great people, retrieving, for example, a famous quote by Socrates: "All I know is that I know nothing."

[0052] 5. Answer generation

[0053] The server uses information from the database to generate an appropriate response for the user, imitating the tone and style of a great man, resulting in text of the form "All I know is that I know nothing."

[0054] 6. Speech Synthesis

[0055] The device receives text data from the server and sends it to a speech synthesis engine, which converts it into voice data. Services such as Google Text-to-Speech and Amazon Polly can be used for voice synthesis. The generated voice data sounds like Socrates is actually speaking.

[0056] 7. Responding to the User

[0057] The device then plays the generated audio data back to the user, who can hear a response such as, "All I know is that I know nothing." This process allows the user to interact with historical figures, providing a rich learning experience.

[0058] Specific examples

[0059] Example 1: Dialogue with Socrates

[0060] 1. User Input: The user asks, "Socrates, what is true knowledge?"

[0061] 2. Speech recognition: The device converts speech into text.

[0062] 3. Text processing: The device sends the text to a natural language processing engine to analyze the intent.

[0063] 4. Database lookup: The server searches the database and retrieves the appropriate information.

[0064] 5. Answer Generation: The server generates the answer.

[0065] 6. Speech synthesis: The device converts the generated text into audio data.

[0066] 7. Respond to the user: The terminal plays the audio data to the user.

[0067] In this way, the system provides users with an effective and interactive learning environment by linking together the various means. This invention enables users to acquire in-depth knowledge about historical figures in an interactive format.

[0068] The processing flow will be explained below.

[0069] Step 1:

[0070] Users speak into their smartphone or smart speaker to ask a question to a designated historical figure, for example, "Socrates, what is true knowledge?"

[0071] Step 2:

[0072] The device captures voice input from the user with a microphone and sends this voice data to a speech recognition module, which converts the voice data into text. The converted text is "Socrates, what is true knowledge?"

[0073] Step 3:

[0074] The device sends the text data obtained by speech recognition to a natural language processing engine. The natural language processing engine extracts keywords and contextual information from the text data and analyzes the user's intent. For example, it generates structured data such as "Socrates (great person)" and "true knowledge (question content)."

[0075] Step 4:

[0076] The terminal transmits the parsed structured data to the server.

[0077] Step 5:

[0078] The server receives the structured data and searches a database containing information about great people, such as Socrates' famous quote, "All I know is that I know nothing."

[0079] Step 6:

[0080] The server generates a textual response to the user based on the information retrieved from the database, reflecting the tone and style of the great man.

[0081] Step 7:

[0082] The server returns the generated text data to the terminal.

[0083] Step 8:

[0084] The device then sends the received text data to a speech synthesis engine, which converts it into voice data. The speech synthesis engine uses Google Text-to-Speech or Amazon Polly to convert the text into natural-sounding speech.

[0085] Step 9:

[0086] The terminal plays the generated voice data to the user, who can hear the answer, for example, "All I know is that I know nothing."

[0087] These are the specific steps the system takes to provide information about historical figures in response to user voice questions. This process allows the user to learn through dialogue with the designated historical figures.

[0088] Example 1

[0089] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0090] In conventional systems, users must rely on searching for information in books or websites when learning about historical figures, which does not provide an interactive learning experience. Furthermore, users without specialized knowledge have difficulty assessing the authenticity and appropriate interpretation of the information. Furthermore, while systems that use voice input and voice output are common, few systems effectively combine advanced natural language processing and database search.

[0091] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0092] In this invention, the server includes means for receiving voice input from a user, means for converting the voice input into text, means for sending the converted text to a natural language processing engine and analyzing the user's intent, means for searching an information storage area and retrieving corresponding information, means having a generative AI model for generating answers based on the retrieved information, means for converting the generated answers into voice data, and means for playing the voice data to the user, thereby enabling the user to interactively converse with historical figures, obtain reliable information via voice, and enjoy a rich learning experience.

[0093] "Voice input" is a method by which a user communicates information or instructions to a system via voice.

[0094] "Convert to text" is the process of converting audio data into text data.

[0095] A "natural language processing engine" is a computer program that analyzes text data and understands its meaning.

[0096] "Information storage area" refers to the database or storage system where data is saved.

[0097] A "generative AI model" is an artificial intelligence model that generates appropriate answers to users based on acquired information.

[0098] "Audio Data" means data digitally represented in audio format.

[0099] "Playing to the user" refers to the process of making the generated audio data audible to the user through an output device such as a speaker.

[0100] "Distributed delivery service" refers to a cloud service that is delivered over the Internet and that users can access from multiple locations.

[0101] System Overview

[0102] The present invention provides a system that allows a user to interactively learn information about an important person designated by the user. The system includes means for speech input, text conversion, natural language processing, database search, answer generation, speech synthesis, and speech playback.

[0103] Program processing details

[0104] User Input

[0105] The user uses a voice input device (e.g., a smartphone or a smart speaker) to input a question by voice, for example, saying, "Tell me about a particular important person."

[0106] Voice Recognition

[0107] The device captures the user's voice input through a microphone and converts it into text using a speech recognition service such as the Google Speech-to-Text API or Amazon Transcribe. For example, the speech might be converted into text like, "Tell me about a specific important person."

[0108] Text Processing

[0109] The device sends the text data obtained by speech recognition to a natural language processing engine. The natural language processing engine (for example, Google Cloud Natural Language API or IBM Watson (registered trademark) Natural Language Understanding) is used to analyze the text data and understand the user's intent. The analysis results in data such as "specific important person (target)" and "please tell me (request)."

[0110] Database search

[0111] The server receives the generated analysis data and searches an information storage area (for example, a MySQL (registered trademark) or PostgreSQL database). As a search result, information about a specific important person is obtained. For example, famous quotes and life information about the specific important person are obtained.

[0112] Answer generation

[0113] The server generates an appropriate answer for the user based on the information retrieved from the database. It uses a generative AI model (such as OpenAI's GPT-3 or BERT) to generate a text answer based on the retrieved information. For example, it generates text in the format "A certain important person left behind the famous quote XX."

[0114] Speech synthesis

[0115] The device receives text data from the server, sends it to a speech synthesis engine, and converts it into voice data. Services such as Google Text-to-Speech and Amazon Polly are used for voice synthesis. The generated voice data is synthesized to sound like a specific important person is actually speaking.

[0116] Responding to the user

[0117] The device then plays the generated voice data back to the user, who can hear a voice response such as, "A particular important person left behind the famous quote XX." Through this process, the user can interactively learn about important people.

[0118] Specific examples

[0119] Example 1: Interaction with a specific important person

[0120] 1. User Input: The user asks, "Tell me about a particular important person."

[0121] 2. Speech recognition: The device converts the speech into text using the Google Speech-to-Text API. The resulting text is "Tell me about a specific important person."

[0122] 3. Text processing: The device sends the text to the Google Cloud Natural Language API, which analyzes the intent and generates structured data.

[0123] 4. Database lookup: The server uses MySQL to lookup the database and retrieve information about a particular VIP.

[0124] 5. Answer generation: The server uses GPT-3 to generate an answer, creating the text "A certain important person left behind the famous quote XX."

[0125] 6. Text-to-speech: Your device will convert text to speech using Google Text-to-Speech.

[0126] 7. Response to the user: The terminal plays the audio data to the user, and the user hears, "A certain important person left behind the famous quote XX."

[0127] Prompt Sentence Examples

[0128] "Ask questions about important historical figures. For example, 'Tell me about a particular important person.'"

[0129] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0130] Step 1:

[0131] A user uses a voice input device (such as a smartphone or smart speaker) to input a question by voice. For example, they might say, "Tell me about a specific important person." At this stage, the input is voice data and has not yet been converted into text data.

[0132] Step 2:

[0133] The device captures the user's voice input through a microphone. It uses a voice recognition service (for example, Google Speech-to-Text API) to convert the voice data into text data. Specifically, it sends the voice data to a cloud service and receives the returned text data. The input is voice data, and the output is text data, such as "Tell me about a specific important person."

[0134] Step 3:

[0135] The device sends the acquired text data to a natural language processing engine (for example, Google Cloud Natural Language API) for analysis. Specifically, it performs syntactic analysis of the text and extracts keywords to understand the user's intent. The input is text data, and the output is data indicating the intent, such as "specific important person (target)" and "please tell me (request)."

[0136] Step 4:

[0137] The server receives the intent data sent from the device and searches an information storage area (for example, a MySQL database). It generates a search query and queries the database to obtain information about a specific important person. The specific operation is to execute an SQL query and extract the relevant records. The input is the intent data, and the output is information data as a search result.

[0138] Step 5:

[0139] The server generates an answer using a generative AI model (e.g., GPT-3) based on the information data obtained from the search results. Specifically, the process involves inputting the information data into the AI ​​model and outputting a corresponding text answer. The input is the information data, and the output is a text answer such as "A certain important person left behind the famous quote XX."

[0140] Step 6:

[0141] The device sends the answer text received from the server to a speech synthesis engine (e.g., Google Text-to-Speech) and converts it into voice data. During this process, it calls an API to convert text data into voice format and receives the result. The input is the answer text, and the output is voice data.

[0142] Step 7:

[0143] The terminal plays the generated voice data to the user. At this stage, the speaker is used to output the voice and convey the answer to the user. The specific operation is to pass the voice data to the playback device and output it. The input is the voice data, and the output is the voice information that the user hears.

[0144] (Application example 1)

[0145] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0146] Conventional learning systems have made it difficult for users to learn detailed information about historical figures of interest through a direct dialogue. Furthermore, users cannot easily obtain the information they are looking for through voice, making it difficult to provide an interactive learning experience. Furthermore, there are limitations to the accuracy and realism of the generated answers, and improvements are needed to increase user satisfaction.

[0147] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0148] In this invention, the server includes means for receiving voice input from a user, means for converting the voice input into text, means for sending the converted text to a natural language processing engine and analyzing the user's intent, means for searching a database based on the analyzed intent and obtaining corresponding information, means for generating an answer based on the obtained information, means for converting the generated answer into voice data, means for playing the voice data to the user, means for displaying the generated answer as text data, and means for generating an answer in the form of a dialogue between a predetermined specific person. This allows a user to easily learn in the form of a dialogue with a historical figure through voice input, thereby providing a highly realistic, interactive learning experience.

[0149] A "means for receiving voice input from a user" is a device capable of capturing voice data such as questions or commands uttered by a user.

[0150] The "means for converting voice input into text" is a technology that converts captured voice data into text information, and utilizes voice recognition technology.

[0151] "Means of sending the converted text to a natural language processing engine and analyzing the user's intent" is a technology that uses advanced algorithms to analyze text data to understand the intent of a user's questions or commands.

[0152] The "means for searching a database based on the analyzed intent and retrieving corresponding information" is a system that has the function of searching and retrieving highly relevant information from within a database.

[0153] The "means for generating an answer based on acquired information" is a system that has the function of constructing an appropriate answer to a user's question based on information acquired from a database.

[0154] The "means for converting the generated answer into voice data" is a technique for converting the generated text-format answer into data that can be reproduced by voice.

[0155] The "means for reproducing the audio data to the user" refers to a reproducing device for allowing the user to listen to the converted audio data.

[0156] The "means for displaying the generated answer as text data" is a device that has the function of displaying the answer not only as voice but also as text on a screen.

[0157] "Means in which the generated answers are in the form of a dialogue by a specific person" refers to a technology in which the answers are likened to a specific historical figure and are provided in the same tone and style as that figure.

[0158] System Overview

[0159] The system that realizes this application example integrates the functions of voice input, voice recognition, natural language processing, database search, answer generation, voice synthesis, result playback, and text display. The specific processing of each subsystem is explained below.

[0160] Hardware and software used

[0161] Hardware:

[0162] microphone

[0163] speaker

[0164] Displays (smartphones, etc.)

[0165] High-performance devices (smartphones, servers)

[0166] software:

[0167] Python: A programming language for implementing the overall logic of a program

[0168] speech_recognition package: A library for converting voice input to text using the Google Speech-to-Text API.

[0169] transformers package: A library for analyzing user intent and generating appropriate answers using Hugging Face's natural language processing model.

[0170] gTTS (Google Text-to-Speech): A library for converting generated text into audio data.

[0171] playsound: A library for playing converted audio data

[0172] Details of data processing and calculation

[0173] 1. Voice Input and Recognition

[0174] The user inputs a question by voice through the microphone. For example, in the scenario of "asking a question to a specific historical figure," the user might say, "Historical figure, please tell me your famous quote." This is captured by the microphone and converted into text data using the speech_recognition package.

[0175] 2. Natural Language Processing

[0176] The device sends the converted text to a natural language processing engine, which analyzes the user's question, extracts keywords and contextual information, and generates structured data using Hugging Face's transformers package.

[0177] 3. Database search and answer generation

[0178] The server receives the parsed structured data and searches for relevant information in a database. Based on the information obtained, it generates an appropriate answer to the user's question. In this process, the generated text is modeled after the tone and writing style of famous people, so that the answer is provided in a conversational style that is specific to the person.

[0179] 4. Speech synthesis and playback

[0180] The device sends the generated text data to a speech synthesis engine and converts it into voice data. This process uses the gTTS library. The generated voice data is played back to the user using the playsound library, and is also displayed as text data on the screen.

[0181] Specific examples

[0182] For example, consider the case where a user launches an application and asks, "Napoleon, tell me about the importance of strategy." In this case, the system performs speech recognition and converts the speech into text data. It then uses a natural language processing engine to analyze the question and retrieves an appropriate answer about Napoleon from a database. The answer generated, "Without strategy, no army can achieve victory," is converted into audio data and played back to the user. This entire process occurs in real time, providing the user with an interactive learning experience.

[0183] Prompt Sentence Examples

[0184] To generate a question for a particular historical figure, a prompt like the following can be used:

[0185] Prompt: "Generate a dialogue about the importance of strategy for the historical figure Napoleon."

[0186] Thus, the present invention utilizes advanced speech recognition and natural language processing techniques to provide a system that allows users to interactively learn about historical figures, thereby providing a richer learning experience for users.

[0187] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0188] Step 1:

[0189] Capture audio input

[0190] The user can enter questions or commands by voice through a microphone. The user's voice is then captured as an analog signal by the microphone. For example, a question such as "What is your most famous quote, historical figure?" can be captured as user input.

[0191] Step 2:

[0192] Speech-to-text conversion

[0193] The device converts the captured voice data to text using the speech_recognition package. In this process, speech recognition technology is applied to digitize the voice signal, which is then converted to text data. The input is voice data, and the output is the corresponding text data.

[0194] Step 3:

[0195] Intention analysis using natural language processing

[0196] The device sends the converted text data to a natural language processing engine, which uses the transformers package to analyze the user's intent. Specifically, it extracts keywords and contextual information from the text data and generates structured data. The input of this step is the text data, and the output is the analyzed structured data.

[0197] Step 4:

[0198] Database search

[0199] The server receives the parsed structured data and uses it to search a database containing information about historical figures, and executes queries to retrieve relevant information. The input is the structured data, and the output is relevant information.

[0200] Step 5:

[0201] Answer generation

[0202] The server generates appropriate answers for the user based on information retrieved from the database. It uses a generative AI model to generate dialogue-style answers from specific historical figures. The generated answers mimic the tone and writing style of the specific figures. The input is the retrieved information, and the output is the generated text answer.

[0203] Step 6:

[0204] Conversion to audio data

[0205] The device converts the generated text data into speech data using the gTTS library. The text is sent to a speech synthesis engine, which generates corresponding speech data. The input is the generated text data, and the output is speech data.

[0206] Step 7:

[0207] Playback of audio data and display of text

[0208] The device plays the generated audio data using the playsound library and simultaneously displays it as text data on the display. In this step, the user hears the answer aloud and the text is also displayed on the screen. The input is audio data and text data, and the output is audio playback and text display.

[0209] This series of processes allows users to ask questions about historical figures by voice and receive answers in audio and text, providing an interactive and realistic learning experience.

[0210] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0211] System Overview

[0212] This invention is a system that allows users to voice-input questions to historical figures specified by the user and provides answers to those questions in an interactive format. This system converts the voice input into text, analyzes the intent of the question using a natural language processing engine, retrieves information from a database, generates answers, and plays them back to the user in voice. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it achieves more human-like dialogue.

[0213] Process Overview

[0214] 1. User Input

[0215] Users speak questions into their smartphones or smart speakers, for example, "Socrates, what is true knowledge?"

[0216] 2. Voice Recognition

[0217] The device captures voice input from the user and sends this voice data to a voice recognition module, which converts the voice data into text data. The converted text might be, for example, "Socrates, what is true knowledge?"

[0218] 3. Text Processing

[0219] The device sends the text data obtained by speech recognition to a natural language processing engine, which analyzes the user's intent. The natural language processing engine then analyzes the text data for keywords and contextual information to generate structured data.

[0220] 4. Emotion analysis

[0221] The device sends text data and voice data to the emotion engine, which analyzes the user's emotions. The emotion engine identifies the user's emotions from the tone of the voice and the content of the text. For example, it analyzes the user's emotional state, such as whether they are excited, sad, or calm.

[0222] 5. Database Search

[0223] The server receives information from the natural language processing engine and emotion engine and searches a database containing information about great people. For example, it retrieves a famous quote by Socrates: "All I know is that I know nothing."

[0224] 6. Answer generation

[0225] The server generates an appropriate response to the user based on the information retrieved from the database and the analysis results of the emotion engine. For example, if the user is excited, it generates a response in a calm tone.

[0226] 7. Speech Synthesis

[0227] The device converts the generated text data into voice data. The speech synthesis engine converts the generated text into natural-sounding speech and adjusts the tone based on the results of emotion analysis. For example, if the user is sad, the device will generate a gentler tone of voice.

[0228] 8. Responding to the User

[0229] The terminal plays the generated voice data to the user, who can hear a response such as "All I know is that I know nothing" spoken in an appropriate emotional tone.

[0230] Specific examples

[0231] Example 1: Dialogue with Socrates

[0232] 1. User Input: The user asks, "Socrates, what is true knowledge?"

[0233] 2. Speech recognition: The device converts speech into text.

[0234] 3. Text processing: The device sends the text to a natural language processing engine to analyze the intent.

[0235] 4. Emotion analysis: The device analyzes the user's emotions. For example, it determines that the user is calm.

[0236] 5. Database lookup: The server searches the database and retrieves the appropriate information.

[0237] 6. Answer generation: The server generates an answer in a calm tone based on the results of sentiment analysis.

[0238] 7. Speech synthesis: The device converts the generated text into audio data.

[0239] 8. Response to the user: The terminal plays back in a calm tone, "All I know is that I know nothing."

[0240] In this way, each of the methods in this system works together to provide users with an effective and interactive learning environment. Furthermore, by combining it with an emotion engine, flexible responses based on the user's emotions become possible, resulting in more human-like dialogue.

[0241] The processing flow will be explained below.

[0242] Step 1:

[0243] Users can speak into their smartphone or smart speaker to ask questions to historical figures, such as, "Socrates, what is true knowledge?"

[0244] Step 2:

[0245] The device captures voice input from the user with a microphone and sends this voice data to a voice recognition module. The voice recognition module converts the voice data into text data. The converted text becomes, "Socrates, what is true knowledge?"

[0246] Step 3:

[0247] The device sends the text data obtained by speech recognition to a natural language processing (NLP) engine. The NLP engine extracts keywords and contextual information from the text data and analyzes the user's intent. For example, it generates structured data such as "Socrates (great man)" and "true knowledge (question content)."

[0248] Step 4:

[0249] The device sends the text data and voice data to the emotion engine, which analyzes the user's emotion from the voice tone and text content. For example, it identifies the user's emotional state, such as calm, excited, or sad.

[0250] Step 5:

[0251] The device transmits the intent-analyzed structured data and the results of emotion analysis to the server.

[0252] Step 6:

[0253] The server receives the structured data and searches a database containing information about great people. For example, based on the keywords "Socrates" and "true knowledge," it retrieves Socrates' famous quote, "All I know is that I know nothing."

[0254] Step 7:

[0255] The server generates appropriate responses to the user based on the information retrieved from the database and the results of emotion analysis. For example, if the server determines that the user is calm, it generates responses that mimic the tone of Socrates, who has a calm mind.

[0256] Step 8:

[0257] The server transmits the generated text data to the terminal.

[0258] Step 9:

[0259] The device sends the text data received from the server to a speech synthesis engine, which converts the text data into speech data and adjusts the tone of the speech based on the results of emotion analysis. For example, if the user is sad, the device generates a response in a gentle tone.

[0260] Step 10:

[0261] The terminal plays the generated voice data to the user, who can hear a response such as "All I know is that I know nothing" spoken in an appropriate emotional tone.

[0262] These are the specific processing steps required to provide information about historical figures in response to a user's voice question. This process allows the user to learn through a dialogue with the designated historical figure, and by combining it with an emotion engine, it is possible to achieve a more human-like dialogue.

[0263] Example 2

[0264] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0265] In modern learning systems, it is difficult for users to ask historical figures questions and receive interactive, emotionally relevant answers. Traditional systems often provide cold responses, which can discourage users from learning. Furthermore, the quality of voice-based engines and the accuracy of natural language processing are limited, leaving issues with the accuracy and persuasiveness of the information.

[0266] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving a voice input, means for converting the voice input into text, means for sending the converted text to a natural language processing engine and analyzing the intention, means for searching a database based on the analyzed intention and acquiring corresponding information, means for analyzing the user's emotions using an emotion analysis engine, means for generating an answer based on the acquired information and the analyzed emotion information, means for converting the generated answer into voice data, and means for playing the voice data to the user. This makes it possible to provide an interactive answer according to the user's emotions toward the historical figure about whom the question was asked.

[0267] "Means for receiving voice input" is a general term for devices and modules that capture voices uttered by the user and input them into the system.

[0268] "Means for converting voice input to text" refers to a device or module for analyzing captured voice data and converting it into corresponding text data.

[0269] A "natural language processing engine" is a general term for software or systems that analyze text data and extract user intent and keywords.

[0270] "Means for analyzing intent" refers to a device or module that uses a natural language processing engine to perform processing to analyze the intent of a question from input text data.

[0271] "Means for searching a database and retrieving corresponding information" refers to a device or module for searching related information in a database based on the analyzed intent and retrieving the required information.

[0272] "Emotion analysis engine" is a general term for software or systems that analyze a user's voice or text data and identify the user's emotional state.

[0273] "Means for analyzing emotions" refers to a device or module that uses an emotion analysis engine to perform processing to analyze the user's emotional state.

[0274] The "means for generating an answer" refers to a device or module for generating an appropriate answer to the user based on the acquired information and emotional information.

[0275] "Means for converting generated answers into voice data" refers to a device or module for converting generated text answers into voice data.

[0276] "Means for playing back audio data to the user" refers to a device or module that plays back the generated audio data in order to deliver it to the user.

[0277] "Biographical information" is a general term for data, facts, quotes, and other information relating to historical or famous figures.

[0278] "Cloud-based services" refer to remote computing resources, data storage, and software services delivered over the internet.

[0279] MODE FOR CARRYING OUT THE INVENTION

[0280] System Overview

[0281] This invention is an interactive learning system that allows users to ask questions using voice input and provides emotionally appropriate answers based on information about the historical figures in question. The system converts the voice input into text, analyzes the user's intention using a natural language processing engine, and retrieves corresponding information from a database. It also has the function of analyzing the user's emotions using an emotion analysis engine, generating answers in an appropriate tone, and playing them back as voice.

[0282] Hardware and software used

[0283] 1. Hardware:

[0284] Smartphones and smart speakers: Devices for voice input and playback.

[0285] Server: A central processing unit that searches the database and generates answers.

[0286] 2. Software:

[0287] Speech Recognition Module: Converts voice data into text data using the Google Speech-to-Text API.

[0288] Natural Language Processing Engine: OpenAI GPT-3 is used to analyze the intent of the text.

[0289] Sentiment Analysis Engine: Analyzes user emotions using IBM Watson Tone Analyzer.

[0290] Speech synthesis engine: Generates speech from text using Amazon Polly.

[0291] Processing flow

[0292] The specific operations of each processing step of this system are shown below.

[0293] 1. User Input

[0294] Users speak questions into their smartphones or smart speakers, for example, "Socrates, what is true knowledge?"

[0295] 2. Voice Recognition

[0296] The device captures the user's voice input and converts it into text using the Google Speech-to-Text API. Specifically, the speech "Socrates, what is true knowledge?" is converted into text "Socrates, what is true knowledge?"

[0297] 3. Text Processing

[0298] The device then sends the converted text to OpenAI GPT-3 to analyze the user's intent, during which the system extracts the keywords "Socrates" and "knowledge" to understand the gist of the question.

[0299] 4. Emotion analysis

[0300] The device sends text and voice data to the IBM Watson Tone Analyzer, which analyzes the user's emotions, identifying whether the user is calm, excited, sad, or other such emotions.

[0301] 5. Database Search

[0302] The server receives information from the natural language processing engine and emotion engine and searches a database of people, such as the quote "All I know is that I know nothing" for "Socrates." The database uses a cloud-based MySQL database.

[0303] 6. Answer generation

[0304] The server generates an appropriate response to the user based on the acquired information and the analyzed emotional information. For a calm user, the server generates a response in a calm tone.

[0305] 7. Speech Synthesis

[0306] The device converts the generated text into speech using Amazon Polly, a speech synthesis engine that adjusts tone based on sentiment analysis.

[0307] 8. Responding to the User

[0308] The terminal plays the generated voice data back to the user. For example, the user can hear a calm response such as "All I know is that I know nothing."

[0309] Specific examples

[0310] Example 1: Dialogue with Socrates

[0311] If a user asks, "Socrates, what is true knowledge?" the system behaves as follows:

[0312] 1. Speech is converted into text using speech recognition.

[0313] 2. The natural language processing engine analyzes the intent.

[0314] 3. The sentiment is analyzed by the sentiment analysis engine.

[0315] 4. The corresponding information is retrieved from the database.

[0316] 5. The answer is generated.

[0317] 6. It is converted into audio and played back to the user.

[0318] Example prompts for generative AI models

[0319] "Show me what your system would do if a user calmly asked, 'Socrates, what is true knowledge?'"

[0320] The system allows users to receive real-time emotional responses, creating a more interactive and effective learning experience.

[0321] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0322] Step 1:

[0323] Users speak questions into their smartphone or smart speaker, such as "Socrates, what is true knowledge?" This voice input is captured by the device's microphone.

[0324] Step 2:

[0325] The device sends the captured voice data to the speech recognition module, which uses the Google Speech-to-Text API. Specifically, the voice signal is sent to the server, which converts it into text data via the API: "Socrates, what is true knowledge?" The input is the user's voice, and the output is the corresponding text data.

[0326] Step 3:

[0327] The device sends the converted text data to a natural language processing engine (e.g., OpenAI GPT-3). The natural language processing engine extracts important keywords and contextual information from the text data and analyzes the user's intent. The input here is the converted text data, and the output is the analyzed user's intent information. Specifically, the keywords "Socrates" and "knowledge" are extracted from the text "Socrates, what is true knowledge?"

[0328] Step 4:

[0329] The device simultaneously sends text and voice data to an emotion analysis engine (e.g., IBM Watson Tone Analyzer). The emotion analysis engine analyzes the tone of the voice and the content of the text to identify the user's emotional state. The input is text and voice data, and the output is the user's emotional information. Specific emotions that can be identified include calm, excitement, and sadness. In this example, the user is analyzed as calm.

[0330] Step 5:

[0331] The server receives information from the natural language processing engine and sentiment analysis engine and searches the database. The database uses a cloud-based MySQL database. In this step, related information in the database is searched for based on the analyzed keywords "Socrates" and "knowledge" to obtain the appropriate answer. The input is the analyzed keywords and sentiment information, and the output is the obtained information. For example, the search result obtained is "All I know is that I know nothing."

[0332] Step 6:

[0333] The server generates an appropriate answer for the user based on the acquired information and emotional information. Since the answer needs to be in a calm tone, the server generates a calm tone text such as "All I know is that I know nothing." The input is the acquired information and emotional information, and the output is the generated answer text.

[0334] Step 7:

[0335] The device converts the generated text data into voice data. Amazon Polly is used here. The voice synthesis engine adjusts the tone of the voice based on the results of emotion analysis. The input is the generated text data, and the output is voice data with a tone that corresponds to the emotion. Specifically, voice is generated in a calm tone.

[0336] Step 8:

[0337] The terminal plays the generated voice data to the user. The voice played is a calm response saying, "All I know is that I know nothing." The user can hear this aloud. The input is the generated voice data, and the output is the voice played to the user.

[0338] (Application example 2)

[0339] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0340] In traditional brick-and-mortar stores, it has been difficult to provide accurate and emotionally appropriate information to customer questions. Furthermore, when historical background or detailed knowledge is required, the lack of knowledge and inconsistency of responses from store staff can be problematic. This can lead to a poor customer experience.

[0341] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0342] In this invention, the server includes means for analyzing the user's emotions, means for referencing the user's emotion data when generating the answer, and means that is installed in the guide robot and provides services to customers in the physical store. This enables flexible and accurate responses according to the customer's emotions and realizes the provision of consistent and detailed information in response to the question.

[0343] "Voice input" is a means by which a user provides data to a system using speech.

[0344] "Text conversion" is the process of converting audio data into text data.

[0345] A "natural language processing engine" is software that performs semantic analysis on text data and understands the user's intent.

[0346] "Database search" is an operation for searching information in a database based on the analyzed intent and obtaining the required data.

[0347] "Answer generation" is the process of creating an appropriate answer to a user's question based on the acquired information.

[0348] "Audio data conversion" is the process of converting the generated text data into an audio file and letting the user listen to it.

[0349] "Emotion analysis" is the process of evaluating and determining a user's emotional state from their voice or text.

[0350] "Response playback" is the process of playing back the generated voice data to the user.

[0351] A "guide robot" is a robot that is installed in physical stores and other locations to act as a guide and assist customers.

[0352] A "brick and mortar store" is a store that offers goods or services in a physical location.

[0353] This invention describes an embodiment of a system that allows a user to voice-input a question to a historical figure designated by the user and provides an answer to that question in an interactive format. This system has the functions of converting the voice input into text, analyzing the intent of the question using a natural language processing engine, retrieving information from a database, generating an answer, and playing it back to the user in voice. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to achieve more human-like dialogue.

[0354] Hardware and software used

[0355] Hardware:

[0356] Microphone (for voice input)

[0357] Speaker (for audio output)

[0358] Guidance robots (installed in brick-and-mortar stores for customer service)

[0359] software:

[0360] speech_recognition library (speech to text conversion)

[0361] pyttsx3 library (text-to-speech synthesis)

[0362] QA model (question answering) from the transformers library

[0363] emotion_recognition library (emotion analysis)

[0364] Processing Overview

[0365] 1. User Input:

[0366] The user can input questions to the guide robot by voice through a microphone. For example, the user can ask, "What time does this store open?"

[0367] 2. Speech Recognition:

[0368] The device captures the user's voice input and uses the speech_recognition library to convert the voice data into text, such as "What time does this store open?"

[0369] 3. Text Analysis:

[0370] The device sends the text data obtained by speech recognition to the natural language processing engine of the transformers library to analyze the user's intent. The natural language processing engine analyzes keywords and contextual information from the text data and generates structured data.

[0371] 4. Emotion analysis:

[0372] The device sends text and voice data to the emotion_recognition library to analyze the user's emotions. The emotion engine identifies the user's emotions from the tone of the voice and the content of the text. For example, it analyzes that the user is calm.

[0373] 5. Database Search:

[0374] The server receives information from the natural language processing engine and emotion engine and searches a database containing information about historical figures, obtaining information such as "This store has been in business since 1950."

[0375] 6. Answer generation:

[0376] The server generates an appropriate response to the user based on the acquired information and the analysis results of the emotion engine. For example, if the user is calm, it generates a response in a normal tone.

[0377] 7. Speech synthesis:

[0378] The device converts the generated text data into speech data, using the pyttsx3 library to convert the generated text into natural-sounding speech and adjust the tone based on the results of sentiment analysis.

[0379] 8. Response to the user:

[0380] The device plays the generated voice data to the user through a speaker, and the user can hear a response such as "This store has been in business since 1950" in an appropriate emotional tone.

[0381] Examples of prompt statements

[0382] For example, if a customer visits a store with their child and asks the robot, "What time does this store open?", the following prompt sentence will be generated:

[0383] User: How long has this store been open?

[0384] Emotion: neutral

[0385] A: This store has been in business since 1950.

[0386] This system will improve the customer experience in-store and enable accurate and flexible information provision.

[0387] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0388] Step 1:

[0389] User Input

[0390] The user inputs a question to the guide robot by voice through a microphone. For example, "What time does this store open?" The input of this step is the user's voice, and the output is voice data.

[0391] Step 2:

[0392] Voice Recognition

[0393] The device captures the user's voice input and converts it to text using the speech_recognition library. For example, the converted text might be, "What time does this store open?" The input for this step is voice data, and the output is text data.

[0394] Step 3:

[0395] Text analytics

[0396] The device sends the text data obtained by speech recognition to the natural language processing engine of the transformers library to analyze the user's intent. The natural language processing engine analyzes keywords and contextual information from the text data and generates structured data. The input of this step is text data, and the output is the analyzed structured data.

[0397] Step 4:

[0398] Emotion analysis

[0399] The device sends text and audio data to the emotion_recognition library to analyze the user's emotions. The emotion engine identifies the user's emotions from the tone of the voice and the content of the text. For example, it identifies emotions such as calm, excitement, and sadness. The input of this step is text and audio data, and the output is the user's emotional state.

[0400] Step 5:

[0401] Database search

[0402] The server receives information from the natural language processing engine and the emotion engine and searches a database containing information about historical figures. For example, it can obtain information such as "We have been in business since 1950." The input of this step is the parsed structured data and emotion data, and the output is information as search results.

[0403] Step 6:

[0404] Answer generation

[0405] The server generates an appropriate response to the user based on the acquired information and the analysis results of the emotion engine. For example, if the user is calm, it generates a response in a normal tone, such as "This store has been in business since 1950." The input for this step is the search result information and emotion data, and the output is the generated response text.

[0406] Step 7:

[0407] Speech synthesis

[0408] The device converts the generated text data into speech data using the pyttsx3 library. The tone is also adjusted based on the results of sentiment analysis. The input of this step is the response text, and the output is speech data.

[0409] Step 8:

[0410] Responding to the user

[0411] The terminal plays the generated voice data to the user through a speaker. The user can hear, for example, a response such as "This store has been in business since 1950" in an appropriate emotional tone. The input of this step is the voice data, and the output is audio information that the user can hear.

[0412] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0413] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0414] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0415] [Second embodiment]

[0416] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0417] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0418] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0419] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0420] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0421] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0422] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0423] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0424] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0425] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0426] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0427] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0428] System Overview

[0429] The present invention is a system that allows users to interactively learn information about a historical figure they specify. This system includes means for the user to input voice, convert the voice input into text data, and perform analysis, database search, answer generation, speech synthesis, and playback using a natural language processing engine. The specific processing of each subsystem is described below.

[0430] Process Overview

[0431] 1. User Input

[0432] Users can speak a question into their smartphone or smart speaker, for example, "Socrates, what is true knowledge?"

[0433] 2. Voice Recognition

[0434] The device captures the user's voice input with a microphone and converts the speech into text data. Cloud services such as the Google Speech-to-Text API and Amazon Transcribe can be used for speech recognition. The converted text data will read, "Socrates, what is true knowledge?"

[0435] 3. Text Processing

[0436] The device sends the text data obtained by speech recognition to a natural language processing (NLP) engine, which analyzes the user's intent. The NLP engine extracts keywords and contextual information from the text data and generates structured data. For example, it generates structured data such as "Socrates (great person)" and "true knowledge (question content)."

[0437] 4. Database Search

[0438] The server receives the structured data and searches a database containing information about great people, retrieving, for example, a famous quote by Socrates: "All I know is that I know nothing."

[0439] 5. Answer generation

[0440] The server uses information from the database to generate an appropriate response for the user, imitating the tone and style of a great man, resulting in text of the form "All I know is that I know nothing."

[0441] 6. Speech Synthesis

[0442] The device receives text data from the server and sends it to a speech synthesis engine, which converts it into voice data. Services such as Google Text-to-Speech and Amazon Polly can be used for voice synthesis. The generated voice data sounds like Socrates is actually speaking.

[0443] 7. Responding to the User

[0444] The device then plays the generated audio data back to the user, who can hear a response such as, "All I know is that I know nothing." This process allows the user to interact with historical figures, providing a rich learning experience.

[0445] Specific examples

[0446] Example 1: Dialogue with Socrates

[0447] 1. User Input: The user asks, "Socrates, what is true knowledge?"

[0448] 2. Speech recognition: The device converts speech into text.

[0449] 3. Text processing: The device sends the text to a natural language processing engine to analyze the intent.

[0450] 4. Database lookup: The server searches the database and retrieves the appropriate information.

[0451] 5. Answer Generation: The server generates the answer.

[0452] 6. Speech synthesis: The device converts the generated text into audio data.

[0453] 7. Respond to the user: The terminal plays the audio data to the user.

[0454] In this way, the system provides users with an effective and interactive learning environment by linking together the various means. This invention enables users to acquire in-depth knowledge about historical figures in an interactive format.

[0455] The processing flow will be explained below.

[0456] Step 1:

[0457] Users speak into their smartphone or smart speaker to ask a question to a designated historical figure, for example, "Socrates, what is true knowledge?"

[0458] Step 2:

[0459] The device captures voice input from the user with a microphone and sends this voice data to a speech recognition module, which converts the voice data into text. The converted text is "Socrates, what is true knowledge?"

[0460] Step 3:

[0461] The device sends the text data obtained by speech recognition to a natural language processing engine. The natural language processing engine extracts keywords and contextual information from the text data and analyzes the user's intent. For example, it generates structured data such as "Socrates (great person)" and "true knowledge (question content)."

[0462] Step 4:

[0463] The terminal transmits the parsed structured data to the server.

[0464] Step 5:

[0465] The server receives the structured data and searches a database containing information about great people, such as Socrates' famous quote, "All I know is that I know nothing."

[0466] Step 6:

[0467] The server generates a textual response to the user based on the information retrieved from the database, reflecting the tone and style of the great man.

[0468] Step 7:

[0469] The server returns the generated text data to the terminal.

[0470] Step 8:

[0471] The device then sends the received text data to a speech synthesis engine, which converts it into voice data. The speech synthesis engine uses Google Text-to-Speech or Amazon Polly to convert the text into natural-sounding speech.

[0472] Step 9:

[0473] The terminal plays the generated voice data to the user, who can hear the answer, for example, "All I know is that I know nothing."

[0474] These are the specific steps the system takes to provide information about historical figures in response to user voice questions. This process allows the user to learn through dialogue with the designated historical figures.

[0475] Example 1

[0476] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0477] In conventional systems, users must rely on searching for information in books or websites when learning about historical figures, which does not provide an interactive learning experience. Furthermore, users without specialized knowledge have difficulty assessing the authenticity and appropriate interpretation of the information. Furthermore, while systems that use voice input and voice output are common, few systems effectively combine advanced natural language processing and database search.

[0478] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0479] In this invention, the server includes means for receiving voice input from a user, means for converting the voice input into text, means for sending the converted text to a natural language processing engine and analyzing the user's intent, means for searching an information storage area and retrieving corresponding information, means having a generative AI model for generating answers based on the retrieved information, means for converting the generated answers into voice data, and means for playing the voice data to the user, thereby enabling the user to interactively converse with historical figures, obtain reliable information via voice, and enjoy a rich learning experience.

[0480] "Voice input" is a method by which a user communicates information or instructions to a system via voice.

[0481] "Convert to text" is the process of converting audio data into text data.

[0482] A "natural language processing engine" is a computer program that analyzes text data and understands its meaning.

[0483] "Information storage area" refers to the database or storage system where data is saved.

[0484] A "generative AI model" is an artificial intelligence model that generates appropriate answers to users based on acquired information.

[0485] "Audio Data" means data digitally represented in audio format.

[0486] "Playing to the user" refers to the process of making the generated audio data audible to the user through an output device such as a speaker.

[0487] "Distributed delivery service" refers to a cloud service that is delivered over the Internet and that users can access from multiple locations.

[0488] System Overview

[0489] The present invention provides a system that allows a user to interactively learn information about an important person designated by the user. The system includes means for speech input, text conversion, natural language processing, database search, answer generation, speech synthesis, and speech playback.

[0490] Program processing details

[0491] User Input

[0492] The user uses a voice input device (e.g., a smartphone or a smart speaker) to input a question by voice, for example, saying, "Tell me about a particular important person."

[0493] Voice Recognition

[0494] The device captures the user's voice input through a microphone and converts it into text using a speech recognition service such as the Google Speech-to-Text API or Amazon Transcribe. For example, the speech might be converted into text like, "Tell me about a specific important person."

[0495] Text Processing

[0496] The device sends the text data obtained by voice recognition to a natural language processing engine. The natural language processing engine (for example, Google Cloud Natural Language API or IBM Watson Natural Language Understanding) analyzes the text data and understands the user's intent. The analysis results in data such as "specific important person (target)" and "please tell me (request)."

[0497] Database search

[0498] The server receives the generated analysis data and searches an information storage area (for example, a MySQL or PostgreSQL database). As a result of the search, information about a specific important person is obtained. For example, famous quotes and life information about the specific important person are obtained.

[0499] Answer generation

[0500] The server generates an appropriate answer for the user based on the information retrieved from the database. It uses a generative AI model (such as OpenAI's GPT-3 or BERT) to generate a response text based on the retrieved information. For example, it generates text in the format "A certain important person left behind the famous quote XX."

[0501] Speech synthesis

[0502] The device receives text data from the server, sends it to a speech synthesis engine, and converts it into voice data. Services such as Google Text-to-Speech and Amazon Polly are used for voice synthesis. The generated voice data is synthesized to sound like a specific important person is actually speaking.

[0503] Responding to the user

[0504] The device then plays the generated voice data back to the user, who can hear a voice response such as, "A particular important person left behind the famous quote XX." Through this process, the user can interactively learn about important people.

[0505] Specific examples

[0506] Example 1: Interaction with a specific important person

[0507] 1. User Input: The user asks, "Tell me about a particular important person."

[0508] 2. Speech recognition: The device converts the speech into text using the Google Speech-to-Text API. The resulting text is "Tell me about a specific important person."

[0509] 3. Text processing: The device sends the text to the Google Cloud Natural Language API, which analyzes the intent and generates structured data.

[0510] 4. Database lookup: The server uses MySQL to lookup the database and retrieve information about a particular VIP.

[0511] 5. Answer generation: The server uses GPT-3 to generate an answer, creating the text "A certain important person left behind the famous quote XX."

[0512] 6. Text-to-speech: Your device will convert text to speech using Google Text-to-Speech.

[0513] 7. Response to the user: The terminal plays the audio data to the user, and the user hears, "A certain important person left behind the famous quote XX."

[0514] Prompt Sentence Examples

[0515] "Ask questions about important historical figures. For example, 'Tell me about a particular important person.'"

[0516] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0517] Step 1:

[0518] A user uses a voice input device (such as a smartphone or smart speaker) to input a question by voice. For example, they might say, "Tell me about a specific important person." At this stage, the input is voice data and has not yet been converted into text data.

[0519] Step 2:

[0520] The device captures the user's voice input through a microphone. It uses a voice recognition service (for example, Google Speech-to-Text API) to convert the voice data into text data. Specifically, it sends the voice data to a cloud service and receives the returned text data. The input is voice data, and the output is text data, such as "Tell me about a specific important person."

[0521] Step 3:

[0522] The device sends the acquired text data to a natural language processing engine (for example, Google Cloud Natural Language API) for analysis. Specifically, it performs syntactic analysis of the text and extracts keywords to understand the user's intent. The input is text data, and the output is data indicating the intent, such as "specific important person (target)" and "please tell me (request)."

[0523] Step 4:

[0524] The server receives the intent data sent from the device and searches an information storage area (for example, a MySQL database). It generates a search query and queries the database to obtain information about a specific important person. The specific operation is to execute an SQL query and extract the relevant records. The input is the intent data, and the output is information data as a search result.

[0525] Step 5:

[0526] The server generates an answer using a generative AI model (e.g., GPT-3) based on the information data obtained from the search results. Specifically, the process involves inputting the information data into the AI ​​model and outputting a corresponding text answer. The input is the information data, and the output is a text answer such as "A certain important person left behind the famous quote XX."

[0527] Step 6:

[0528] The device sends the answer text received from the server to a speech synthesis engine (e.g., Google Text-to-Speech) and converts it into voice data. During this process, it calls an API to convert text data into voice format and receives the result. The input is the answer text, and the output is voice data.

[0529] Step 7:

[0530] The terminal plays the generated voice data to the user. At this stage, the speaker is used to output the voice and convey the answer to the user. The specific operation is to pass the voice data to the playback device and output it. The input is the voice data, and the output is the voice information that the user hears.

[0531] (Application example 1)

[0532] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0533] Conventional learning systems have made it difficult for users to learn detailed information about historical figures of interest through a direct dialogue. Furthermore, users cannot easily obtain the information they are looking for through voice, making it difficult to provide an interactive learning experience. Furthermore, there are limitations to the accuracy and realism of the generated answers, and improvements are needed to increase user satisfaction.

[0534] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0535] In this invention, the server includes means for receiving voice input from a user, means for converting the voice input into text, means for sending the converted text to a natural language processing engine and analyzing the user's intent, means for searching a database based on the analyzed intent and obtaining corresponding information, means for generating an answer based on the obtained information, means for converting the generated answer into voice data, means for playing the voice data to the user, means for displaying the generated answer as text data, and means for generating an answer in the form of a dialogue between a predetermined specific person. This allows a user to easily learn in the form of a dialogue with a historical figure through voice input, thereby providing a highly realistic, interactive learning experience.

[0536] A "means for receiving voice input from a user" is a device capable of capturing voice data such as questions or commands uttered by a user.

[0537] The "means for converting voice input into text" is a technology that converts captured voice data into text information, and utilizes voice recognition technology.

[0538] "Means of sending the converted text to a natural language processing engine and analyzing the user's intent" is a technology that uses advanced algorithms to analyze text data to understand the intent of a user's questions or commands.

[0539] The "means for searching a database based on the analyzed intent and retrieving corresponding information" is a system that has the function of searching and retrieving highly relevant information from within a database.

[0540] The "means for generating an answer based on acquired information" is a system that has the function of constructing an appropriate answer to a user's question based on information acquired from a database.

[0541] The "means for converting the generated answer into voice data" is a technique for converting the generated text-format answer into data that can be reproduced by voice.

[0542] The "means for reproducing the audio data to the user" refers to a reproducing device for allowing the user to listen to the converted audio data.

[0543] The "means for displaying the generated answer as text data" is a device that has the function of displaying the answer not only as voice but also as text on a screen.

[0544] "Means in which the generated answers are in the form of a dialogue by a specific person" refers to a technology in which the answers are likened to a specific historical figure and are provided in the same tone and style as that figure.

[0545] System Overview

[0546] The system that realizes this application example integrates the functions of voice input, voice recognition, natural language processing, database search, answer generation, voice synthesis, result playback, and text display. The specific processing of each subsystem is explained below.

[0547] Hardware and software used

[0548] Hardware:

[0549] microphone

[0550] speaker

[0551] Displays (smartphones, etc.)

[0552] High-performance devices (smartphones, servers)

[0553] software:

[0554] Python: A programming language for implementing the overall logic of a program

[0555] speech_recognition package: A library for converting voice input to text using the Google Speech-to-Text API.

[0556] transformers package: A library for analyzing user intent and generating appropriate answers using Hugging Face's natural language processing model.

[0557] gTTS (Google Text-to-Speech): A library for converting generated text into audio data.

[0558] playsound: A library for playing converted audio data

[0559] Details of data processing and calculation

[0560] 1. Voice Input and Recognition

[0561] The user inputs a question by voice through the microphone. For example, in the scenario of "asking a question to a specific historical figure," the user might say, "Historical figure, please tell me your famous quote." This is captured by the microphone and converted into text data using the speech_recognition package.

[0562] 2. Natural Language Processing

[0563] The device sends the converted text to a natural language processing engine, which analyzes the user's question, extracts keywords and contextual information, and generates structured data using Hugging Face's transformers package.

[0564] 3. Database search and answer generation

[0565] The server receives the parsed structured data and searches for relevant information in a database. Based on the information obtained, it generates an appropriate answer to the user's question. In this process, the generated text is modeled after the tone and writing style of famous people, so that the answer is provided in a conversational style that is specific to the person.

[0566] 4. Speech synthesis and playback

[0567] The device sends the generated text data to a speech synthesis engine and converts it into voice data. This process uses the gTTS library. The generated voice data is played back to the user using the playsound library, and is also displayed as text data on the screen.

[0568] Specific examples

[0569] For example, consider the case where a user launches an application and asks, "Napoleon, tell me about the importance of strategy." In this case, the system performs speech recognition and converts the speech into text data. It then uses a natural language processing engine to analyze the question and retrieves an appropriate answer about Napoleon from a database. The answer generated, "Without strategy, no army can achieve victory," is converted into audio data and played back to the user. This entire process occurs in real time, providing the user with an interactive learning experience.

[0570] Prompt Sentence Examples

[0571] To generate a question for a particular historical figure, a prompt like the following can be used:

[0572] Prompt: "Generate a dialogue about the importance of strategy for the historical figure Napoleon."

[0573] Thus, the present invention utilizes advanced speech recognition and natural language processing techniques to provide a system that allows users to interactively learn about historical figures, thereby providing a richer learning experience for users.

[0574] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0575] Step 1:

[0576] Capture audio input

[0577] The user can enter questions or commands by voice through a microphone. The user's voice is then captured as an analog signal by the microphone. For example, a question such as "What is your most famous quote, historical figure?" can be captured as user input.

[0578] Step 2:

[0579] Speech-to-text conversion

[0580] The device converts the captured voice data to text using the speech_recognition package. In this process, speech recognition technology is applied to digitize the voice signal, which is then converted to text data. The input is voice data, and the output is the corresponding text data.

[0581] Step 3:

[0582] Intention analysis using natural language processing

[0583] The device sends the converted text data to a natural language processing engine, which uses the transformers package to analyze the user's intent. Specifically, it extracts keywords and contextual information from the text data and generates structured data. The input of this step is the text data, and the output is the analyzed structured data.

[0584] Step 4:

[0585] Database search

[0586] The server receives the parsed structured data and uses it to search a database containing information about historical figures, and executes queries to retrieve relevant information. The input is the structured data, and the output is relevant information.

[0587] Step 5:

[0588] Answer generation

[0589] The server generates appropriate answers for the user based on information retrieved from the database. It uses a generative AI model to generate dialogue-style answers from specific historical figures. The generated answers mimic the tone and writing style of the specific figures. The input is the retrieved information, and the output is the generated text answer.

[0590] Step 6:

[0591] Conversion to audio data

[0592] The device converts the generated text data into speech data using the gTTS library. The text is sent to a speech synthesis engine, which generates corresponding speech data. The input is the generated text data, and the output is speech data.

[0593] Step 7:

[0594] Playback of audio data and display of text

[0595] The device plays the generated audio data using the playsound library and simultaneously displays it as text data on the display. In this step, the user hears the answer aloud and the text is also displayed on the screen. The input is audio data and text data, and the output is audio playback and text display.

[0596] This series of processes allows users to ask questions about historical figures by voice and receive answers in audio and text, providing an interactive and realistic learning experience.

[0597] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0598] System Overview

[0599] This invention is a system that allows users to voice-input questions to historical figures specified by the user and provides answers to those questions in an interactive format. This system converts the voice input into text, analyzes the intent of the question using a natural language processing engine, retrieves information from a database, generates answers, and plays them back to the user in voice. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it achieves more human-like dialogue.

[0600] Process Overview

[0601] 1. User Input

[0602] Users speak questions into their smartphones or smart speakers, for example, "Socrates, what is true knowledge?"

[0603] 2. Voice Recognition

[0604] The device captures voice input from the user and sends this voice data to a voice recognition module, which converts the voice data into text data. The converted text might be, for example, "Socrates, what is true knowledge?"

[0605] 3. Text Processing

[0606] The device sends the text data obtained by speech recognition to a natural language processing engine, which analyzes the user's intent. The natural language processing engine then analyzes the text data for keywords and contextual information to generate structured data.

[0607] 4. Emotion analysis

[0608] The device sends text data and voice data to the emotion engine, which analyzes the user's emotions. The emotion engine identifies the user's emotions from the tone of the voice and the content of the text. For example, it analyzes the user's emotional state, such as whether they are excited, sad, or calm.

[0609] 5. Database Search

[0610] The server receives information from the natural language processing engine and emotion engine and searches a database containing information about great people. For example, it retrieves a famous quote by Socrates: "All I know is that I know nothing."

[0611] 6. Answer generation

[0612] The server generates an appropriate response to the user based on the information retrieved from the database and the analysis results of the emotion engine. For example, if the user is excited, it generates a response in a calm tone.

[0613] 7. Speech Synthesis

[0614] The device converts the generated text data into voice data. The speech synthesis engine converts the generated text into natural-sounding speech and adjusts the tone based on the results of emotion analysis. For example, if the user is sad, the device will generate a gentler tone of voice.

[0615] 8. Responding to the User

[0616] The terminal plays the generated voice data to the user, who can hear a response such as "All I know is that I know nothing" spoken in an appropriate emotional tone.

[0617] Specific examples

[0618] Example 1: Dialogue with Socrates

[0619] 1. User Input: The user asks, "Socrates, what is true knowledge?"

[0620] 2. Speech recognition: The device converts speech into text.

[0621] 3. Text processing: The device sends the text to a natural language processing engine to analyze the intent.

[0622] 4. Emotion analysis: The device analyzes the user's emotions. For example, it determines that the user is calm.

[0623] 5. Database lookup: The server searches the database and retrieves the appropriate information.

[0624] 6. Answer generation: The server generates an answer in a calm tone based on the results of sentiment analysis.

[0625] 7. Speech synthesis: The device converts the generated text into audio data.

[0626] 8. Response to the user: The terminal plays back in a calm tone, "All I know is that I know nothing."

[0627] In this way, each of the methods in this system works together to provide users with an effective and interactive learning environment. Furthermore, by combining it with an emotion engine, flexible responses based on the user's emotions become possible, resulting in more human-like dialogue.

[0628] The processing flow will be explained below.

[0629] Step 1:

[0630] Users can speak into their smartphone or smart speaker to ask questions to historical figures, such as, "Socrates, what is true knowledge?"

[0631] Step 2:

[0632] The device captures voice input from the user with a microphone and sends this voice data to a voice recognition module. The voice recognition module converts the voice data into text data. The converted text becomes, "Socrates, what is true knowledge?"

[0633] Step 3:

[0634] The device sends the text data obtained by speech recognition to a natural language processing (NLP) engine. The NLP engine extracts keywords and contextual information from the text data and analyzes the user's intent. For example, it generates structured data such as "Socrates (great man)" and "true knowledge (question content)."

[0635] Step 4:

[0636] The device sends the text data and voice data to the emotion engine, which analyzes the user's emotion from the voice tone and text content. For example, it identifies the user's emotional state, such as calm, excited, or sad.

[0637] Step 5:

[0638] The device transmits the intent-analyzed structured data and the results of emotion analysis to the server.

[0639] Step 6:

[0640] The server receives the structured data and searches a database containing information about great people. For example, based on the keywords "Socrates" and "true knowledge," it retrieves Socrates' famous quote, "All I know is that I know nothing."

[0641] Step 7:

[0642] The server generates appropriate responses to the user based on the information retrieved from the database and the results of emotion analysis. For example, if the server determines that the user is calm, it generates responses that mimic the tone of Socrates, who has a calm mind.

[0643] Step 8:

[0644] The server transmits the generated text data to the terminal.

[0645] Step 9:

[0646] The device sends the text data received from the server to a speech synthesis engine, which converts the text data into speech data and adjusts the tone of the speech based on the results of emotion analysis. For example, if the user is sad, the device generates a response in a gentle tone.

[0647] Step 10:

[0648] The terminal plays the generated voice data to the user, who can hear a response such as "All I know is that I know nothing" spoken in an appropriate emotional tone.

[0649] These are the specific processing steps required to provide information about historical figures in response to a user's voice question. This process allows the user to learn through a dialogue with the designated historical figure, and by combining it with an emotion engine, it is possible to achieve a more human-like dialogue.

[0650] Example 2

[0651] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0652] In modern learning systems, it is difficult for users to ask historical figures questions and receive interactive, emotionally relevant answers. Traditional systems often provide cold responses, which can discourage users from learning. Furthermore, the quality of voice-based engines and the accuracy of natural language processing are limited, leaving issues with the accuracy and persuasiveness of the information.

[0653] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving a voice input, means for converting the voice input into text, means for sending the converted text to a natural language processing engine and analyzing the intention, means for searching a database based on the analyzed intention and acquiring corresponding information, means for analyzing the user's emotions using an emotion analysis engine, means for generating an answer based on the acquired information and the analyzed emotion information, means for converting the generated answer into voice data, and means for playing the voice data to the user. This makes it possible to provide an interactive answer according to the user's emotions toward the historical figure about whom the question was asked.

[0654] "Means for receiving voice input" is a general term for devices and modules that capture voices uttered by the user and input them into the system.

[0655] "Means for converting voice input to text" refers to a device or module for analyzing captured voice data and converting it into corresponding text data.

[0656] A "natural language processing engine" is a general term for software or systems that analyze text data and extract user intent and keywords.

[0657] "Means for analyzing intent" refers to a device or module that uses a natural language processing engine to perform processing to analyze the intent of a question from input text data.

[0658] "Means for searching a database and retrieving corresponding information" refers to a device or module for searching related information in a database based on the analyzed intent and retrieving the required information.

[0659] "Emotion analysis engine" is a general term for software or systems that analyze a user's voice or text data and identify the user's emotional state.

[0660] "Means for analyzing emotions" refers to a device or module that uses an emotion analysis engine to perform processing to analyze the user's emotional state.

[0661] The "means for generating an answer" refers to a device or module for generating an appropriate answer to the user based on the acquired information and emotional information.

[0662] "Means for converting generated answers into voice data" refers to a device or module for converting generated text answers into voice data.

[0663] "Means for playing back audio data to the user" refers to a device or module that plays back the generated audio data in order to deliver it to the user.

[0664] "Biographical information" is a general term for data, facts, quotes, and other information relating to historical or famous figures.

[0665] "Cloud-based services" refer to remote computing resources, data storage, and software services delivered over the internet.

[0666] MODE FOR CARRYING OUT THE INVENTION

[0667] System Overview

[0668] This invention is an interactive learning system that allows users to ask questions using voice input and provides emotionally appropriate answers based on information about the historical figures in question. The system converts the voice input into text, analyzes the user's intention using a natural language processing engine, and retrieves corresponding information from a database. It also has the function of analyzing the user's emotions using an emotion analysis engine, generating answers in an appropriate tone, and playing them back as voice.

[0669] Hardware and software used

[0670] 1. Hardware:

[0671] Smartphones and smart speakers: Devices for voice input and playback.

[0672] Server: A central processing unit that searches the database and generates answers.

[0673] 2. Software:

[0674] Speech Recognition Module: Converts voice data into text data using the Google Speech-to-Text API.

[0675] Natural Language Processing Engine: OpenAI GPT-3 is used to analyze the intent of the text.

[0676] Sentiment Analysis Engine: Analyzes user emotions using IBM Watson Tone Analyzer.

[0677] Speech synthesis engine: Generates speech from text using Amazon Polly.

[0678] Processing flow

[0679] The specific operations of each processing step of this system are shown below.

[0680] 1. User Input

[0681] Users speak questions into their smartphones or smart speakers, for example, "Socrates, what is true knowledge?"

[0682] 2. Voice Recognition

[0683] The device captures the user's voice input and converts it into text using the Google Speech-to-Text API. Specifically, the speech "Socrates, what is true knowledge?" is converted into text "Socrates, what is true knowledge?"

[0684] 3. Text Processing

[0685] The device then sends the converted text to OpenAI GPT-3 to analyze the user's intent, during which the system extracts the keywords "Socrates" and "knowledge" to understand the gist of the question.

[0686] 4. Emotion analysis

[0687] The device sends text and voice data to the IBM Watson Tone Analyzer, which analyzes the user's emotions, identifying whether the user is calm, excited, sad, or other such emotions.

[0688] 5. Database Search

[0689] The server receives information from the natural language processing engine and emotion engine and searches a database of people, such as the quote "All I know is that I know nothing" for "Socrates." The database uses a cloud-based MySQL database.

[0690] 6. Answer generation

[0691] The server generates an appropriate response to the user based on the acquired information and the analyzed emotional information. For a calm user, the server generates a response in a calm tone.

[0692] 7. Speech Synthesis

[0693] The device converts the generated text into speech using Amazon Polly, a speech synthesis engine that adjusts tone based on sentiment analysis.

[0694] 8. Responding to the User

[0695] The terminal plays the generated voice data back to the user. For example, the user can hear a calm response such as "All I know is that I know nothing."

[0696] Specific examples

[0697] Example 1: Dialogue with Socrates

[0698] If a user asks, "Socrates, what is true knowledge?" the system behaves as follows:

[0699] 1. Speech is converted into text using speech recognition.

[0700] 2. The natural language processing engine analyzes the intent.

[0701] 3. The sentiment is analyzed by the sentiment analysis engine.

[0702] 4. The corresponding information is retrieved from the database.

[0703] 5. The answer is generated.

[0704] 6. It is converted into audio and played back to the user.

[0705] Example prompts for generative AI models

[0706] "Show me what your system would do if a user calmly asked, 'Socrates, what is true knowledge?'"

[0707] The system allows users to receive real-time emotional responses, creating a more interactive and effective learning experience.

[0708] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0709] Step 1:

[0710] Users speak questions into their smartphone or smart speaker, such as "Socrates, what is true knowledge?" This voice input is captured by the device's microphone.

[0711] Step 2:

[0712] The device sends the captured voice data to the speech recognition module, which uses the Google Speech-to-Text API. Specifically, the voice signal is sent to the server, which converts it into text data via the API: "Socrates, what is true knowledge?" The input is the user's voice, and the output is the corresponding text data.

[0713] Step 3:

[0714] The device sends the converted text data to a natural language processing engine (e.g., OpenAI GPT-3). The natural language processing engine extracts important keywords and contextual information from the text data and analyzes the user's intent. The input here is the converted text data, and the output is the analyzed user's intent information. Specifically, the keywords "Socrates" and "knowledge" are extracted from the text "Socrates, what is true knowledge?"

[0715] Step 4:

[0716] The device simultaneously sends text and voice data to an emotion analysis engine (e.g., IBM Watson Tone Analyzer). The emotion analysis engine analyzes the tone of the voice and the content of the text to identify the user's emotional state. The input is text and voice data, and the output is the user's emotional information. Specific emotions that can be identified include calm, excitement, and sadness. In this example, the user is analyzed as calm.

[0717] Step 5:

[0718] The server receives information from the natural language processing engine and sentiment analysis engine and searches the database. The database uses a cloud-based MySQL database. In this step, related information in the database is searched for based on the analyzed keywords "Socrates" and "knowledge" to obtain the appropriate answer. The input is the analyzed keywords and sentiment information, and the output is the obtained information. For example, the search result obtained is "All I know is that I know nothing."

[0719] Step 6:

[0720] The server generates an appropriate answer for the user based on the acquired information and emotional information. Since the answer needs to be in a calm tone, the server generates a calm tone text such as "All I know is that I know nothing." The input is the acquired information and emotional information, and the output is the generated answer text.

[0721] Step 7:

[0722] The device converts the generated text data into voice data. Amazon Polly is used here. The voice synthesis engine adjusts the tone of the voice based on the results of emotion analysis. The input is the generated text data, and the output is voice data with a tone that corresponds to the emotion. Specifically, voice is generated in a calm tone.

[0723] Step 8:

[0724] The terminal plays the generated voice data to the user. The voice played is a calm response saying, "All I know is that I know nothing." The user can hear this aloud. The input is the generated voice data, and the output is the voice played to the user.

[0725] (Application example 2)

[0726] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0727] In traditional brick-and-mortar stores, it has been difficult to provide accurate and emotionally appropriate information to customer questions. Furthermore, when historical background or detailed knowledge is required, the lack of knowledge and inconsistency of responses from store staff can be problematic. This can lead to a poor customer experience.

[0728] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0729] In this invention, the server includes means for analyzing the user's emotions, means for referencing the user's emotion data when generating the answer, and means that is installed in the guide robot and provides services to customers in the physical store. This enables flexible and accurate responses according to the customer's emotions and realizes the provision of consistent and detailed information in response to the question.

[0730] "Voice input" is a means by which a user provides data to a system using speech.

[0731] "Text conversion" is the process of converting audio data into text data.

[0732] A "natural language processing engine" is software that performs semantic analysis on text data and understands the user's intent.

[0733] "Database search" is an operation for searching information in a database based on the analyzed intent and obtaining the required data.

[0734] "Answer generation" is the process of creating an appropriate answer to a user's question based on the acquired information.

[0735] "Audio data conversion" is the process of converting the generated text data into an audio file and letting the user listen to it.

[0736] "Emotion analysis" is the process of evaluating and determining a user's emotional state from their voice or text.

[0737] "Response playback" is the process of playing back the generated voice data to the user.

[0738] A "guide robot" is a robot that is installed in physical stores and other locations to act as a guide and assist customers.

[0739] A "brick and mortar store" is a store that offers goods or services in a physical location.

[0740] This invention describes an embodiment of a system that allows a user to voice-input a question to a historical figure designated by the user and provides an answer to that question in an interactive format. This system has the functions of converting the voice input into text, analyzing the intent of the question using a natural language processing engine, retrieving information from a database, generating an answer, and playing it back to the user in voice. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to achieve more human-like dialogue.

[0741] Hardware and software used

[0742] Hardware:

[0743] Microphone (for voice input)

[0744] Speaker (for audio output)

[0745] Guidance robots (installed in brick-and-mortar stores for customer service)

[0746] software:

[0747] speech_recognition library (speech to text conversion)

[0748] pyttsx3 library (text-to-speech synthesis)

[0749] QA model (question answering) from the transformers library

[0750] emotion_recognition library (emotion analysis)

[0751] Processing Overview

[0752] 1. User Input:

[0753] The user can input questions to the guide robot by voice through a microphone. For example, the user can ask, "What time does this store open?"

[0754] 2. Speech Recognition:

[0755] The device captures the user's voice input and uses the speech_recognition library to convert the voice data into text, such as "What time does this store open?"

[0756] 3. Text Analysis:

[0757] The device sends the text data obtained by speech recognition to the natural language processing engine of the transformers library to analyze the user's intent. The natural language processing engine analyzes keywords and contextual information from the text data and generates structured data.

[0758] 4. Emotion analysis:

[0759] The device sends text and voice data to the emotion_recognition library to analyze the user's emotions. The emotion engine identifies the user's emotions from the tone of the voice and the content of the text. For example, it analyzes that the user is calm.

[0760] 5. Database Search:

[0761] The server receives information from the natural language processing engine and emotion engine and searches a database containing information about historical figures, obtaining information such as "This store has been in business since 1950."

[0762] 6. Answer generation:

[0763] The server generates an appropriate response to the user based on the acquired information and the analysis results of the emotion engine. For example, if the user is calm, it generates a response in a normal tone.

[0764] 7. Speech synthesis:

[0765] The device converts the generated text data into speech data, using the pyttsx3 library to convert the generated text into natural-sounding speech and adjust the tone based on the results of sentiment analysis.

[0766] 8. Response to the user:

[0767] The device plays the generated voice data to the user through a speaker, and the user can hear a response such as "This store has been in business since 1950" in an appropriate emotional tone.

[0768] Examples of prompt statements

[0769] For example, if a customer visits a store with their child and asks the robot, "What time does this store open?", the following prompt sentence will be generated:

[0770] User: How long has this store been open?

[0771] Emotion: neutral

[0772] A: This store has been in business since 1950.

[0773] This system will improve the customer experience in-store and enable accurate and flexible information provision.

[0774] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0775] Step 1:

[0776] User Input

[0777] The user inputs a question to the guide robot by voice through a microphone. For example, "What time does this store open?" The input of this step is the user's voice, and the output is voice data.

[0778] Step 2:

[0779] Voice Recognition

[0780] The device captures the user's voice input and converts it to text using the speech_recognition library. For example, the converted text might be, "What time does this store open?" The input for this step is voice data, and the output is text data.

[0781] Step 3:

[0782] Text analytics

[0783] The device sends the text data obtained by speech recognition to the natural language processing engine of the transformers library to analyze the user's intent. The natural language processing engine analyzes keywords and contextual information from the text data and generates structured data. The input of this step is text data, and the output is the analyzed structured data.

[0784] Step 4:

[0785] Emotion analysis

[0786] The device sends text and audio data to the emotion_recognition library to analyze the user's emotions. The emotion engine identifies the user's emotions from the tone of the voice and the content of the text. For example, it identifies emotions such as calm, excitement, and sadness. The input of this step is text and audio data, and the output is the user's emotional state.

[0787] Step 5:

[0788] Database search

[0789] The server receives information from the natural language processing engine and the emotion engine and searches a database containing information about historical figures. For example, it can obtain information such as "We have been in business since 1950." The input of this step is the parsed structured data and emotion data, and the output is information as search results.

[0790] Step 6:

[0791] Answer generation

[0792] The server generates an appropriate response to the user based on the acquired information and the analysis results of the emotion engine. For example, if the user is calm, it generates a response in a normal tone, such as "This store has been in business since 1950." The input for this step is the search result information and emotion data, and the output is the generated response text.

[0793] Step 7:

[0794] Speech synthesis

[0795] The device converts the generated text data into speech data using the pyttsx3 library. The tone is also adjusted based on the results of sentiment analysis. The input of this step is the response text, and the output is speech data.

[0796] Step 8:

[0797] Responding to the user

[0798] The terminal plays the generated voice data to the user through a speaker. The user can hear, for example, a response such as "This store has been in business since 1950" in an appropriate emotional tone. The input of this step is the voice data, and the output is audio information that the user can hear.

[0799] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0800] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0801] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0802] [Third embodiment]

[0803] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0804] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0805] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0806] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0807] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0808] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0809] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0810] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0811] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0812] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0813] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0814] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0815] System Overview

[0816] The present invention is a system that allows users to interactively learn information about a historical figure they specify. This system includes means for the user to input voice, convert the voice input into text data, and perform analysis, database search, answer generation, speech synthesis, and playback using a natural language processing engine. The specific processing of each subsystem is described below.

[0817] Process Overview

[0818] 1. User Input

[0819] Users can speak a question into their smartphone or smart speaker, for example, "Socrates, what is true knowledge?"

[0820] 2. Voice Recognition

[0821] The device captures the user's voice input with a microphone and converts the speech into text data. Cloud services such as the Google Speech-to-Text API and Amazon Transcribe can be used for speech recognition. The converted text data will read, "Socrates, what is true knowledge?"

[0822] 3. Text Processing

[0823] The device sends the text data obtained by speech recognition to a natural language processing (NLP) engine, which analyzes the user's intent. The NLP engine extracts keywords and contextual information from the text data and generates structured data. For example, it generates structured data such as "Socrates (great person)" and "true knowledge (question content)."

[0824] 4. Database Search

[0825] The server receives the structured data and searches a database containing information about great people, retrieving, for example, a famous quote by Socrates: "All I know is that I know nothing."

[0826] 5. Answer generation

[0827] The server uses information from the database to generate an appropriate response for the user, imitating the tone and style of a great man, resulting in text of the form "All I know is that I know nothing."

[0828] 6. Speech Synthesis

[0829] The device receives text data from the server and sends it to a speech synthesis engine, which converts it into voice data. Services such as Google Text-to-Speech and Amazon Polly can be used for voice synthesis. The generated voice data sounds like Socrates is actually speaking.

[0830] 7. Responding to the User

[0831] The device then plays the generated audio data back to the user, who can hear a response such as, "All I know is that I know nothing." This process allows the user to interact with historical figures, providing a rich learning experience.

[0832] Specific examples

[0833] Example 1: Dialogue with Socrates

[0834] 1. User Input: The user asks, "Socrates, what is true knowledge?"

[0835] 2. Speech recognition: The device converts speech into text.

[0836] 3. Text processing: The device sends the text to a natural language processing engine to analyze the intent.

[0837] 4. Database lookup: The server searches the database and retrieves the appropriate information.

[0838] 5. Answer Generation: The server generates the answer.

[0839] 6. Speech synthesis: The device converts the generated text into audio data.

[0840] 7. Respond to the user: The terminal plays the audio data to the user.

[0841] In this way, the system provides users with an effective and interactive learning environment by linking together the various means. This invention enables users to acquire in-depth knowledge about historical figures in an interactive format.

[0842] The processing flow will be explained below.

[0843] Step 1:

[0844] Users speak into their smartphone or smart speaker to ask a question to a designated historical figure, for example, "Socrates, what is true knowledge?"

[0845] Step 2:

[0846] The device captures voice input from the user with a microphone and sends this voice data to a speech recognition module, which converts the voice data into text. The converted text is "Socrates, what is true knowledge?"

[0847] Step 3:

[0848] The device sends the text data obtained by speech recognition to a natural language processing engine. The natural language processing engine extracts keywords and contextual information from the text data and analyzes the user's intent. For example, it generates structured data such as "Socrates (great person)" and "true knowledge (question content)."

[0849] Step 4:

[0850] The terminal transmits the parsed structured data to the server.

[0851] Step 5:

[0852] The server receives the structured data and searches a database containing information about great people, such as Socrates' famous quote, "All I know is that I know nothing."

[0853] Step 6:

[0854] The server generates a textual response to the user based on the information retrieved from the database, reflecting the tone and style of the great man.

[0855] Step 7:

[0856] The server returns the generated text data to the terminal.

[0857] Step 8:

[0858] The device then sends the received text data to a speech synthesis engine, which converts it into voice data. The speech synthesis engine uses Google Text-to-Speech or Amazon Polly to convert the text into natural-sounding speech.

[0859] Step 9:

[0860] The terminal plays the generated voice data to the user, who can hear the answer, for example, "All I know is that I know nothing."

[0861] These are the specific steps the system takes to provide information about historical figures in response to user voice questions. This process allows the user to learn through dialogue with the designated historical figures.

[0862] Example 1

[0863] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0864] In conventional systems, users must rely on searching for information in books or websites when learning about historical figures, which does not provide an interactive learning experience. Furthermore, users without specialized knowledge have difficulty assessing the authenticity and appropriate interpretation of the information. Furthermore, while systems that use voice input and voice output are common, few systems effectively combine advanced natural language processing and database search.

[0865] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0866] In this invention, the server includes means for receiving voice input from a user, means for converting the voice input into text, means for sending the converted text to a natural language processing engine and analyzing the user's intent, means for searching an information storage area and retrieving corresponding information, means having a generative AI model for generating answers based on the retrieved information, means for converting the generated answers into voice data, and means for playing the voice data to the user, thereby enabling the user to interactively converse with historical figures, obtain reliable information via voice, and enjoy a rich learning experience.

[0867] "Voice input" is a method by which a user communicates information or instructions to a system via voice.

[0868] "Convert to text" is the process of converting audio data into text data.

[0869] A "natural language processing engine" is a computer program that analyzes text data and understands its meaning.

[0870] "Information storage area" refers to the database or storage system where data is saved.

[0871] A "generative AI model" is an artificial intelligence model that generates appropriate answers to users based on acquired information.

[0872] "Audio Data" means data digitally represented in audio format.

[0873] "Playing to the user" refers to the process of making the generated audio data audible to the user through an output device such as a speaker.

[0874] "Distributed delivery service" refers to a cloud service that is delivered over the Internet and that users can access from multiple locations.

[0875] System Overview

[0876] The present invention provides a system that allows a user to interactively learn information about an important person designated by the user. The system includes means for speech input, text conversion, natural language processing, database search, answer generation, speech synthesis, and speech playback.

[0877] Program processing details

[0878] User Input

[0879] The user uses a voice input device (e.g., a smartphone or a smart speaker) to input a question by voice, for example, saying, "Tell me about a particular important person."

[0880] Voice Recognition

[0881] The device captures the user's voice input through a microphone and converts it into text using a speech recognition service such as the Google Speech-to-Text API or Amazon Transcribe. For example, the speech might be converted into text like, "Tell me about a specific important person."

[0882] Text Processing

[0883] The device sends the text data obtained by voice recognition to a natural language processing engine. The natural language processing engine (for example, Google Cloud Natural Language API or IBM Watson Natural Language Understanding) analyzes the text data and understands the user's intent. The analysis results in data such as "specific important person (target)" and "please tell me (request)."

[0884] Database search

[0885] The server receives the generated analysis data and searches an information storage area (for example, a MySQL or PostgreSQL database). As a result of the search, information about a specific important person is obtained. For example, famous quotes and life information about the specific important person are obtained.

[0886] Answer generation

[0887] The server generates an appropriate answer for the user based on the information retrieved from the database. It uses a generative AI model (such as OpenAI's GPT-3 or BERT) to generate a response text based on the retrieved information. For example, it generates text in the format "A certain important person left behind the famous quote XX."

[0888] Speech synthesis

[0889] The device receives text data from the server, sends it to a speech synthesis engine, and converts it into voice data. Services such as Google Text-to-Speech and Amazon Polly are used for voice synthesis. The generated voice data is synthesized to sound like a specific important person is actually speaking.

[0890] Responding to the user

[0891] The device then plays the generated voice data back to the user, who can hear a voice response such as, "A particular important person left behind the famous quote XX." Through this process, the user can interactively learn about important people.

[0892] Specific examples

[0893] Example 1: Interaction with a specific important person

[0894] 1. User Input: The user asks, "Tell me about a particular important person."

[0895] 2. Speech recognition: The device converts the speech into text using the Google Speech-to-Text API. The resulting text is "Tell me about a specific important person."

[0896] 3. Text processing: The device sends the text to the Google Cloud Natural Language API, which analyzes the intent and generates structured data.

[0897] 4. Database lookup: The server uses MySQL to lookup the database and retrieve information about a particular VIP.

[0898] 5. Answer generation: The server uses GPT-3 to generate an answer, creating the text "A certain important person left behind the famous quote XX."

[0899] 6. Text-to-speech: Your device will convert text to speech using Google Text-to-Speech.

[0900] 7. Response to the user: The terminal plays the audio data to the user, and the user hears, "A certain important person left behind the famous quote XX."

[0901] Prompt Sentence Examples

[0902] "Ask questions about important historical figures. For example, 'Tell me about a particular important person.'"

[0903] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0904] Step 1:

[0905] A user uses a voice input device (such as a smartphone or smart speaker) to input a question by voice. For example, they might say, "Tell me about a specific important person." At this stage, the input is voice data and has not yet been converted into text data.

[0906] Step 2:

[0907] The device captures the user's voice input through a microphone. It uses a voice recognition service (for example, Google Speech-to-Text API) to convert the voice data into text data. Specifically, it sends the voice data to a cloud service and receives the returned text data. The input is voice data, and the output is text data, such as "Tell me about a specific important person."

[0908] Step 3:

[0909] The device sends the acquired text data to a natural language processing engine (for example, Google Cloud Natural Language API) for analysis. Specifically, it performs syntactic analysis of the text and extracts keywords to understand the user's intent. The input is text data, and the output is data indicating the intent, such as "specific important person (target)" and "please tell me (request)."

[0910] Step 4:

[0911] The server receives the intent data sent from the device and searches an information storage area (for example, a MySQL database). It generates a search query and queries the database to obtain information about a specific important person. The specific operation is to execute an SQL query and extract the relevant records. The input is the intent data, and the output is information data as a search result.

[0912] Step 5:

[0913] The server generates an answer using a generative AI model (e.g., GPT-3) based on the information data obtained from the search results. Specifically, the process involves inputting the information data into the AI ​​model and outputting a corresponding text answer. The input is the information data, and the output is a text answer such as "A certain important person left behind the famous quote XX."

[0914] Step 6:

[0915] The device sends the answer text received from the server to a speech synthesis engine (e.g., Google Text-to-Speech) and converts it into voice data. During this process, it calls an API to convert text data into voice format and receives the result. The input is the answer text, and the output is voice data.

[0916] Step 7:

[0917] The terminal plays the generated voice data to the user. At this stage, the speaker is used to output the voice and convey the answer to the user. The specific operation is to pass the voice data to the playback device and output it. The input is the voice data, and the output is the voice information that the user hears.

[0918] (Application example 1)

[0919] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0920] Conventional learning systems have made it difficult for users to learn detailed information about historical figures of interest through a direct dialogue. Furthermore, users cannot easily obtain the information they are looking for through voice, making it difficult to provide an interactive learning experience. Furthermore, there are limitations to the accuracy and realism of the generated answers, and improvements are needed to increase user satisfaction.

[0921] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0922] In this invention, the server includes means for receiving voice input from a user, means for converting the voice input into text, means for sending the converted text to a natural language processing engine and analyzing the user's intent, means for searching a database based on the analyzed intent and obtaining corresponding information, means for generating an answer based on the obtained information, means for converting the generated answer into voice data, means for playing the voice data to the user, means for displaying the generated answer as text data, and means for generating an answer in the form of a dialogue between a predetermined specific person. This allows a user to easily learn in the form of a dialogue with a historical figure through voice input, thereby providing a highly realistic, interactive learning experience.

[0923] A "means for receiving voice input from a user" is a device capable of capturing voice data such as questions or commands uttered by a user.

[0924] The "means for converting voice input into text" is a technology that converts captured voice data into text information, and utilizes voice recognition technology.

[0925] "Means of sending the converted text to a natural language processing engine and analyzing the user's intent" is a technology that uses advanced algorithms to analyze text data to understand the intent of a user's questions or commands.

[0926] The "means for searching a database based on the analyzed intent and retrieving corresponding information" is a system that has the function of searching and retrieving highly relevant information from within a database.

[0927] The "means for generating an answer based on acquired information" is a system that has the function of constructing an appropriate answer to a user's question based on information acquired from a database.

[0928] The "means for converting the generated answer into voice data" is a technique for converting the generated text-format answer into data that can be reproduced by voice.

[0929] The "means for reproducing the audio data to the user" refers to a reproducing device for allowing the user to listen to the converted audio data.

[0930] The "means for displaying the generated answer as text data" is a device that has the function of displaying the answer not only as voice but also as text on a screen.

[0931] "Means in which the generated answers are in the form of a dialogue by a specific person" refers to a technology in which the answers are likened to a specific historical figure and are provided in the same tone and style as that figure.

[0932] System Overview

[0933] The system that realizes this application example integrates the functions of voice input, voice recognition, natural language processing, database search, answer generation, voice synthesis, result playback, and text display. The specific processing of each subsystem is explained below.

[0934] Hardware and software used

[0935] Hardware:

[0936] microphone

[0937] speaker

[0938] Displays (smartphones, etc.)

[0939] High-performance devices (smartphones, servers)

[0940] software:

[0941] Python: A programming language for implementing the overall logic of a program

[0942] speech_recognition package: A library for converting voice input to text using the Google Speech-to-Text API.

[0943] transformers package: A library for analyzing user intent and generating appropriate answers using Hugging Face's natural language processing model.

[0944] gTTS (Google Text-to-Speech): A library for converting generated text into audio data.

[0945] playsound: A library for playing converted audio data

[0946] Details of data processing and calculation

[0947] 1. Voice Input and Recognition

[0948] The user inputs a question by voice through the microphone. For example, in the scenario of "asking a question to a specific historical figure," the user might say, "Historical figure, please tell me your famous quote." This is captured by the microphone and converted into text data using the speech_recognition package.

[0949] 2. Natural Language Processing

[0950] The device sends the converted text to a natural language processing engine, which analyzes the user's question, extracts keywords and contextual information, and generates structured data using Hugging Face's transformers package.

[0951] 3. Database search and answer generation

[0952] The server receives the parsed structured data and searches for relevant information in a database. Based on the information obtained, it generates an appropriate answer to the user's question. In this process, the generated text is modeled after the tone and writing style of famous people, so that the answer is provided in a conversational style that is specific to the person.

[0953] 4. Speech synthesis and playback

[0954] The device sends the generated text data to a speech synthesis engine and converts it into voice data. This process uses the gTTS library. The generated voice data is played back to the user using the playsound library, and is also displayed as text data on the screen.

[0955] Specific examples

[0956] For example, consider the case where a user launches an application and asks, "Napoleon, tell me about the importance of strategy." In this case, the system performs speech recognition and converts the speech into text data. It then uses a natural language processing engine to analyze the question and retrieves an appropriate answer about Napoleon from a database. The answer generated, "Without strategy, no army can achieve victory," is converted into audio data and played back to the user. This entire process occurs in real time, providing the user with an interactive learning experience.

[0957] Prompt Sentence Examples

[0958] To generate a question for a particular historical figure, a prompt like the following can be used:

[0959] Prompt: "Generate a dialogue about the importance of strategy for the historical figure Napoleon."

[0960] Thus, the present invention utilizes advanced speech recognition and natural language processing techniques to provide a system that allows users to interactively learn about historical figures, thereby providing a richer learning experience for users.

[0961] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0962] Step 1:

[0963] Capture audio input

[0964] The user can enter questions or commands by voice through a microphone. The user's voice is then captured as an analog signal by the microphone. For example, a question such as "What is your most famous quote, historical figure?" can be captured as user input.

[0965] Step 2:

[0966] Speech-to-text conversion

[0967] The device converts the captured voice data to text using the speech_recognition package. In this process, speech recognition technology is applied to digitize the voice signal, which is then converted to text data. The input is voice data, and the output is the corresponding text data.

[0968] Step 3:

[0969] Intention analysis using natural language processing

[0970] The device sends the converted text data to a natural language processing engine, which uses the transformers package to analyze the user's intent. Specifically, it extracts keywords and contextual information from the text data and generates structured data. The input of this step is the text data, and the output is the analyzed structured data.

[0971] Step 4:

[0972] Database search

[0973] The server receives the parsed structured data and uses it to search a database containing information about historical figures, and executes queries to retrieve relevant information. The input is the structured data, and the output is relevant information.

[0974] Step 5:

[0975] Answer generation

[0976] The server generates appropriate answers for the user based on information retrieved from the database. It uses a generative AI model to generate dialogue-style answers from specific historical figures. The generated answers mimic the tone and writing style of the specific figures. The input is the retrieved information, and the output is the generated text answer.

[0977] Step 6:

[0978] Conversion to audio data

[0979] The device converts the generated text data into speech data using the gTTS library. The text is sent to a speech synthesis engine, which generates corresponding speech data. The input is the generated text data, and the output is speech data.

[0980] Step 7:

[0981] Playback of audio data and display of text

[0982] The device plays the generated audio data using the playsound library and simultaneously displays it as text data on the display. In this step, the user hears the answer aloud and the text is also displayed on the screen. The input is audio data and text data, and the output is audio playback and text display.

[0983] This series of processes allows users to ask questions about historical figures by voice and receive answers in audio and text, providing an interactive and realistic learning experience.

[0984] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0985] System Overview

[0986] This invention is a system that allows users to voice-input questions to historical figures specified by the user and provides answers to those questions in an interactive format. This system converts the voice input into text, analyzes the intent of the question using a natural language processing engine, retrieves information from a database, generates answers, and plays them back to the user in voice. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it achieves more human-like dialogue.

[0987] Process Overview

[0988] 1. User Input

[0989] Users speak questions into their smartphones or smart speakers, for example, "Socrates, what is true knowledge?"

[0990] 2. Voice Recognition

[0991] The device captures voice input from the user and sends this voice data to a voice recognition module, which converts the voice data into text data. The converted text might be, for example, "Socrates, what is true knowledge?"

[0992] 3. Text Processing

[0993] The device sends the text data obtained by speech recognition to a natural language processing engine, which analyzes the user's intent. The natural language processing engine then analyzes the text data for keywords and contextual information to generate structured data.

[0994] 4. Emotion analysis

[0995] The device sends text data and voice data to the emotion engine, which analyzes the user's emotions. The emotion engine identifies the user's emotions from the tone of the voice and the content of the text. For example, it analyzes the user's emotional state, such as whether they are excited, sad, or calm.

[0996] 5. Database Search

[0997] The server receives information from the natural language processing engine and emotion engine and searches a database containing information about great people. For example, it retrieves a famous quote by Socrates: "All I know is that I know nothing."

[0998] 6. Answer generation

[0999] The server generates an appropriate response to the user based on the information retrieved from the database and the analysis results of the emotion engine. For example, if the user is excited, it generates a response in a calm tone.

[1000] 7. Speech Synthesis

[1001] The device converts the generated text data into voice data. The speech synthesis engine converts the generated text into natural-sounding speech and adjusts the tone based on the results of emotion analysis. For example, if the user is sad, the device will generate a gentler tone of voice.

[1002] 8. Responding to the User

[1003] The terminal plays the generated voice data to the user, who can hear a response such as "All I know is that I know nothing" spoken in an appropriate emotional tone.

[1004] Specific examples

[1005] Example 1: Dialogue with Socrates

[1006] 1. User Input: The user asks, "Socrates, what is true knowledge?"

[1007] 2. Speech recognition: The device converts speech into text.

[1008] 3. Text processing: The device sends the text to a natural language processing engine to analyze the intent.

[1009] 4. Emotion analysis: The device analyzes the user's emotions. For example, it determines that the user is calm.

[1010] 5. Database lookup: The server searches the database and retrieves the appropriate information.

[1011] 6. Answer generation: The server generates an answer in a calm tone based on the results of sentiment analysis.

[1012] 7. Speech synthesis: The device converts the generated text into audio data.

[1013] 8. Response to the user: The terminal plays back in a calm tone, "All I know is that I know nothing."

[1014] In this way, each of the methods in this system works together to provide users with an effective and interactive learning environment. Furthermore, by combining it with an emotion engine, flexible responses based on the user's emotions become possible, resulting in more human-like dialogue.

[1015] The processing flow will be explained below.

[1016] Step 1:

[1017] Users can speak into their smartphone or smart speaker to ask questions to historical figures, such as, "Socrates, what is true knowledge?"

[1018] Step 2:

[1019] The device captures voice input from the user with a microphone and sends this voice data to a voice recognition module. The voice recognition module converts the voice data into text data. The converted text becomes, "Socrates, what is true knowledge?"

[1020] Step 3:

[1021] The device sends the text data obtained by speech recognition to a natural language processing (NLP) engine. The NLP engine extracts keywords and contextual information from the text data and analyzes the user's intent. For example, it generates structured data such as "Socrates (great man)" and "true knowledge (question content)."

[1022] Step 4:

[1023] The device sends the text data and voice data to the emotion engine, which analyzes the user's emotion from the voice tone and text content. For example, it identifies the user's emotional state, such as calm, excited, or sad.

[1024] Step 5:

[1025] The device transmits the intent-analyzed structured data and the results of emotion analysis to the server.

[1026] Step 6:

[1027] The server receives the structured data and searches a database containing information about great people. For example, based on the keywords "Socrates" and "true knowledge," it retrieves Socrates' famous quote, "All I know is that I know nothing."

[1028] Step 7:

[1029] The server generates appropriate responses to the user based on the information retrieved from the database and the results of emotion analysis. For example, if the server determines that the user is calm, it generates responses that mimic the tone of Socrates, who has a calm mind.

[1030] Step 8:

[1031] The server transmits the generated text data to the terminal.

[1032] Step 9:

[1033] The device sends the text data received from the server to a speech synthesis engine, which converts the text data into speech data and adjusts the tone of the speech based on the results of emotion analysis. For example, if the user is sad, the device generates a response in a gentle tone.

[1034] Step 10:

[1035] The terminal plays the generated voice data to the user, who can hear a response such as "All I know is that I know nothing" spoken in an appropriate emotional tone.

[1036] These are the specific processing steps required to provide information about historical figures in response to a user's voice question. This process allows the user to learn through a dialogue with the designated historical figure, and by combining it with an emotion engine, it is possible to achieve a more human-like dialogue.

[1037] Example 2

[1038] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1039] In modern learning systems, it is difficult for users to ask historical figures questions and receive interactive, emotionally relevant answers. Traditional systems often provide cold responses, which can discourage users from learning. Furthermore, the quality of voice-based engines and the accuracy of natural language processing are limited, leaving issues with the accuracy and persuasiveness of the information.

[1040] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving a voice input, means for converting the voice input into text, means for sending the converted text to a natural language processing engine and analyzing the intention, means for searching a database based on the analyzed intention and acquiring corresponding information, means for analyzing the user's emotions using an emotion analysis engine, means for generating an answer based on the acquired information and the analyzed emotion information, means for converting the generated answer into voice data, and means for playing the voice data to the user. This makes it possible to provide an interactive answer according to the user's emotions toward the historical figure about whom the question was asked.

[1041] "Means for receiving voice input" is a general term for devices and modules that capture voices uttered by the user and input them into the system.

[1042] "Means for converting voice input to text" refers to a device or module for analyzing captured voice data and converting it into corresponding text data.

[1043] A "natural language processing engine" is a general term for software or systems that analyze text data and extract user intent and keywords.

[1044] "Means for analyzing intent" refers to a device or module that uses a natural language processing engine to perform processing to analyze the intent of a question from input text data.

[1045] "Means for searching a database and retrieving corresponding information" refers to a device or module for searching related information in a database based on the analyzed intent and retrieving the required information.

[1046] "Emotion analysis engine" is a general term for software or systems that analyze a user's voice or text data and identify the user's emotional state.

[1047] "Means for analyzing emotions" refers to a device or module that uses an emotion analysis engine to perform processing to analyze the user's emotional state.

[1048] The "means for generating an answer" refers to a device or module for generating an appropriate answer to the user based on the acquired information and emotional information.

[1049] "Means for converting generated answers into voice data" refers to a device or module for converting generated text answers into voice data.

[1050] "Means for playing back audio data to the user" refers to a device or module that plays back the generated audio data in order to deliver it to the user.

[1051] "Biographical information" is a general term for data, facts, quotes, and other information relating to historical or famous figures.

[1052] "Cloud-based services" refer to remote computing resources, data storage, and software services delivered over the internet.

[1053] MODE FOR CARRYING OUT THE INVENTION

[1054] System Overview

[1055] This invention is an interactive learning system that allows users to ask questions using voice input and provides emotionally appropriate answers based on information about the historical figures in question. The system converts the voice input into text, analyzes the user's intention using a natural language processing engine, and retrieves corresponding information from a database. It also has the function of analyzing the user's emotions using an emotion analysis engine, generating answers in an appropriate tone, and playing them back as voice.

[1056] Hardware and software used

[1057] 1. Hardware:

[1058] Smartphones and smart speakers: Devices for voice input and playback.

[1059] Server: A central processing unit that searches the database and generates answers.

[1060] 2. Software:

[1061] Speech Recognition Module: Converts voice data into text data using the Google Speech-to-Text API.

[1062] Natural Language Processing Engine: OpenAI GPT-3 is used to analyze the intent of the text.

[1063] Sentiment Analysis Engine: Analyzes user emotions using IBM Watson Tone Analyzer.

[1064] Speech synthesis engine: Generates speech from text using Amazon Polly.

[1065] Processing flow

[1066] The specific operations of each processing step of this system are shown below.

[1067] 1. User Input

[1068] Users speak questions into their smartphones or smart speakers, for example, "Socrates, what is true knowledge?"

[1069] 2. Voice Recognition

[1070] The device captures the user's voice input and converts it into text using the Google Speech-to-Text API. Specifically, the speech "Socrates, what is true knowledge?" is converted into text "Socrates, what is true knowledge?"

[1071] 3. Text Processing

[1072] The device then sends the converted text to OpenAI GPT-3 to analyze the user's intent, during which the system extracts the keywords "Socrates" and "knowledge" to understand the gist of the question.

[1073] 4. Emotion analysis

[1074] The device sends text and voice data to the IBM Watson Tone Analyzer, which analyzes the user's emotions, identifying whether the user is calm, excited, sad, or other such emotions.

[1075] 5. Database Search

[1076] The server receives information from the natural language processing engine and emotion engine and searches a database of people, such as the quote "All I know is that I know nothing" for "Socrates." The database uses a cloud-based MySQL database.

[1077] 6. Answer generation

[1078] The server generates an appropriate response to the user based on the acquired information and the analyzed emotional information. For a calm user, the server generates a response in a calm tone.

[1079] 7. Speech Synthesis

[1080] The device converts the generated text into speech using Amazon Polly, a speech synthesis engine that adjusts tone based on sentiment analysis.

[1081] 8. Responding to the User

[1082] The terminal plays the generated voice data back to the user. For example, the user can hear a calm response such as "All I know is that I know nothing."

[1083] Specific examples

[1084] Example 1: Dialogue with Socrates

[1085] If a user asks, "Socrates, what is true knowledge?" the system behaves as follows:

[1086] 1. Speech is converted into text using speech recognition.

[1087] 2. The natural language processing engine analyzes the intent.

[1088] 3. The sentiment is analyzed by the sentiment analysis engine.

[1089] 4. The corresponding information is retrieved from the database.

[1090] 5. The answer is generated.

[1091] 6. It is converted into audio and played back to the user.

[1092] Example prompts for generative AI models

[1093] "Show me what your system would do if a user calmly asked, 'Socrates, what is true knowledge?'"

[1094] The system allows users to receive real-time emotional responses, creating a more interactive and effective learning experience.

[1095] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1096] Step 1:

[1097] Users speak questions into their smartphone or smart speaker, such as "Socrates, what is true knowledge?" This voice input is captured by the device's microphone.

[1098] Step 2:

[1099] The device sends the captured voice data to the speech recognition module, which uses the Google Speech-to-Text API. Specifically, the voice signal is sent to the server, which converts it into text data via the API: "Socrates, what is true knowledge?" The input is the user's voice, and the output is the corresponding text data.

[1100] Step 3:

[1101] The device sends the converted text data to a natural language processing engine (e.g., OpenAI GPT-3). The natural language processing engine extracts important keywords and contextual information from the text data and analyzes the user's intent. The input here is the converted text data, and the output is the analyzed user's intent information. Specifically, the keywords "Socrates" and "knowledge" are extracted from the text "Socrates, what is true knowledge?"

[1102] Step 4:

[1103] The device simultaneously sends text and voice data to an emotion analysis engine (e.g., IBM Watson Tone Analyzer). The emotion analysis engine analyzes the tone of the voice and the content of the text to identify the user's emotional state. The input is text and voice data, and the output is the user's emotional information. Specific emotions that can be identified include calm, excitement, and sadness. In this example, the user is analyzed as calm.

[1104] Step 5:

[1105] The server receives information from the natural language processing engine and sentiment analysis engine and searches the database. The database uses a cloud-based MySQL database. In this step, related information in the database is searched for based on the analyzed keywords "Socrates" and "knowledge" to obtain the appropriate answer. The input is the analyzed keywords and sentiment information, and the output is the obtained information. For example, the search result obtained is "All I know is that I know nothing."

[1106] Step 6:

[1107] The server generates an appropriate answer for the user based on the acquired information and emotional information. Since the answer needs to be in a calm tone, the server generates a calm tone text such as "All I know is that I know nothing." The input is the acquired information and emotional information, and the output is the generated answer text.

[1108] Step 7:

[1109] The device converts the generated text data into voice data. Amazon Polly is used here. The voice synthesis engine adjusts the tone of the voice based on the results of emotion analysis. The input is the generated text data, and the output is voice data with a tone that corresponds to the emotion. Specifically, voice is generated in a calm tone.

[1110] Step 8:

[1111] The terminal plays the generated voice data to the user. The voice played is a calm response saying, "All I know is that I know nothing." The user can hear this aloud. The input is the generated voice data, and the output is the voice played to the user.

[1112] (Application example 2)

[1113] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1114] In traditional brick-and-mortar stores, it has been difficult to provide accurate and emotionally appropriate information to customer questions. Furthermore, when historical background or detailed knowledge is required, the lack of knowledge and inconsistency of responses from store staff can be problematic. This can lead to a poor customer experience.

[1115] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1116] In this invention, the server includes means for analyzing the user's emotions, means for referencing the user's emotion data when generating the answer, and means that is installed in the guide robot and provides services to customers in the physical store. This enables flexible and accurate responses according to the customer's emotions and realizes the provision of consistent and detailed information in response to the question.

[1117] "Voice input" is a means by which a user provides data to a system using speech.

[1118] "Text conversion" is the process of converting audio data into text data.

[1119] A "natural language processing engine" is software that performs semantic analysis on text data and understands the user's intent.

[1120] "Database search" is an operation for searching information in a database based on the analyzed intent and obtaining the required data.

[1121] "Answer generation" is the process of creating an appropriate answer to a user's question based on the acquired information.

[1122] "Audio data conversion" is the process of converting the generated text data into an audio file and letting the user listen to it.

[1123] "Emotion analysis" is the process of evaluating and determining a user's emotional state from their voice or text.

[1124] "Response playback" is the process of playing back the generated voice data to the user.

[1125] A "guide robot" is a robot that is installed in physical stores and other locations to act as a guide and assist customers.

[1126] A "brick and mortar store" is a store that offers goods or services in a physical location.

[1127] This invention describes an embodiment of a system that allows a user to voice-input a question to a historical figure designated by the user and provides an answer to that question in an interactive format. This system has the functions of converting the voice input into text, analyzing the intent of the question using a natural language processing engine, retrieving information from a database, generating an answer, and playing it back to the user in voice. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to achieve more human-like dialogue.

[1128] Hardware and software used

[1129] Hardware:

[1130] Microphone (for voice input)

[1131] Speaker (for audio output)

[1132] Guidance robots (installed in brick-and-mortar stores for customer service)

[1133] software:

[1134] speech_recognition library (speech to text conversion)

[1135] pyttsx3 library (text-to-speech synthesis)

[1136] QA model (question answering) from the transformers library

[1137] emotion_recognition library (emotion analysis)

[1138] Processing Overview

[1139] 1. User Input:

[1140] The user can input questions to the guide robot by voice through a microphone. For example, the user can ask, "What time does this store open?"

[1141] 2. Speech Recognition:

[1142] The device captures the user's voice input and uses the speech_recognition library to convert the voice data into text, such as "What time does this store open?"

[1143] 3. Text Analysis:

[1144] The device sends the text data obtained by speech recognition to the natural language processing engine of the transformers library to analyze the user's intent. The natural language processing engine analyzes keywords and contextual information from the text data and generates structured data.

[1145] 4. Emotion analysis:

[1146] The device sends text and voice data to the emotion_recognition library to analyze the user's emotions. The emotion engine identifies the user's emotions from the tone of the voice and the content of the text. For example, it analyzes that the user is calm.

[1147] 5. Database Search:

[1148] The server receives information from the natural language processing engine and emotion engine and searches a database containing information about historical figures, obtaining information such as "This store has been in business since 1950."

[1149] 6. Answer generation:

[1150] The server generates an appropriate response to the user based on the acquired information and the analysis results of the emotion engine. For example, if the user is calm, it generates a response in a normal tone.

[1151] 7. Speech synthesis:

[1152] The device converts the generated text data into speech data, using the pyttsx3 library to convert the generated text into natural-sounding speech and adjust the tone based on the results of sentiment analysis.

[1153] 8. Response to the user:

[1154] The device plays the generated voice data to the user through a speaker, and the user can hear a response such as "This store has been in business since 1950" in an appropriate emotional tone.

[1155] Examples of prompt statements

[1156] For example, if a customer visits a store with their child and asks the robot, "What time does this store open?", the following prompt sentence will be generated:

[1157] User: How long has this store been open?

[1158] Emotion: neutral

[1159] A: This store has been in business since 1950.

[1160] This system will improve the customer experience in-store and enable accurate and flexible information provision.

[1161] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1162] Step 1:

[1163] User Input

[1164] The user inputs a question to the guide robot by voice through a microphone. For example, "What time does this store open?" The input of this step is the user's voice, and the output is voice data.

[1165] Step 2:

[1166] Voice Recognition

[1167] The device captures the user's voice input and converts it to text using the speech_recognition library. For example, the converted text might be, "What time does this store open?" The input for this step is voice data, and the output is text data.

[1168] Step 3:

[1169] Text analytics

[1170] The device sends the text data obtained by speech recognition to the natural language processing engine of the transformers library to analyze the user's intent. The natural language processing engine analyzes keywords and contextual information from the text data and generates structured data. The input of this step is text data, and the output is the analyzed structured data.

[1171] Step 4:

[1172] Emotion analysis

[1173] The device sends text and audio data to the emotion_recognition library to analyze the user's emotions. The emotion engine identifies the user's emotions from the tone of the voice and the content of the text. For example, it identifies emotions such as calm, excitement, and sadness. The input of this step is text and audio data, and the output is the user's emotional state.

[1174] Step 5:

[1175] Database search

[1176] The server receives information from the natural language processing engine and the emotion engine and searches a database containing information about historical figures. For example, it can obtain information such as "We have been in business since 1950." The input of this step is the parsed structured data and emotion data, and the output is information as search results.

[1177] Step 6:

[1178] Answer generation

[1179] The server generates an appropriate response to the user based on the acquired information and the analysis results of the emotion engine. For example, if the user is calm, it generates a response in a normal tone, such as "This store has been in business since 1950." The input for this step is the search result information and emotion data, and the output is the generated response text.

[1180] Step 7:

[1181] Speech synthesis

[1182] The device converts the generated text data into speech data using the pyttsx3 library. The tone is also adjusted based on the results of sentiment analysis. The input of this step is the response text, and the output is speech data.

[1183] Step 8:

[1184] Responding to the user

[1185] The terminal plays the generated voice data to the user through a speaker. The user can hear, for example, a response such as "This store has been in business since 1950" in an appropriate emotional tone. The input of this step is the voice data, and the output is audio information that the user can hear.

[1186] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1187] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1188] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1189] [Fourth embodiment]

[1190] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1191] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1192] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1193] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1194] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1195] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1196] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1197] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1198] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1199] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1200] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1201] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1202] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1203] System Overview

[1204] The present invention is a system that allows users to interactively learn information about a historical figure they specify. This system includes means for the user to input voice, convert the voice input into text data, and perform analysis, database search, answer generation, speech synthesis, and playback using a natural language processing engine. The specific processing of each subsystem is described below.

[1205] Process Overview

[1206] 1. User Input

[1207] Users can speak a question into their smartphone or smart speaker, for example, "Socrates, what is true knowledge?"

[1208] 2. Voice Recognition

[1209] The device captures the user's voice input with a microphone and converts the speech into text data. Cloud services such as the Google Speech-to-Text API and Amazon Transcribe can be used for speech recognition. The converted text data will read, "Socrates, what is true knowledge?"

[1210] 3. Text Processing

[1211] The device sends the text data obtained by speech recognition to a natural language processing (NLP) engine, which analyzes the user's intent. The NLP engine extracts keywords and contextual information from the text data and generates structured data. For example, it generates structured data such as "Socrates (great person)" and "true knowledge (question content)."

[1212] 4. Database Search

[1213] The server receives the structured data and searches a database containing information about great people, retrieving, for example, a famous quote by Socrates: "All I know is that I know nothing."

[1214] 5. Answer generation

[1215] The server uses information from the database to generate an appropriate response for the user, imitating the tone and style of a great man, resulting in text of the form "All I know is that I know nothing."

[1216] 6. Speech Synthesis

[1217] The device receives text data from the server and sends it to a speech synthesis engine, which converts it into voice data. Services such as Google Text-to-Speech and Amazon Polly can be used for voice synthesis. The generated voice data sounds like Socrates is actually speaking.

[1218] 7. Responding to the User

[1219] The device then plays the generated audio data back to the user, who can hear a response such as, "All I know is that I know nothing." This process allows the user to interact with historical figures, providing a rich learning experience.

[1220] Specific examples

[1221] Example 1: Dialogue with Socrates

[1222] 1. User Input: The user asks, "Socrates, what is true knowledge?"

[1223] 2. Speech recognition: The device converts speech into text.

[1224] 3. Text processing: The device sends the text to a natural language processing engine to analyze the intent.

[1225] 4. Database lookup: The server searches the database and retrieves the appropriate information.

[1226] 5. Answer Generation: The server generates the answer.

[1227] 6. Speech synthesis: The device converts the generated text into audio data.

[1228] 7. Respond to the user: The terminal plays the audio data to the user.

[1229] In this way, the system provides users with an effective and interactive learning environment by linking together the various means. This invention enables users to acquire in-depth knowledge about historical figures in an interactive format.

[1230] The processing flow will be explained below.

[1231] Step 1:

[1232] Users speak into their smartphone or smart speaker to ask a question to a designated historical figure, for example, "Socrates, what is true knowledge?"

[1233] Step 2:

[1234] The device captures voice input from the user with a microphone and sends this voice data to a speech recognition module, which converts the voice data into text. The converted text is "Socrates, what is true knowledge?"

[1235] Step 3:

[1236] The device sends the text data obtained by speech recognition to a natural language processing engine. The natural language processing engine extracts keywords and contextual information from the text data and analyzes the user's intent. For example, it generates structured data such as "Socrates (great person)" and "true knowledge (question content)."

[1237] Step 4:

[1238] The terminal transmits the parsed structured data to the server.

[1239] Step 5:

[1240] The server receives the structured data and searches a database containing information about great people, such as Socrates' famous quote, "All I know is that I know nothing."

[1241] Step 6:

[1242] The server generates a textual response to the user based on the information retrieved from the database, reflecting the tone and style of the great man.

[1243] Step 7:

[1244] The server returns the generated text data to the terminal.

[1245] Step 8:

[1246] The device then sends the received text data to a speech synthesis engine, which converts it into voice data. The speech synthesis engine uses Google Text-to-Speech or Amazon Polly to convert the text into natural-sounding speech.

[1247] Step 9:

[1248] The terminal plays the generated voice data to the user, who can hear the answer, for example, "All I know is that I know nothing."

[1249] These are the specific steps the system takes to provide information about historical figures in response to user voice questions. This process allows the user to learn through dialogue with the designated historical figures.

[1250] Example 1

[1251] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1252] In conventional systems, users must rely on searching for information in books or websites when learning about historical figures, which does not provide an interactive learning experience. Furthermore, users without specialized knowledge have difficulty assessing the authenticity and appropriate interpretation of the information. Furthermore, while systems that use voice input and voice output are common, few systems effectively combine advanced natural language processing and database search.

[1253] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1254] In this invention, the server includes means for receiving voice input from a user, means for converting the voice input into text, means for sending the converted text to a natural language processing engine and analyzing the user's intent, means for searching an information storage area and retrieving corresponding information, means having a generative AI model for generating answers based on the retrieved information, means for converting the generated answers into voice data, and means for playing the voice data to the user, thereby enabling the user to interactively converse with historical figures, obtain reliable information via voice, and enjoy a rich learning experience.

[1255] "Voice input" is a method by which a user communicates information or instructions to a system via voice.

[1256] "Convert to text" is the process of converting audio data into text data.

[1257] A "natural language processing engine" is a computer program that analyzes text data and understands its meaning.

[1258] "Information storage area" refers to the database or storage system where data is saved.

[1259] A "generative AI model" is an artificial intelligence model that generates appropriate answers to users based on acquired information.

[1260] "Audio Data" means data digitally represented in audio format.

[1261] "Playing to the user" refers to the process of making the generated audio data audible to the user through an output device such as a speaker.

[1262] "Distributed delivery service" refers to a cloud service that is delivered over the Internet and that users can access from multiple locations.

[1263] System Overview

[1264] The present invention provides a system that allows a user to interactively learn information about an important person designated by the user. The system includes means for speech input, text conversion, natural language processing, database search, answer generation, speech synthesis, and speech playback.

[1265] Program processing details

[1266] User Input

[1267] The user uses a voice input device (e.g., a smartphone or a smart speaker) to input a question by voice, for example, saying, "Tell me about a particular important person."

[1268] Voice Recognition

[1269] The device captures the user's voice input through a microphone and converts it into text using a speech recognition service such as the Google Speech-to-Text API or Amazon Transcribe. For example, the speech might be converted into text like, "Tell me about a specific important person."

[1270] Text Processing

[1271] The device sends the text data obtained by voice recognition to a natural language processing engine. The natural language processing engine (for example, Google Cloud Natural Language API or IBM Watson Natural Language Understanding) analyzes the text data and understands the user's intent. The analysis results in data such as "specific important person (target)" and "please tell me (request)."

[1272] Database search

[1273] The server receives the generated analysis data and searches an information storage area (for example, a MySQL or PostgreSQL database). As a result of the search, information about a specific important person is obtained. For example, famous quotes and life information about the specific important person are obtained.

[1274] Answer generation

[1275] The server generates an appropriate answer for the user based on the information retrieved from the database. It uses a generative AI model (such as OpenAI's GPT-3 or BERT) to generate a response text based on the retrieved information. For example, it generates text in the format "A certain important person left behind the famous quote XX."

[1276] Speech synthesis

[1277] The device receives text data from the server, sends it to a speech synthesis engine, and converts it into voice data. Services such as Google Text-to-Speech and Amazon Polly are used for voice synthesis. The generated voice data is synthesized to sound like a specific important person is actually speaking.

[1278] Responding to the user

[1279] The device then plays the generated voice data back to the user, who can hear a voice response such as, "A particular important person left behind the famous quote XX." Through this process, the user can interactively learn about important people.

[1280] Specific examples

[1281] Example 1: Interaction with a specific important person

[1282] 1. User Input: The user asks, "Tell me about a particular important person."

[1283] 2. Speech recognition: The device converts the speech into text using the Google Speech-to-Text API. The resulting text is "Tell me about a specific important person."

[1284] 3. Text processing: The device sends the text to the Google Cloud Natural Language API, which analyzes the intent and generates structured data.

[1285] 4. Database lookup: The server uses MySQL to lookup the database and retrieve information about a particular VIP.

[1286] 5. Answer generation: The server uses GPT-3 to generate an answer, creating the text "A certain important person left behind the famous quote XX."

[1287] 6. Text-to-speech: Your device will convert text to speech using Google Text-to-Speech.

[1288] 7. Response to the user: The terminal plays the audio data to the user, and the user hears, "A certain important person left behind the famous quote XX."

[1289] Prompt Sentence Examples

[1290] "Ask questions about important historical figures. For example, 'Tell me about a particular important person.'"

[1291] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1292] Step 1:

[1293] A user uses a voice input device (such as a smartphone or smart speaker) to input a question by voice. For example, they might say, "Tell me about a specific important person." At this stage, the input is voice data and has not yet been converted into text data.

[1294] Step 2:

[1295] The device captures the user's voice input through a microphone. It uses a voice recognition service (for example, Google Speech-to-Text API) to convert the voice data into text data. Specifically, it sends the voice data to a cloud service and receives the returned text data. The input is voice data, and the output is text data, such as "Tell me about a specific important person."

[1296] Step 3:

[1297] The device sends the acquired text data to a natural language processing engine (for example, Google Cloud Natural Language API) for analysis. Specifically, it performs syntactic analysis of the text and extracts keywords to understand the user's intent. The input is text data, and the output is data indicating the intent, such as "specific important person (target)" and "please tell me (request)."

[1298] Step 4:

[1299] The server receives the intent data sent from the device and searches an information storage area (for example, a MySQL database). It generates a search query and queries the database to obtain information about a specific important person. The specific operation is to execute an SQL query and extract the relevant records. The input is the intent data, and the output is information data as a search result.

[1300] Step 5:

[1301] The server generates an answer using a generative AI model (e.g., GPT-3) based on the information data obtained from the search results. Specifically, the process involves inputting the information data into the AI ​​model and outputting a corresponding text answer. The input is the information data, and the output is a text answer such as "A certain important person left behind the famous quote XX."

[1302] Step 6:

[1303] The device sends the answer text received from the server to a speech synthesis engine (e.g., Google Text-to-Speech) and converts it into voice data. During this process, it calls an API to convert text data into voice format and receives the result. The input is the answer text, and the output is voice data.

[1304] Step 7:

[1305] The terminal plays the generated voice data to the user. At this stage, the speaker is used to output the voice and convey the answer to the user. The specific operation is to pass the voice data to the playback device and output it. The input is the voice data, and the output is the voice information that the user hears.

[1306] (Application example 1)

[1307] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1308] Conventional learning systems have made it difficult for users to learn detailed information about historical figures of interest through a direct dialogue. Furthermore, users cannot easily obtain the information they are looking for through voice, making it difficult to provide an interactive learning experience. Furthermore, there are limitations to the accuracy and realism of the generated answers, and improvements are needed to increase user satisfaction.

[1309] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1310] In this invention, the server includes means for receiving voice input from a user, means for converting the voice input into text, means for sending the converted text to a natural language processing engine and analyzing the user's intent, means for searching a database based on the analyzed intent and obtaining corresponding information, means for generating an answer based on the obtained information, means for converting the generated answer into voice data, means for playing the voice data to the user, means for displaying the generated answer as text data, and means for generating an answer in the form of a dialogue between a predetermined specific person. This allows a user to easily learn in the form of a dialogue with a historical figure through voice input, thereby providing a highly realistic, interactive learning experience.

[1311] A "means for receiving voice input from a user" is a device capable of capturing voice data such as questions or commands uttered by a user.

[1312] The "means for converting voice input into text" is a technology that converts captured voice data into text information, and utilizes voice recognition technology.

[1313] "Means of sending the converted text to a natural language processing engine and analyzing the user's intent" is a technology that uses advanced algorithms to analyze text data to understand the intent of a user's questions or commands.

[1314] The "means for searching a database based on the analyzed intent and retrieving corresponding information" is a system that has the function of searching and retrieving highly relevant information from within a database.

[1315] The "means for generating an answer based on acquired information" is a system that has the function of constructing an appropriate answer to a user's question based on information acquired from a database.

[1316] The "means for converting the generated answer into voice data" is a technique for converting the generated text-format answer into data that can be reproduced by voice.

[1317] The "means for reproducing the audio data to the user" refers to a reproducing device for allowing the user to listen to the converted audio data.

[1318] The "means for displaying the generated answer as text data" is a device that has the function of displaying the answer not only as voice but also as text on a screen.

[1319] "Means in which the generated answers are in the form of a dialogue by a specific person" refers to a technology in which the answers are likened to a specific historical figure and are provided in the same tone and style as that figure.

[1320] System Overview

[1321] The system that realizes this application example integrates the functions of voice input, voice recognition, natural language processing, database search, answer generation, voice synthesis, result playback, and text display. The specific processing of each subsystem is explained below.

[1322] Hardware and software used

[1323] Hardware:

[1324] microphone

[1325] speaker

[1326] Displays (smartphones, etc.)

[1327] High-performance devices (smartphones, servers)

[1328] software:

[1329] Python: A programming language for implementing the overall logic of a program

[1330] speech_recognition package: A library for converting voice input to text using the Google Speech-to-Text API.

[1331] transformers package: A library for analyzing user intent and generating appropriate answers using Hugging Face's natural language processing model.

[1332] gTTS (Google Text-to-Speech): A library for converting generated text into audio data.

[1333] playsound: A library for playing converted audio data

[1334] Details of data processing and calculation

[1335] 1. Voice Input and Recognition

[1336] The user inputs a question by voice through the microphone. For example, in the scenario of "asking a question to a specific historical figure," the user might say, "Historical figure, please tell me your famous quote." This is captured by the microphone and converted into text data using the speech_recognition package.

[1337] 2. Natural Language Processing

[1338] The device sends the converted text to a natural language processing engine, which analyzes the user's question, extracts keywords and contextual information, and generates structured data using Hugging Face's transformers package.

[1339] 3. Database search and answer generation

[1340] The server receives the parsed structured data and searches for relevant information in a database. Based on the information obtained, it generates an appropriate answer to the user's question. In this process, the generated text is modeled after the tone and writing style of famous people, so that the answer is provided in a conversational style that is specific to the person.

[1341] 4. Speech synthesis and playback

[1342] The device sends the generated text data to a speech synthesis engine and converts it into voice data. This process uses the gTTS library. The generated voice data is played back to the user using the playsound library, and is also displayed as text data on the screen.

[1343] Specific examples

[1344] For example, consider the case where a user launches an application and asks, "Napoleon, tell me about the importance of strategy." In this case, the system performs speech recognition and converts the speech into text data. It then uses a natural language processing engine to analyze the question and retrieves an appropriate answer about Napoleon from a database. The answer generated, "Without strategy, no army can achieve victory," is converted into audio data and played back to the user. This entire process occurs in real time, providing the user with an interactive learning experience.

[1345] Prompt Sentence Examples

[1346] To generate a question for a particular historical figure, a prompt like the following can be used:

[1347] Prompt: "Generate a dialogue about the importance of strategy for the historical figure Napoleon."

[1348] Thus, the present invention utilizes advanced speech recognition and natural language processing techniques to provide a system that allows users to interactively learn about historical figures, thereby providing a richer learning experience for users.

[1349] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1350] Step 1:

[1351] Capture audio input

[1352] The user can enter questions or commands by voice through a microphone. The user's voice is then captured as an analog signal by the microphone. For example, a question such as "What is your most famous quote, historical figure?" can be captured as user input.

[1353] Step 2:

[1354] Speech-to-text conversion

[1355] The device converts the captured voice data to text using the speech_recognition package. In this process, speech recognition technology is applied to digitize the voice signal, which is then converted to text data. The input is voice data, and the output is the corresponding text data.

[1356] Step 3:

[1357] Intention analysis using natural language processing

[1358] The device sends the converted text data to a natural language processing engine, which uses the transformers package to analyze the user's intent. Specifically, it extracts keywords and contextual information from the text data and generates structured data. The input of this step is the text data, and the output is the analyzed structured data.

[1359] Step 4:

[1360] Database search

[1361] The server receives the parsed structured data and uses it to search a database containing information about historical figures, and executes queries to retrieve relevant information. The input is the structured data, and the output is relevant information.

[1362] Step 5:

[1363] Answer generation

[1364] The server generates appropriate answers for the user based on information retrieved from the database. It uses a generative AI model to generate dialogue-style answers from specific historical figures. The generated answers mimic the tone and writing style of the specific figures. The input is the retrieved information, and the output is the generated text answer.

[1365] Step 6:

[1366] Conversion to audio data

[1367] The device converts the generated text data into speech data using the gTTS library. The text is sent to a speech synthesis engine, which generates corresponding speech data. The input is the generated text data, and the output is speech data.

[1368] Step 7:

[1369] Playback of audio data and display of text

[1370] The device plays the generated audio data using the playsound library and simultaneously displays it as text data on the display. In this step, the user hears the answer aloud and the text is also displayed on the screen. The input is audio data and text data, and the output is audio playback and text display.

[1371] This series of processes allows users to ask questions about historical figures by voice and receive answers in audio and text, providing an interactive and realistic learning experience.

[1372] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1373] System Overview

[1374] This invention is a system that allows users to voice-input questions to historical figures specified by the user and provides answers to those questions in an interactive format. This system converts the voice input into text, analyzes the intent of the question using a natural language processing engine, retrieves information from a database, generates answers, and plays them back to the user in voice. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it achieves more human-like dialogue.

[1375] Process Overview

[1376] 1. User Input

[1377] Users speak questions into their smartphones or smart speakers, for example, "Socrates, what is true knowledge?"

[1378] 2. Voice Recognition

[1379] The device captures voice input from the user and sends this voice data to a voice recognition module, which converts the voice data into text data. The converted text might be, for example, "Socrates, what is true knowledge?"

[1380] 3. Text Processing

[1381] The device sends the text data obtained by speech recognition to a natural language processing engine, which analyzes the user's intent. The natural language processing engine then analyzes the text data for keywords and contextual information to generate structured data.

[1382] 4. Emotion analysis

[1383] The device sends text data and voice data to the emotion engine, which analyzes the user's emotions. The emotion engine identifies the user's emotions from the tone of the voice and the content of the text. For example, it analyzes the user's emotional state, such as whether they are excited, sad, or calm.

[1384] 5. Database Search

[1385] The server receives information from the natural language processing engine and emotion engine and searches a database containing information about great people. For example, it retrieves a famous quote by Socrates: "All I know is that I know nothing."

[1386] 6. Answer generation

[1387] The server generates an appropriate response to the user based on the information retrieved from the database and the analysis results of the emotion engine. For example, if the user is excited, it generates a response in a calm tone.

[1388] 7. Speech Synthesis

[1389] The device converts the generated text data into voice data. The speech synthesis engine converts the generated text into natural-sounding speech and adjusts the tone based on the results of emotion analysis. For example, if the user is sad, the device will generate a gentler tone of voice.

[1390] 8. Responding to the User

[1391] The terminal plays the generated voice data to the user, who can hear a response such as "All I know is that I know nothing" spoken in an appropriate emotional tone.

[1392] Specific examples

[1393] Example 1: Dialogue with Socrates

[1394] 1. User Input: The user asks, "Socrates, what is true knowledge?"

[1395] 2. Speech recognition: The device converts speech into text.

[1396] 3. Text processing: The device sends the text to a natural language processing engine to analyze the intent.

[1397] 4. Emotion analysis: The device analyzes the user's emotions. For example, it determines that the user is calm.

[1398] 5. Database lookup: The server searches the database and retrieves the appropriate information.

[1399] 6. Answer generation: The server generates an answer in a calm tone based on the results of sentiment analysis.

[1400] 7. Speech synthesis: The device converts the generated text into audio data.

[1401] 8. Response to the user: The terminal plays back in a calm tone, "All I know is that I know nothing."

[1402] In this way, each of the methods in this system works together to provide users with an effective and interactive learning environment. Furthermore, by combining it with an emotion engine, flexible responses based on the user's emotions become possible, resulting in more human-like dialogue.

[1403] The processing flow will be explained below.

[1404] Step 1:

[1405] Users can speak into their smartphone or smart speaker to ask questions to historical figures, such as, "Socrates, what is true knowledge?"

[1406] Step 2:

[1407] The device captures voice input from the user with a microphone and sends this voice data to a voice recognition module. The voice recognition module converts the voice data into text data. The converted text becomes, "Socrates, what is true knowledge?"

[1408] Step 3:

[1409] The device sends the text data obtained by speech recognition to a natural language processing (NLP) engine. The NLP engine extracts keywords and contextual information from the text data and analyzes the user's intent. For example, it generates structured data such as "Socrates (great man)" and "true knowledge (question content)."

[1410] Step 4:

[1411] The device sends the text data and voice data to the emotion engine, which analyzes the user's emotion from the voice tone and text content. For example, it identifies the user's emotional state, such as calm, excited, or sad.

[1412] Step 5:

[1413] The device transmits the intent-analyzed structured data and the results of emotion analysis to the server.

[1414] Step 6:

[1415] The server receives the structured data and searches a database containing information about great people. For example, based on the keywords "Socrates" and "true knowledge," it retrieves Socrates' famous quote, "All I know is that I know nothing."

[1416] Step 7:

[1417] The server generates appropriate responses to the user based on the information retrieved from the database and the results of emotion analysis. For example, if the server determines that the user is calm, it generates responses that mimic the tone of Socrates, who has a calm mind.

[1418] Step 8:

[1419] The server transmits the generated text data to the terminal.

[1420] Step 9:

[1421] The device sends the text data received from the server to a speech synthesis engine, which converts the text data into speech data and adjusts the tone of the speech based on the results of emotion analysis. For example, if the user is sad, the device generates a response in a gentle tone.

[1422] Step 10:

[1423] The terminal plays the generated voice data to the user, who can hear a response such as "All I know is that I know nothing" spoken in an appropriate emotional tone.

[1424] These are the specific processing steps required to provide information about historical figures in response to a user's voice question. This process allows the user to learn through a dialogue with the designated historical figure, and by combining it with an emotion engine, it is possible to achieve a more human-like dialogue.

[1425] Example 2

[1426] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1427] In modern learning systems, it is difficult for users to ask historical figures questions and receive interactive, emotionally relevant answers. Traditional systems often provide cold responses, which can discourage users from learning. Furthermore, the quality of voice-based engines and the accuracy of natural language processing are limited, leaving issues with the accuracy and persuasiveness of the information.

[1428] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving a voice input, means for converting the voice input into text, means for sending the converted text to a natural language processing engine and analyzing the intention, means for searching a database based on the analyzed intention and acquiring corresponding information, means for analyzing the user's emotions using an emotion analysis engine, means for generating an answer based on the acquired information and the analyzed emotion information, means for converting the generated answer into voice data, and means for playing the voice data to the user. This makes it possible to provide an interactive answer according to the user's emotions toward the historical figure about whom the question was asked.

[1429] "Means for receiving voice input" is a general term for devices and modules that capture voices uttered by the user and input them into the system.

[1430] "Means for converting voice input to text" refers to a device or module for analyzing captured voice data and converting it into corresponding text data.

[1431] A "natural language processing engine" is a general term for software or systems that analyze text data and extract user intent and keywords.

[1432] "Means for analyzing intent" refers to a device or module that uses a natural language processing engine to perform processing to analyze the intent of a question from input text data.

[1433] "Means for searching a database and retrieving corresponding information" refers to a device or module for searching related information in a database based on the analyzed intent and retrieving the required information.

[1434] "Emotion analysis engine" is a general term for software or systems that analyze a user's voice or text data and identify the user's emotional state.

[1435] "Means for analyzing emotions" refers to a device or module that uses an emotion analysis engine to perform processing to analyze the user's emotional state.

[1436] The "means for generating an answer" refers to a device or module for generating an appropriate answer to the user based on the acquired information and emotional information.

[1437] "Means for converting generated answers into voice data" refers to a device or module for converting generated text answers into voice data.

[1438] "Means for playing back audio data to the user" refers to a device or module that plays back the generated audio data in order to deliver it to the user.

[1439] "Biographical information" is a general term for data, facts, quotes, and other information relating to historical or famous figures.

[1440] "Cloud-based services" refer to remote computing resources, data storage, and software services delivered over the internet.

[1441] MODE FOR CARRYING OUT THE INVENTION

[1442] System Overview

[1443] This invention is an interactive learning system that allows users to ask questions using voice input and provides emotionally appropriate answers based on information about the historical figures in question. The system converts the voice input into text, analyzes the user's intention using a natural language processing engine, and retrieves corresponding information from a database. It also has the function of analyzing the user's emotions using an emotion analysis engine, generating answers in an appropriate tone, and playing them back as voice.

[1444] Hardware and software used

[1445] 1. Hardware:

[1446] Smartphones and smart speakers: Devices for voice input and playback.

[1447] Server: A central processing unit that searches the database and generates answers.

[1448] 2. Software:

[1449] Speech Recognition Module: Converts voice data into text data using the Google Speech-to-Text API.

[1450] Natural Language Processing Engine: OpenAI GPT-3 is used to analyze the intent of the text.

[1451] Sentiment Analysis Engine: Analyzes user emotions using IBM Watson Tone Analyzer.

[1452] Speech synthesis engine: Generates speech from text using Amazon Polly.

[1453] Processing flow

[1454] The specific operations of each processing step of this system are shown below.

[1455] 1. User Input

[1456] Users speak questions into their smartphones or smart speakers, for example, "Socrates, what is true knowledge?"

[1457] 2. Voice Recognition

[1458] The device captures the user's voice input and converts it into text using the Google Speech-to-Text API. Specifically, the speech "Socrates, what is true knowledge?" is converted into text "Socrates, what is true knowledge?"

[1459] 3. Text Processing

[1460] The device then sends the converted text to OpenAI GPT-3 to analyze the user's intent, during which the system extracts the keywords "Socrates" and "knowledge" to understand the gist of the question.

[1461] 4. Emotion analysis

[1462] The device sends text and voice data to the IBM Watson Tone Analyzer, which analyzes the user's emotions, identifying whether the user is calm, excited, sad, or other such emotions.

[1463] 5. Database Search

[1464] The server receives information from the natural language processing engine and emotion engine and searches a database of people, such as the quote "All I know is that I know nothing" for "Socrates." The database uses a cloud-based MySQL database.

[1465] 6. Answer generation

[1466] The server generates an appropriate response to the user based on the acquired information and the analyzed emotional information. For a calm user, the server generates a response in a calm tone.

[1467] 7. Speech Synthesis

[1468] The device converts the generated text into speech using Amazon Polly, a speech synthesis engine that adjusts tone based on sentiment analysis.

[1469] 8. Responding to the User

[1470] The terminal plays the generated voice data back to the user. For example, the user can hear a calm response such as "All I know is that I know nothing."

[1471] Specific examples

[1472] Example 1: Dialogue with Socrates

[1473] If a user asks, "Socrates, what is true knowledge?" the system behaves as follows:

[1474] 1. Speech is converted into text using speech recognition.

[1475] 2. The natural language processing engine analyzes the intent.

[1476] 3. The sentiment is analyzed by the sentiment analysis engine.

[1477] 4. The corresponding information is retrieved from the database.

[1478] 5. The answer is generated.

[1479] 6. It is converted into audio and played back to the user.

[1480] Example prompts for generative AI models

[1481] "Show me what your system would do if a user calmly asked, 'Socrates, what is true knowledge?'"

[1482] The system allows users to receive real-time emotional responses, creating a more interactive and effective learning experience.

[1483] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1484] Step 1:

[1485] Users speak questions into their smartphone or smart speaker, such as "Socrates, what is true knowledge?" This voice input is captured by the device's microphone.

[1486] Step 2:

[1487] The device sends the captured voice data to the speech recognition module, which uses the Google Speech-to-Text API. Specifically, the voice signal is sent to the server, which converts it into text data via the API: "Socrates, what is true knowledge?" The input is the user's voice, and the output is the corresponding text data.

[1488] Step 3:

[1489] The device sends the converted text data to a natural language processing engine (e.g., OpenAI GPT-3). The natural language processing engine extracts important keywords and contextual information from the text data and analyzes the user's intent. The input here is the converted text data, and the output is the analyzed user's intent information. Specifically, the keywords "Socrates" and "knowledge" are extracted from the text "Socrates, what is true knowledge?"

[1490] Step 4:

[1491] The device simultaneously sends text and voice data to an emotion analysis engine (e.g., IBM Watson Tone Analyzer). The emotion analysis engine analyzes the tone of the voice and the content of the text to identify the user's emotional state. The input is text and voice data, and the output is the user's emotional information. Specific emotions that can be identified include calm, excitement, and sadness. In this example, the user is analyzed as calm.

[1492] Step 5:

[1493] The server receives information from the natural language processing engine and sentiment analysis engine and searches the database. The database uses a cloud-based MySQL database. In this step, related information in the database is searched for based on the analyzed keywords "Socrates" and "knowledge" to obtain the appropriate answer. The input is the analyzed keywords and sentiment information, and the output is the obtained information. For example, the search result obtained is "All I know is that I know nothing."

[1494] Step 6:

[1495] The server generates an appropriate answer for the user based on the acquired information and emotional information. Since the answer needs to be in a calm tone, the server generates a calm tone text such as "All I know is that I know nothing." The input is the acquired information and emotional information, and the output is the generated answer text.

[1496] Step 7:

[1497] The device converts the generated text data into voice data. Amazon Polly is used here. The voice synthesis engine adjusts the tone of the voice based on the results of emotion analysis. The input is the generated text data, and the output is voice data with a tone that corresponds to the emotion. Specifically, voice is generated in a calm tone.

[1498] Step 8:

[1499] The terminal plays the generated voice data to the user. The voice played is a calm response saying, "All I know is that I know nothing." The user can hear this aloud. The input is the generated voice data, and the output is the voice played to the user.

[1500] (Application example 2)

[1501] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1502] In traditional brick-and-mortar stores, it has been difficult to provide accurate and emotionally appropriate information to customer questions. Furthermore, when historical background or detailed knowledge is required, the lack of knowledge and inconsistency of responses from store staff can be problematic. This can lead to a poor customer experience.

[1503] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1504] In this invention, the server includes means for analyzing the user's emotions, means for referencing the user's emotion data when generating the answer, and means that is installed in the guide robot and provides services to customers in the physical store. This enables flexible and accurate responses according to the customer's emotions and realizes the provision of consistent and detailed information in response to the question.

[1505] "Voice input" is a means by which a user provides data to a system using speech.

[1506] "Text conversion" is the process of converting audio data into text data.

[1507] A "natural language processing engine" is software that performs semantic analysis on text data and understands the user's intent.

[1508] "Database search" is an operation for searching information in a database based on the analyzed intent and obtaining the required data.

[1509] "Answer generation" is the process of creating an appropriate answer to a user's question based on the acquired information.

[1510] "Audio data conversion" is the process of converting the generated text data into an audio file and letting the user listen to it.

[1511] "Emotion analysis" is the process of evaluating and determining a user's emotional state from their voice or text.

[1512] "Response playback" is the process of playing back the generated voice data to the user.

[1513] A "guide robot" is a robot that is installed in physical stores and other locations to act as a guide and assist customers.

[1514] A "brick and mortar store" is a store that offers goods or services in a physical location.

[1515] This invention describes an embodiment of a system that allows a user to voice-input a question to a historical figure designated by the user and provides an answer to that question in an interactive format. This system has the functions of converting the voice input into text, analyzing the intent of the question using a natural language processing engine, retrieving information from a database, generating an answer, and playing it back to the user in voice. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it is possible to achieve more human-like dialogue.

[1516] Hardware and software used

[1517] Hardware:

[1518] Microphone (for voice input)

[1519] Speaker (for audio output)

[1520] Guidance robots (installed in brick-and-mortar stores for customer service)

[1521] software:

[1522] speech_recognition library (speech to text conversion)

[1523] pyttsx3 library (text-to-speech synthesis)

[1524] QA model (question answering) from the transformers library

[1525] emotion_recognition library (emotion analysis)

[1526] Processing Overview

[1527] 1. User Input:

[1528] The user can input questions to the guide robot by voice through a microphone. For example, the user can ask, "What time does this store open?"

[1529] 2. Speech Recognition:

[1530] The device captures the user's voice input and uses the speech_recognition library to convert the voice data into text, such as "What time does this store open?"

[1531] 3. Text Analysis:

[1532] The device sends the text data obtained by speech recognition to the natural language processing engine of the transformers library to analyze the user's intent. The natural language processing engine analyzes keywords and contextual information from the text data and generates structured data.

[1533] 4. Emotion analysis:

[1534] The device sends text and voice data to the emotion_recognition library to analyze the user's emotions. The emotion engine identifies the user's emotions from the tone of the voice and the content of the text. For example, it analyzes that the user is calm.

[1535] 5. Database Search:

[1536] The server receives information from the natural language processing engine and emotion engine and searches a database containing information about historical figures, obtaining information such as "This store has been in business since 1950."

[1537] 6. Answer generation:

[1538] The server generates an appropriate response to the user based on the acquired information and the analysis results of the emotion engine. For example, if the user is calm, it generates a response in a normal tone.

[1539] 7. Speech synthesis:

[1540] The device converts the generated text data into speech data, using the pyttsx3 library to convert the generated text into natural-sounding speech and adjust the tone based on the results of sentiment analysis.

[1541] 8. Response to the user:

[1542] The device plays the generated voice data to the user through a speaker, and the user can hear a response such as "This store has been in business since 1950" in an appropriate emotional tone.

[1543] Examples of prompt statements

[1544] For example, if a customer visits a store with their child and asks the robot, "What time does this store open?", the following prompt sentence will be generated:

[1545] User: How long has this store been open?

[1546] Emotion: neutral

[1547] A: This store has been in business since 1950.

[1548] This system will improve the customer experience in-store and enable accurate and flexible information provision.

[1549] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1550] Step 1:

[1551] User Input

[1552] The user inputs a question to the guide robot by voice through a microphone. For example, "What time does this store open?" The input of this step is the user's voice, and the output is voice data.

[1553] Step 2:

[1554] Voice Recognition

[1555] The device captures the user's voice input and converts it to text using the speech_recognition library. For example, the converted text might be, "What time does this store open?" The input for this step is voice data, and the output is text data.

[1556] Step 3:

[1557] Text analytics

[1558] The device sends the text data obtained by speech recognition to the natural language processing engine of the transformers library to analyze the user's intent. The natural language processing engine analyzes keywords and contextual information from the text data and generates structured data. The input of this step is text data, and the output is the analyzed structured data.

[1559] Step 4:

[1560] Emotion analysis

[1561] The device sends text and audio data to the emotion_recognition library to analyze the user's emotions. The emotion engine identifies the user's emotions from the tone of the voice and the content of the text. For example, it identifies emotions such as calm, excitement, and sadness. The input of this step is text and audio data, and the output is the user's emotional state.

[1562] Step 5:

[1563] Database search

[1564] The server receives information from the natural language processing engine and the emotion engine and searches a database containing information about historical figures. For example, it can obtain information such as "We have been in business since 1950." The input of this step is the parsed structured data and emotion data, and the output is information as search results.

[1565] Step 6:

[1566] Answer generation

[1567] The server generates an appropriate response to the user based on the acquired information and the analysis results of the emotion engine. For example, if the user is calm, it generates a response in a normal tone, such as "This store has been in business since 1950." The input for this step is the search result information and emotion data, and the output is the generated response text.

[1568] Step 7:

[1569] Speech synthesis

[1570] The device converts the generated text data into speech data using the pyttsx3 library. The tone is also adjusted based on the results of sentiment analysis. The input of this step is the response text, and the output is speech data.

[1571] Step 8:

[1572] Responding to the user

[1573] The terminal plays the generated voice data to the user through a speaker. The user can hear, for example, a response such as "This store has been in business since 1950" in an appropriate emotional tone. The input of this step is the voice data, and the output is audio information that the user can hear.

[1574] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1575] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1576] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1577] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1578] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1579] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1580] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1581] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1582] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1583] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1584] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1585] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1586] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1587] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1588] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1589] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1590] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1591] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1592] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1593] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1594] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1595] The following is further disclosed regarding the above embodiment.

[1596] (Claim 1)

[1597] means for receiving voice input from a user;

[1598] means for converting said speech input into text;

[1599] means for transmitting the converted text to a natural language processing engine for analyzing the user's intent;

[1600] A means for searching a database based on the analyzed intent and obtaining corresponding information;

[1601] a means for generating a response based on the obtained information;

[1602] means for converting the generated answers into audio data;

[1603] means for playing the audio data to a user;

[1604] A system including:

[1605] (Claim 2)

[1606] 10. The system of claim 1, wherein the database includes information about historical figures.

[1607] (Claim 3)

[1608] 10. The system of claim 1, wherein the natural language processing engine utilizes a cloud-based service.

[1609] "Example 1"

[1610] (Claim 1)

[1611] means for receiving voice input from a user;

[1612] means for converting said speech input into text;

[1613] means for transmitting the converted text to a natural language processing engine for analyzing the user's intent;

[1614] A means for searching an information storage area based on the analyzed intention and acquiring corresponding information;

[1615] A means having a generative AI model that generates an answer based on the acquired information;

[1616] means for converting the generated answers into audio data;

[1617] means for playing the audio data to a user;

[1618] A system including:

[1619] (Claim 2)

[1620] 10. The system of claim 1, wherein the information storage area includes information about historical figures.

[1621] (Claim 3)

[1622] 2. The system according to claim 1, wherein the natural language processing engine utilizes a distributed provision service.

[1623] "Application Example 1"

[1624] (Claim 1)

[1625] means for receiving voice input from a user;

[1626] means for converting said speech input into text;

[1627] means for transmitting the converted text to a natural language processing engine for analyzing the user's intent;

[1628] A means for searching a database based on the analyzed intent and obtaining corresponding information;

[1629] a means for generating a response based on the obtained information;

[1630] means for converting the generated answers into audio data;

[1631] means for playing the audio data to a user;

[1632] A means for displaying the generated answers as text data;

[1633] A means for generating a response in the form of a dialogue by a specific person;

[1634] A system including:

[1635] (Claim 2)

[1636] 10. The system of claim 1, wherein the database includes information about historical figures.

[1637] (Claim 3)

[1638] 10. The system of claim 1, wherein the natural language processing engine utilizes a cloud-based service.

[1639] "Example 2: Combining Emotion Engines"

[1640] (Claim 1)

[1641] means for receiving audio input;

[1642] a means for converting voice input into text;

[1643] means for transmitting the converted text to a natural language processing engine for analyzing intent;

[1644] A means for searching a database based on the analyzed intent and obtaining corresponding information;

[1645] means for analyzing user emotions using a sentiment analysis engine;

[1646] means for generating an answer based on the acquired information and the analyzed emotion information;

[1647] means for converting the generated answers into audio data;

[1648] means for playing the audio data to a user;

[1649] A system including:

[1650] (Claim 2)

[1651] 10. The system of claim 1, wherein the information relates to a person.

[1652] (Claim 3)

[1653] 10. The system of claim 1, wherein the natural language processing engine utilizes a cloud-based service.

[1654] "Application example 2 when combining emotion engines"

[1655] (Claim 1)

[1656] means for receiving voice input from a user;

[1657] means for converting said speech input into text;

[1658] means for transmitting the converted text to a natural language processing engine for analyzing the user's intent;

[1659] A means for searching a database based on the analyzed intent and obtaining corresponding information;

[1660] a means for generating a response based on the obtained information;

[1661] means for converting the generated answers into audio data;

[1662] means for playing the audio data to a user;

[1663] means for analyzing user emotions;

[1664] means for referring to emotion data of the user when generating the answer;

[1665] A system including:

[1666] (Claim 2)

[1667] 10. The system of claim 1, wherein the database includes information about historical figures.

[1668] (Claim 3)

[1669] 10. The system of claim 1, wherein the natural language processing engine utilizes a cloud-based service.

[1670] (Claim 4)

[1671] The system according to claim 1, characterized in that the system is installed in a guide robot and provides services to customers in a physical store.

[1672] (Claim 5)

[1673] 2. The system of claim 1, wherein the emotion analysis means determines the user's emotional state based on speech recognition or text analysis. [Explanation of symbols]

[1674] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving voice input from a user; means for converting said speech input into text; means for transmitting the converted text to a natural language processing engine for analyzing the user's intent; A means for searching a database based on the analyzed intent and obtaining corresponding information; a means for generating a response based on the obtained information; means for converting the generated answers into audio data; means for playing the audio data to a user; A system including:

2. 2. The system of claim 1, wherein the database includes information about historical figures.

3. The system of claim 1 , wherein the natural language processing engine utilizes a cloud-based service.

Images