System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving a speech input of a user; means for recognizing the speech input and converting the speech input into text; means for analyzing the text and understanding an intention of the user; means for generating a response according to the intention of the user; means for converting the generated response into speech data; and means for outputting the speech data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional educational systems have difficulty in providing personalized education tailored to the needs of each individual child, and maximizing learning outcomes remains a challenge. Furthermore, there are also concerns about a lack of real-time learning support and information sharing with parents and educators. The purpose of this invention is to solve these issues, stimulate children's learning motivation, and provide effective educational support. [Means for solving the problem]

[0005] The present invention provides a system including means for receiving a user's voice input, means for recognizing the voice input and converting it into text, means for analyzing the text and understanding the user's intent, means for generating a response according to the user's intent, means for converting the generated response into voice data, and means for outputting the voice data. The system also includes means for recording the user's voice input and the generated response, means for saving the recorded voice data in cloud storage, and means for sharing the data saved in the cloud storage. Furthermore, by including voice authentication means for individually recognizing voice input, it is possible to provide more personalized learning support to each child.

[0006] "Means for receiving user voice input" refers to a microphone or other voice input device for capturing voices uttered by a user.

[0007] "Means for recognizing and converting voice input to text" refers to a speech recognition engine or software that analyzes voice data and converts it into text format.

[0008] "Means for analyzing text and understanding user intent" refers to natural language processing technology used to interpret text obtained through speech recognition and determine what the user is looking for.

[0009] "Means for generating a response according to the user's intention" refers to a generative AI model or response generation algorithm for generating an appropriate response based on the analysis results.

[0010] The "means for converting the generated response into voice data" refers to a voice synthesis engine for converting the text response into voice data.

[0011] "Means for outputting audio data" refers to a speaker or other audio output device for playing back synthesized audio data and conveying it to the user.

[0012] "Means for recording voice input and generated responses" refers to recording devices and software for digitally storing interactions with a user.

[0013] "Means for storing recorded audio data in cloud storage" refers to software and protocols for uploading and storing locally recorded data in a cloud storage service on the Internet.

[0014] "Means for sharing data stored in cloud storage" refers to access control and sharing features that allow data stored in the cloud to be shared with third parties, such as parents or educators, as needed.

[0015] "Voice authentication means for individually recognizing voice input" refers to a voice authentication technology for identifying individual users based on analyzing voice characteristics to distinguish between different users. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The present invention is an educational system that receives voice input from a user (especially a child), analyzes it, and generates an appropriate response. This system includes a series of processes that operate between a server, a terminal, and the user.

[0038] System Overview

[0039] When a user asks a question or makes a request via voice, the device captures the speech and sends it to the server. The server uses a speech recognition engine to convert the speech to text and analyzes the text. It then generates an appropriate response based on a generative AI model, converts the response into audio data, and sends it back to the device. The device then responds to the user via voice, logs the interaction, and stores it in cloud storage.

[0040] Capture and transmit audio input

[0041] Examples:

[0042] When a user speaks to the device, "What are you going to study today?", the device captures the voice input and sends the captured voice data to the server.

[0043] Speech recognition and text conversion

[0044] Examples:

[0045] The server analyzes the received voice data with a speech recognition engine and converts it into text such as "What will you study today?" This text data is sent to the next step for further analysis.

[0046] Text analysis and intent understanding

[0047] Examples:

[0048] The server uses a generative AI model to analyze the text data and understand that the user is asking a question about today's learning content.

[0049] Response Generation

[0050] Examples:

[0051] The server generates a text response such as, "Today's study will be solving math problems." The generative AI model creates advice and suggestions based on the training data.

[0052] Speech synthesis and output

[0053] Examples:

[0054] The server converts the generated text response into voice data using a speech synthesis engine, which is then sent to the device, which then tells the user, "Today's study will be solving math problems."

[0055] Recording and saving conversation logs

[0056] Examples:

[0057] The server logs all user interactions and stores them in cloud storage, where parents and educators can access the logs to track learning progress.

[0058] Voice Authentication

[0059] Examples:

[0060] The device records the user's voice during initial setup and performs voice authentication during subsequent interactions, allowing only authenticated users to use the system.

[0061] In this way, the system of the present invention provides personalized learning support through a series of steps, from voice input to text conversion, response generation, voice output, and log recording. It is also possible to share information with parents and educators, which increases the user's motivation to learn and provides effective educational support.

[0062] The processing flow will be explained below.

[0063] Step 1:

[0064] The user speaks to the device. For example, the user might say, "What will I study today?"

[0065] Step 2:

[0066] The device captures the user's voice. The device's built-in microphone records the user's voice data.

[0067] Step 3:

[0068] The device sends the captured audio data to the server, which then uploads the audio data to the server via the network.

[0069] Step 4:

[0070] The server receives the voice data. The server receives the voice data sent from the terminal.

[0071] Step 5:

[0072] The server uses a speech recognition engine to convert the voice data into text. For example, the server converts the voice data of "What will you study today?" into text format.

[0073] Step 6:

[0074] The server analyzes the text data and uses a generative AI model to analyze the content of the text and understand the user's intent.

[0075] Step 7:

[0076] The server generates a response based on the user's intent. Based on the analysis results, the server generates an appropriate text response such as "Today's study will be solving math problems."

[0077] Step 8:

[0078] The server converts the text response into audio data. The server uses a speech synthesis engine to convert the text response into audio format.

[0079] Step 9:

[0080] The server transmits the generated voice data to the terminal via the network.

[0081] Step 10:

[0082] The terminal receives the voice data. The terminal receives the voice data sent from the server.

[0083] Step 11:

[0084] The device plays back the audio data. Using the speaker, the device plays back the audio, "Today's study will be solving math problems."

[0085] Step 12:

[0086] The server records the user's speech input and the generated responses, and stores this interaction in a log file.

[0087] Step 13:

[0088] The server saves the logs in cloud storage, and the server uploads the recorded audio and text data to cloud storage for storage.

[0089] Step 14:

[0090] The server shares the data stored in the cloud storage, and sets up data sharing so that parents and educators can access it.

[0091] Step 15:

[0092] The device performs voice authentication to uniquely identify the user. During initial setup, the user's voice is recorded and voice authentication technology is used to identify the user in subsequent interactions.

[0093] Example 1

[0094] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0095] Modern education demands personalized learning support. However, conventional learning systems face challenges in efficiently recognizing users' voice input and generating appropriate responses. Furthermore, they lack the ability to record and share individual learning progress. To address these challenges, an advanced system combining speech recognition and generative artificial intelligence models is needed.

[0096] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0097] In this invention, the server includes means for receiving a user's voice input, means for recognizing the voice input and converting it into text, means for analyzing the text and understanding the user's intent, means for generating a response according to the user's intent using a generative AI model, means for converting the generated response into voice data, means for outputting the voice data, and means for recording the voice input and the generated response. This enables personalized learning support, generates appropriate responses from the user's voice input, and makes it possible to effectively record and share learning progress.

[0098] "Means for receiving user voice input" refers to a device or system for capturing user-uttered speech and processing it as data.

[0099] "Means for recognizing and converting said voice input to text" refers to a technology or system that analyzes received voice data and converts it into corresponding text data.

[0100] "Means for analyzing the text and understanding the user's intent" refers to an algorithm or system that analyzes text data to determine what the user is looking for.

[0101] "Means for generating responses based on user intent using a generative artificial intelligence model" refers to the process of using technologies such as deep learning and machine learning to generate appropriate responses based on user input.

[0102] "Means for converting the generated response into voice data" refers to a system that converts a text-format response into voice data using voice synthesis technology.

[0103] "Means for outputting audio data" refers to a speaker or other audio device that allows the user to hear the generated audio data.

[0104] "Means for recording the voice input and generated responses" refers to a system that stores interactions with a user as a log.

[0105] "Means for storing recorded voice data in cloud storage" refers to technology that transfers and stores locally stored voice data in cloud-based storage via the Internet.

[0106] "Means for sharing data stored in cloud storage" refers to the functionality that allows other users or systems with access rights to the stored data to view and use the data.

[0107] "Voice authentication means for uniquely recognizing voice input" refers to technology that identifies and authenticates individuals based on their voice characteristics.

[0108] The present invention provides an educational support system that receives voice input from a user, converts the voice data into text data using a server and a terminal, analyzes the data, and generates an appropriate response. Detailed embodiments for carrying out the present invention will be described below.

[0109] Hardware Configuration

[0110] This system consists of a terminal used by the user and a server that communicates with it. The terminal is equipped with a high-sensitivity microphone and speaker, and includes a network communication module such as Wi-Fi or LTE as needed. The server has a high-performance processor and sufficient memory, and is connected to a cloud network.

[0111] Software Configuration

[0112] The software of this system consists of the following main components:

[0113] 1. Speech recognition engine (e.g., Google Cloud Speech-to-Text API)

[0114] 2. Natural Language Processing (NLP) engine (e.g., Spashitai)

[0115] 3. Generative AI models (e.g., GPT-4 (registered trademark))

[0116] 4. Speech synthesis engine (e.g., Amazon Polly or Google Cloud Text-to-Speech API)

[0117] 5. Cloud storage services (e.g., Amazon S3 and Google Cloud Storage)

[0118] User voice input

[0119] Users speak into the device to ask questions or make requests about their studies, such as, "What are we going to study today?", and the device's sensitive microphone captures this voice.

[0120] Sending audio data

[0121] The device sends the captured audio data in binary format to the server, using Wi-Fi or LTE for network communication.

[0122] Speech recognition and text conversion

[0123] The server then calls the Google Cloud Speech-to-Text API to analyze the received voice data and converts it into text data, such as "What are you going to study today?"

[0124] Text analysis and intent understanding

[0125] The server uses a generative AI model (e.g., GPT-4) to analyze the text data and understand the user's intent. The generative AI model uses the following prompt:

[0126] "Here's a question about what you learned today. Please generate an appropriate response."

[0127] Response Generation

[0128] The server uses a generative AI model to generate a response based on the user's intentions, such as "Today's study will involve solving math problems."

[0129] Speech synthesis and response transmission

[0130] The server converts the generated text response into audio data using Amazon Polly or the Google Cloud Text-to-Speech API, and sends the audio data to the device, which then tells the user through the speaker, "Today's lesson will be solving math problems."

[0131] Recording and saving conversation logs

[0132] The server logs all user interactions and stores them in a cloud storage service (e.g., Amazon S3 or Google Cloud Storage), where the data can be later viewed by parents and educators.

[0133] Voice Authentication

[0134] The terminal uses the user's voice for voice authentication. The user's voice is recorded during initial setup and authentication is performed based on that data. Only authenticated users can use the system.

[0135] As described above, the system of the present invention provides personalized learning support to users by smoothly performing a series of processes from voice input to text conversion, response generation, voice output, and log recording. It also serves as a powerful tool for other users to improve the quality of their education.

[0136] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0137] Step 1:

[0138] The user speaks to the device, "What will I study today?" The device captures the voice using a built-in high-sensitivity microphone. The input is the user's voice input, and the output is the captured voice data.

[0139] Step 2:

[0140] The device transmits the captured audio data to the server using a network communication module (Wi-Fi or LTE). The input is the captured audio data, and the output is the audio data transmitted to the server.

[0141] Step 3:

[0142] The server inputs the received voice data into the Google Cloud Speech-to-Text API and converts the voice into text data. The input is the voice data received by the server, and the output is the converted text data, "What are you going to study today?"

[0143] Step 4:

[0144] The server inputs the text data into a generative AI model (e.g., GPT-4) for analysis. The prompt text is "We have received a question about today's learning content. Please generate an appropriate response." The input is the text data and the prompt text, and the output is the understood intent (understanding the intent of the question) as the result of analysis.

[0145] Step 5:

[0146] The server uses a generative AI model to generate a response based on the user's intent. The input is the analysis result and the user's intent, and the output is the generated text response, "Today's study is to solve math problems."

[0147] Step 6:

[0148] The server inputs the generated text response into a speech synthesis engine (e.g., Amazon Polly) to convert the text to speech data. The input is the generated text response, and the output is the converted speech data.

[0149] Step 7:

[0150] The server sends the converted voice data to the terminal, where the input is the converted voice data and the output is the voice data sent to the terminal.

[0151] Step 8:

[0152] The device plays the voice data received from the server over the speaker and tells the user, "Today's study will be solving math problems." The input is the received voice data, and the output is a voice response.

[0153] Step 9:

[0154] The server records all user interactions as log data and stores it in a cloud storage service (e.g., Amazon S3). The input is the collected voice input and generated response log data, and the output is the cloud-stored log data.

[0155] (Application example 1)

[0156] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0157] Currently, physical stores that sell educational products face the challenge of making it difficult for users to quickly and accurately obtain detailed product information. Furthermore, there is a lack of systems for managing user inquiry histories and comparison information about products, hindering efficient customer service and improving user satisfaction. Furthermore, individual user recognition through voice authentication is insufficient, making it difficult to provide personalized services.

[0158] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0159] In this invention, the server includes: means for receiving a user's voice input; means for recognizing the voice input and converting it into text; means for analyzing the text and understanding the user's intent; means for generating a response according to the user's intent; means for converting the generated response into voice data; means for outputting the voice data; means for allowing users to ask product-related questions at locations selling educational products; means for providing product information based on the questions; means for recording the user's voice input and the generated responses; means for storing the recorded voice data in cloud storage; means for sharing the data stored in the cloud storage; means for managing the user's question and answer history for future reference; voice authentication means for individually recognizing the voice input; and means for comparing the user's question with other products and suggesting recommended products based on the question content. This allows users to quickly and accurately obtain detailed information about educational products, and improves user satisfaction through the management of question history and product comparison suggestions. Furthermore, voice authentication enables individual user recognition, enabling personalized service provision.

[0160] "User's voice input" refers to words or sounds uttered by a user through a voice input device such as a microphone.

[0161] The "means for recognizing and converting speech input to text" refers to a device or method that uses a speech recognition engine to convert speech data into character string data.

[0162] "Means for analyzing said text and understanding the user's intent" refers to the process of using text analysis algorithms and generative AI models to understand what the user is saying and infer their intent.

[0163] The "means for generating a response according to the user's intention" is a device or method for generating an appropriate answer or suggestion based on the user's intention.

[0164] The "means for converting the generated response into voice data" is a voice synthesis engine that converts text data into voice data.

[0165] The "means for outputting audio data" is a device that allows the user to hear the audio data through a speaker or the like.

[0166] "Places that sell educational products" are physical stores that sell or provide learning materials, educational tools, etc.

[0167] The "means for providing product information based on the content of a question" refers to a device or method that responds to a user's question with detailed product information.

[0168] "Means for recording user voice input and generated responses" is a process for saving voice data and text data to a recording device.

[0169] The "means for storing the recorded voice data in cloud storage" is a technology for storing voice data on a remote server via the Internet.

[0170] The "means for sharing data stored in the cloud storage" is a technology for sharing stored data with other users or systems.

[0171] "Means for managing user question and answer history and making it available for future reference" refers to a system that stores question and answer data in an easily accessible format.

[0172] "Voice authentication means for individually recognizing voice input" refers to a technology that recognizes the individual voice characteristics of a user and identifies the user.

[0173] The "means for comparing products with other products and proposing recommended products based on the content of a question" is a technology for generating and presenting comparative information on related products and recommended products based on the content of a user's question.

[0174] This invention provides a system that allows users to ask questions about educational products through voice in a brick-and-mortar store. The system receives the user's voice input, analyzes it, generates an appropriate response, and replies by voice, enabling the provision of information quickly and accurately.

[0175] System configuration

[0176] The system includes the following main components:

[0177] 1. A means of receiving user voice input

[0178] The system receives voice data emitted by a user through a microphone, which the user does using a wearable device such as a smartphone or smart glasses.

[0179] 2. Speech recognition and text conversion methods

[0180] The server converts the received voice data into text using a voice recognition engine (e.g., Google Speech Recognition API).

[0181] 3. Text Analysis and Intention Understanding

[0182] The server uses a generative AI model to analyze text data and understand user intent. Text analysis and intent understanding are achieved by using the OpenAI (registered trademark) API.

[0183] 4. Response Generation Method

[0184] The generative AI model generates a response in text format according to the user's intentions.

[0185] 5. Speech synthesis and output methods

[0186] The server converts the generated text response into audio data using a speech synthesis engine (e.g., the pyttsx3 library) and sends it to the user's device, where it is played back.

[0187] 6. Means of providing product information based on the content of the question

[0188] When a user asks a question about a product by voice, the system provides detailed information about the product based on the content of the question. For example, in response to the question, "What does this math workbook contain?", the system will respond, "This math workbook contains practice problems for basic arithmetic operations and applied problems."

[0189] 7. Voice input and response recording

[0190] It has the means to record the user's voice input and generated responses and store them in cloud storage for future reference and analysis.

[0191] 8. How to manage question and answer history

[0192] The system manages the user's question and answer history and allows them to refer to it as needed. Specifically, it also suggests comparison information with other products and recommended products based on the content of the question.

[0193] 9. Voice Authentication Methods

[0194] The system registers the user's voice during initial setup and performs voice authentication during subsequent use, enabling individual user recognition and providing personalized services.

[0195] Program processing

[0196] The program in this system processes and calculates data in the following steps:

[0197] 1. Capture and transmit voice input

[0198] When a user asks, "What's in this math workbook?" the device captures the audio and sends the data to the server.

[0199] 2. Voice Recognition

[0200] When the server receives the voice data, it uses a speech recognition engine (e.g., Google Speech Recognition API) to convert it into text such as "What's in this math workbook?"

[0201] 3. Text Analysis and Intention Understanding

[0202] The text data is analyzed using a generative AI model to understand that the user is asking about the product content, specifically OpenAI's GPT model.

[0203] 4. Response Generation

[0204] Based on the analysis results, the generative AI model generates a response such as, "This math drill contains practice problems and applied problems for basic arithmetic operations."

[0205] 5. Speech synthesis and output

[0206] The generated text response is converted into voice data using a voice synthesis engine, sent to the terminal, and answered to the user by voice.

[0207] 6. Logging and Storage

[0208] All conversations are logged and saved in cloud storage for future reference.

[0209] This allows users to quickly and accurately obtain detailed information about educational products and enjoy personalized services through voice authentication.

[0210] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0211] Step 1:

[0212] The user asks a question.

[0213] Users use the microphone on their smartphone or smart glasses to voice-inquire about a product, such as "What does this math workbook contain?"

[0214] Step 2:

[0215] Capture and transmit audio input

[0216] The device captures the user's voice, temporarily stores this voice data, and sends it to the server using the device's network function. The input is the user's voice data, and the output is the voice data sent to the server.

[0217] Step 3:

[0218] Voice Recognition

[0219] The server converts the received voice data into text using the Google Speech Recognition API. The input is the user's voice data, and the output is the converted text data. Specifically, it is converted into the text "What is included in this math workbook?"

[0220] Step 4:

[0221] Text analysis and intent understanding

[0222] The server uses a generative AI model (OpenAI's API) to analyze the text data and understand the user's intent. The input is text data, and the output is data that is interpreted as the user's intent. Specifically, it recognizes that the user is asking a question about the content of the drill.

[0223] Step 5:

[0224] Response Generation

[0225] The server uses a generative AI model to generate an appropriate response based on the user's intent. The input is data interpreted as the user's intent, and the output is the generated response text. For example, it generates a response such as, "This math workbook contains practice problems and applied problems for basic arithmetic operations."

[0226] Step 6:

[0227] Speech synthesis and transmission

[0228] The server converts the generated response into audio data using the pyttsx3 library. The input is the text response, and the output is the generated audio data. This audio data is then sent to the device.

[0229] Step 7:

[0230] Voice response output

[0231] The terminal plays the received audio data to the user through the speaker. The input is the audio data sent from the server, and the output is the audio heard by the user.

[0232] Step 8:

[0233] Logging and Storage

[0234] The server records the user's voice input and the generated response and stores them in cloud storage. The input is the voice input and response data, and the output is the log data stored in cloud storage. We store these data for future reference.

[0235] Step 9:

[0236] Question and answer history management

[0237] The server manages the user's questions and answers as a history and makes them available for reference as needed. The input is logged data, and the output is searchable historical data.

[0238] Step 10:

[0239] Voice Authentication

[0240] The server performs voice authentication to identify the individual user. The input is the user's voice data, and the output is the result of voice authentication. If the voice authentication is successful, the user can receive personalized services.

[0241] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0242] The present invention is an educational system that receives user voice input, analyzes it, and generates appropriate responses. Furthermore, by combining it with an emotion engine that recognizes the user's emotions and adjusts the response, it provides more personalized educational support.

[0243] System Overview

[0244] When a user asks a question or makes a request by voice, the device captures the voice and sends it to the server. The server uses a speech recognition engine to convert the voice into text, analyzes the text, generates an appropriate response based on a generative AI model, converts it into voice data, and sends it back to the device. The device responds to the user by voice, and records the interaction and emotional data in a log and saves it in cloud storage.

[0245] Capture and transmit audio input

[0246] Examples:

[0247] When a user speaks to the device, "What are you going to study today?", the device captures the voice input and sends the captured voice data to the server.

[0248] Speech recognition and text conversion

[0249] Examples:

[0250] The server analyzes the received voice data using a speech recognition engine and converts it into text such as "What are you going to study today?" This text data is sent to the next step for further analysis.

[0251] Text analysis and intent understanding

[0252] Examples:

[0253] The server uses a generative AI model to analyze the text data and understand that the user is asking a question about today's learning content.

[0254] emotion recognition

[0255] Examples:

[0256] The server uses an emotion engine to analyze the user's emotions from the voice data, for example, identifying emotions such as nervousness, excitement, sadness, etc.

[0257] Response Generation

[0258] Examples:

[0259] The server generates a text response such as, "Today's study will be solving math problems." It then adjusts the response based on the user's emotions. For example, if it detects that the user is nervous, it adds encouraging words such as, "It's okay, take your time."

[0260] Speech synthesis and output

[0261] Examples:

[0262] The server converts the generated text response into voice data using a speech synthesis engine. The converted voice data is sent to the device, which then tells the user, "Today's lesson is to solve math problems. It's okay, take your time."

[0263] Recording and storing conversation logs and emotional data

[0264] Examples:

[0265] The server logs user interactions and emotional analysis data and stores it in cloud storage, where parents and educators can access the logs to track learning progress and emotional changes.

[0266] Voice Authentication

[0267] Examples:

[0268] The device performs voice authentication to uniquely identify the user. The user's voice is recorded during initial setup, and voice authentication technology is used to identify the user during subsequent interactions.

[0269] In this way, the system of the present invention provides personalized learning support through a series of steps, from voice input to text conversion, response generation, voice output, emotion analysis, and log recording. It is also possible to share information with parents and educators, which increases the user's motivation to learn and provides effective educational support.

[0270] The processing flow will be explained below.

[0271] Step 1:

[0272] The user speaks to the device. For example, the user might say, "What are you going to study today?"

[0273] Step 2:

[0274] The device captures the user's voice. The device's built-in microphone records the user's voice data.

[0275] Step 3:

[0276] The device sends the captured audio data to the server, which then uploads the audio data to the server via the network.

[0277] Step 4:

[0278] The server receives the voice data. The server receives the voice data sent from the terminal.

[0279] Step 5:

[0280] The server uses a speech recognition engine to convert the voice data into text. For example, the server converts the voice data of "What are you going to study today?" into text format.

[0281] Step 6:

[0282] The server analyzes the text data. Using a generative AI model, the server analyzes the content of the text and understands the user's intent. The server determines that the user's question is related to today's learning content.

[0283] Step 7:

[0284] The server uses an emotion engine to analyze the user's emotion from the voice data, and determines that the user is, for example, "excited."

[0285] Step 8:

[0286] The server generates a response based on the user's intention and emotions. Based on the analysis results, the server generates a text response such as "Today's study is to solve math problems," and adds encouraging words such as "Have fun while you do it!", taking into account the user's excitement.

[0287] Step 9:

[0288] The server converts the text response into audio data. The server uses a speech synthesis engine to convert the generated text response into audio format.

[0289] Step 10:

[0290] The server transmits the voice data to the terminal, and the server sends the generated voice data to the terminal via the network.

[0291] Step 11:

[0292] The terminal receives the voice data. The terminal receives the voice data sent from the server.

[0293] Step 12:

[0294] The device plays back the audio data. Using the speaker, the device conveys to the user the following message: "Today's lesson is to solve a collection of math problems. Have fun doing it!"

[0295] Step 13:

[0296] The server records the user's speech input and the generated responses, and stores this interaction in a log file.

[0297] Step 14:

[0298] The server saves the logs in cloud storage, and then uploads the recorded voice data, text data, and emotion data to cloud storage for storage.

[0299] Step 15:

[0300] The server shares the data stored in the cloud storage, and sets up data sharing so that parents and educators can access it.

[0301] Step 16:

[0302] The device performs voice authentication to uniquely identify the user. The user's voice is recorded during initial setup, and voice authentication technology is used to identify the user during subsequent interactions.

[0303] In this way, the system of the present invention provides personalized learning support through a series of steps from user voice input to analysis, response generation, voice output, emotion analysis, and log recording.

[0304] Example 2

[0305] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0306] The main function of conventional educational support systems was to convert a user's voice input into text and generate an appropriate response. However, these systems were unable to provide responses that took the user's emotional state into account, making it difficult to provide fully personalized learning support. Furthermore, they lacked the authentication functionality to individually recognize each user's voice input, making operation difficult when multiple users were using the system. This made it impossible to provide efficient and effective educational support.

[0307] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0308] In this invention, the server includes means for receiving a user's voice input, means for recognizing the voice input and converting it into text, means for analyzing the text and understanding the user's intention, means for recognizing the user's emotions, means for generating a response according to the user's intention and emotion, means for converting the generated response into voice data, and means for outputting the voice data. This makes it possible to provide an appropriate response to the user's voice input that takes emotion into consideration. Furthermore, by further including authentication means for individually recognizing voice inputs, operation when multiple users use the system becomes smoother.

[0309] "Voice input" refers to the act of converting a user's speech into a digital signal and transmitting it to the system, or that digital signal.

[0310] "Text-to-text" refers to the process of converting digital data into text using speech recognition technology to represent spoken input as text.

[0311] "Intent understanding" refers to the process of analyzing input text data and recognizing what the user is looking for.

[0312] "Emotion recognition" refers to the process of analyzing and identifying a user's emotional state from voice or text data.

[0313] "Response generation" refers to the process of creating an appropriate reply based on the user's intent and feelings.

[0314] "Speech synthesis" refers to the process of converting text data into speech data and outputting it as natural-sounding speech.

[0315] "Network storage" refers to online storage space where data is stored on remote servers or cloud services and accessed and shared as needed.

[0316] "Authentication method" refers to a technology or process that identifies an individual user by analyzing the user's voice characteristics.

[0317] The present invention is an educational support system that receives user voice input, analyzes it using a speech recognition and emotion recognition engine, and generates an appropriate response. This system provides personalized educational support by combining a speech recognition engine, a text analysis engine, an emotion recognition engine, a generative AI model, and a speech synthesis engine.

[0318] When a user speaks, the device captures the speech and sends it to a server. The server then uses a speech recognition engine to convert the speech data into text. This speech recognition uses commonly available speech recognition technology (e.g., a cloud-based speech recognition service). The converted text data is then sent to a text analysis engine (e.g., a generative AI model) to analyze the user's intent.

[0319] Furthermore, the voice data is analyzed by an emotion recognition engine to identify the user's emotional state. Emotion recognition is performed using commonly available emotion analysis techniques (e.g., emotion recognition services and engines). Based on the analysis results, an appropriate response is generated in text format by a generative AI model.

[0320] Next, the generated text response is converted into voice data using a voice synthesis engine. This voice synthesis is performed using a commonly used voice synthesis technology (e.g., a voice synthesis service). The converted voice data is sent to the terminal and a voice response is provided to the user.

[0321] The system also has the ability to record interaction logs and emotional data and save them in network storage. Parents and educators can access these logs to check learning progress and emotional changes. For example, this data can be securely stored and shared using cloud storage such as AWS (registered trademark) S3.

[0322] The system also uses voice authentication technology to identify individual users. During initial setup, the user's voice is recorded and saved in a profile. The system then matches each voice input with this profile to identify the user. This technology makes it possible to provide individual learning support when multiple users use the same system.

[0323] Specific examples

[0324] For example, if a user says to the device, "What are you going to study today?", the device captures the speech and sends it to the server. The server uses a speech recognition engine to convert the speech into text, "What are you going to study today?" and then uses a generative AI model to analyze the text and understand that the user is asking a question about what they're studying today. If the emotion recognition engine detects nervousness in the user's voice, the generated response might be something like, "Today's study is to solve math problems. It's okay, take your time." The server then converts this text response into audio data using a speech synthesis engine and sends it back to the device. Finally, the device tells the user, "Today's study is to solve math problems. It's okay, take your time."

[0325] Prompt Sentence Examples

[0326] User Question: "What are you studying today?"

[0327] User voice analysis results: "Nervous"

[0328] Generate Response: "Today's lesson is to do some math problems. It's okay, just take your time."

[0329] By combining these specific technologies, the system of the present invention provides advanced and personalized educational support that meets the specific needs of each user.

[0330] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0331] Step 1:

[0332] Capture and transmit audio input

[0333] When a user speaks into the device, the device uses its built-in microphone to capture the voice. Specifically, the user's voice (input) is converted into a digital signal, and the voice data is encoded and sent to the server (output). For example, if a user says, "What are we going to study today?", the voice is captured and digitized. This series of operations is performed in real time.

[0334] Step 2:

[0335] Speech recognition and text conversion

[0336] The server sends the voice data (input) received from the device to a voice recognition engine. Specifically, the voice data is converted into text data (output) using a voice recognition engine (for example, a cloud-based voice recognition service). For example, the voice data "What are you going to study today?" is converted into text as "What are you going to study today?" This converted text data is sent to the next step.

[0337] Step 3:

[0338] Text analysis and intent understanding

[0339] The server analyzes the text data (input) received from the speech recognition engine. Specifically, it uses a generative AI model (e.g., a generative AI model) to analyze the text data and understand the user's intent (output). For example, it analyzes the text "What are we going to study today?" and identifies the intent as "A question about today's learning content." This intent data is sent to the next step.

[0340] Step 4:

[0341] emotion recognition

[0342] The server sends the voice data and text data (input) from the speech recognition engine to the emotion recognition engine. The emotion recognition engine (e.g., emotion analysis service) is used to analyze the user's emotional state (output). For example, it identifies tension, excitement, happiness, etc. from the tone and speed of the voice. This emotion data is sent to the next step.

[0343] Step 5:

[0344] Response Generation

[0345] The server uses a generative AI model to generate an appropriate response based on the intent data and emotion data (input). Specifically, it creates a text response and adjusts it based on the emotion data (output). For example, if it recognizes that the user is nervous, it adds encouraging words such as, "Today's study is to solve your math problems. It's okay, take your time." This adjusted response is sent to the next step.

[0346] Step 6:

[0347] Speech synthesis and output

[0348] The server sends the generated text response (input) to a speech synthesis engine, which converts it into voice data. The speech synthesis engine (e.g., a speech synthesis service) generates the text as natural-sounding voice data (output). For example, voice data such as "Today's study is to solve the math problems. It's okay, take your time" is generated. This voice data is then sent to the device.

[0349] Step 7:

[0350] Recording and storing conversation logs and emotional data

[0351] The server logs user interactions and emotional data (input) and stores them in network storage, such as AWS S3, for later access by parents and educators (output). Specifically, the server compiles user questions, system responses, and emotional analysis results into a single log file.

[0352] Step 8:

[0353] Voice Authentication

[0354] During initial setup, the device records the user's voice and saves it in a profile. It then uses voice authentication technology (e.g., a voice authentication service) to identify the user with each interaction (input and output). For example, new voice input can be matched against the profile to authenticate the user. This allows multiple users to receive personalized learning assistance even when using the same system.

[0355] (Application example 2)

[0356] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0357] Conventional customer support and guidance systems in brick-and-mortar stores could only provide uniform answers to customer questions, making it difficult to respond taking into account the circumstances and emotions of each individual customer. They also lacked a mechanism for accumulating and analyzing customer responses and emotions. As a result, customer satisfaction declined, and improving the quality of store service became an issue.

[0358] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving a user's voice input, means for recognizing the voice input and converting it into text, means for analyzing the text and understanding the user's intention, means for generating a response according to the user's intention, means for recognizing the user's emotion and adjusting the response based on the emotion, means for converting the generated response into voice data, and means for outputting the voice data. This makes it possible to provide a personalized response to a customer's question according to the situation and emotion, thereby improving customer satisfaction and service quality.

[0359] The "means for receiving user voice input" is a device or module for capturing voice uttered by the user and incorporating it into the system.

[0360] The "means for recognizing the voice input and converting it into text" refers to an algorithm or engine for analyzing received voice data and converting it into text information.

[0361] "Means for analyzing the text and understanding the user's intent" refers to AI models and natural language processing technologies that analyze text data to understand the content of questions and requests uttered by users.

[0362] "Means for generating a response according to the user's intent" refers to a generative AI model or response engine for generating appropriate and effective answers based on the user's intent.

[0363] "Means for recognizing the user's emotions and adjusting the response based on those emotions" refers to an algorithm or engine that analyzes voice data or text data to identify the user's emotions and optimizes the response content based on the results.

[0364] The "means for converting the generated response into voice data" is a voice synthesis engine for converting the response in text format into voice data.

[0365] The "means for outputting audio data" refers to an output device for audibly transmitting audio data to the user.

[0366] "Cloud storage" is a data storage service for storing data in a remote location accessible via the Internet.

[0367] "Voice authentication means" is a technology for individually identifying a speaker from voice data.

[0368] System Configuration

[0369] This invention is a system for providing advanced customer support and guidance in a customer support and guidance system, and is composed of the following main components:

[0370] 1. Voice input receiving devices: These include smart glasses and head-mounted displays. These devices capture the user's voice input.

[0371] 2. Speech recognition engine: An engine that converts voice data received on the server side into text data. As a specific example, Google Cloud Speech-to-Text API is used.

[0372] 3. Text analysis module: A generative AI model that analyzes the text data and understands the user's intent. OpenAI GPT-4 is used as a concrete example.

[0373] 4. Emotion Recognition Engine: An algorithm for analyzing user emotions from voice data, allowing for tailored responses.

[0374] 5. Speech synthesis engine: An engine for converting the generated text response into audio data. For example, Google Cloud Text-to-Speech API is used.

[0375] 6. Cloud Storage: Data storage service for storing and sharing generated response and emotion data.

[0376] System Operation

[0377] Capture and transmit audio input

[0378] When a user asks a question into the smart glasses or head-mounted display, the device captures the voice input and sends it to the server. For example, the user might say, "What are the features of this product?"

[0379] Speech recognition and text conversion

[0380] The server analyzes the received voice data using the Google Cloud Speech-to-Text API and converts the voice data into text data. This step generates the text "Please tell me the features of this product."

[0381] Text analysis and intent understanding

[0382] The server uses a generative AI model (OpenAI GPT-4) to analyze the text data and understand the intent of the user's question. For example, in this case, it interprets the user as "asking about the product's features."

[0383] emotion recognition

[0384] The voice data is analyzed using the emotion_recognition algorithm to identify the user's emotions, such as "nervous," "excited," or "sad."

[0385] Response Generation

[0386] The server generates an appropriate response based on the analysis results. For example, a text response such as "This product is made of high-quality materials and is particularly durable" is generated. If the user's emotion is identified as "tension," the response will include a phrase such as "Don't worry, take your time looking around."

[0387] Speech synthesis and output

[0388] The generated text response is converted to audio data using the Google Cloud Text-to-Speech API and sent to a smart eyeglasses or head-mounted display, where the user receives the answer via audio.

[0389] Specific prompt examples

[0390] A user asks, "What is this product?" How do you respond?

[0391] Data recording and storage

[0392] The server logs user interaction and emotion data and stores it in cloud storage, which is then analyzed and used to improve the store's services.

[0393] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0394] Step 1:

[0395] A user asks a question to smart glasses or a head-mounted display. The user's voice input is captured by the device and the voice data is sent to the server. The input is the user's voice data, and the output is the voice data sent to the server.

[0396] Step 2:

[0397] The server analyzes the received voice data using a speech recognition engine (specifically, the Google Cloud Speech-to-Text API). The input is the voice data sent to the server, and the output is the converted text data. The server then sends this data to the next step.

[0398] Step 3:

[0399] The server uses a text analysis module to analyze the text data and understand the user's intent using a generative AI model (OpenAI GPT-4). For example, if a user says, "Please tell me the features of this product," the server interprets the intent as "I am asking about the product's features." The input is the text data to be analyzed, and the output is the interpreted user intent.

[0400] Step 4:

[0401] The server analyzes the voice and text data using the emotion_recognition algorithm to identify the user's emotions. The input is voice or text data, and the output is the emotion recognition result, such as "nervous" or "excited."

[0402] Step 5:

[0403] The server uses a generative AI model to generate a response based on the user's understood intent and the identified emotion. For example, it generates a text response such as "This product is made of high-quality materials and is particularly durable." If the emotion is "tension," it adds "Don't worry, take your time looking around." The input is the user's intent and emotion, and the output is the generated text response.

[0404] Step 6:

[0405] The server converts the generated text response into audio data using a speech synthesis engine (Google Cloud Text-to-Speech API). The input is the text response, and the output is audio data. This audio data is sent to the device.

[0406] Step 7:

[0407] The terminal plays the received voice data and conveys the response to the user. The input is the voice data sent from the server, and the output is the voice response conveyed to the user.

[0408] Step 8:

[0409] The server logs user interaction and emotional data and stores them in cloud storage. The input is the user's voice input, response, and emotional data, and the output is the log data stored in cloud storage. The stored data is later analyzed and used to improve the service.

[0410] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0411] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0412] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0413] [Second embodiment]

[0414] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0415] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0416] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0417] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0418] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0419] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0420] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0421] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0422] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0423] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0424] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0425] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0426] The present invention is an educational system that receives voice input from a user (especially a child), analyzes it, and generates an appropriate response. This system includes a series of processes that operate between a server, a terminal, and the user.

[0427] System Overview

[0428] When a user asks a question or makes a request via voice, the device captures the speech and sends it to the server. The server uses a speech recognition engine to convert the speech to text and analyzes the text. It then generates an appropriate response based on a generative AI model, converts the response into audio data, and sends it back to the device. The device then responds to the user via voice, logs the interaction, and stores it in cloud storage.

[0429] Capture and transmit audio input

[0430] Examples:

[0431] When a user speaks to the device, "What are you going to study today?", the device captures the voice input and sends the captured voice data to the server.

[0432] Speech recognition and text conversion

[0433] Examples:

[0434] The server analyzes the received voice data with a speech recognition engine and converts it into text such as "What will you study today?" This text data is sent to the next step for further analysis.

[0435] Text analysis and intent understanding

[0436] Examples:

[0437] The server uses a generative AI model to analyze the text data and understand that the user is asking a question about today's learning content.

[0438] Response Generation

[0439] Examples:

[0440] The server generates a text response such as, "Today's study will be solving math problems." The generative AI model creates advice and suggestions based on the training data.

[0441] Speech synthesis and output

[0442] Examples:

[0443] The server converts the generated text response into voice data using a speech synthesis engine, which is then sent to the device, which then tells the user, "Today's study will be solving math problems."

[0444] Recording and saving conversation logs

[0445] Examples:

[0446] The server logs all user interactions and stores them in cloud storage, where parents and educators can access the logs to track learning progress.

[0447] Voice Authentication

[0448] Examples:

[0449] The device records the user's voice during initial setup and performs voice authentication during subsequent interactions, allowing only authenticated users to use the system.

[0450] In this way, the system of the present invention provides personalized learning support through a series of steps, from voice input to text conversion, response generation, voice output, and log recording. It is also possible to share information with parents and educators, which increases the user's motivation to learn and provides effective educational support.

[0451] The processing flow will be explained below.

[0452] Step 1:

[0453] The user speaks to the device. For example, the user might say, "What will I study today?"

[0454] Step 2:

[0455] The device captures the user's voice. The device's built-in microphone records the user's voice data.

[0456] Step 3:

[0457] The device sends the captured audio data to the server, which then uploads the audio data to the server via the network.

[0458] Step 4:

[0459] The server receives the voice data. The server receives the voice data sent from the terminal.

[0460] Step 5:

[0461] The server uses a speech recognition engine to convert the voice data into text. For example, the server converts the voice data of "What will you study today?" into text format.

[0462] Step 6:

[0463] The server analyzes the text data and uses a generative AI model to analyze the content of the text and understand the user's intent.

[0464] Step 7:

[0465] The server generates a response based on the user's intent. Based on the analysis results, the server generates an appropriate text response such as "Today's study will be solving math problems."

[0466] Step 8:

[0467] The server converts the text response into audio data. The server uses a speech synthesis engine to convert the text response into audio format.

[0468] Step 9:

[0469] The server transmits the generated voice data to the terminal via the network.

[0470] Step 10:

[0471] The terminal receives the voice data. The terminal receives the voice data sent from the server.

[0472] Step 11:

[0473] The device plays back the audio data. Using the speaker, the device plays back the audio, "Today's study will be solving math problems."

[0474] Step 12:

[0475] The server records the user's speech input and the generated responses, and stores this interaction in a log file.

[0476] Step 13:

[0477] The server saves the logs in cloud storage, and the server uploads the recorded audio and text data to cloud storage for storage.

[0478] Step 14:

[0479] The server shares the data stored in the cloud storage, and sets up data sharing so that parents and educators can access it.

[0480] Step 15:

[0481] The device performs voice authentication to uniquely identify the user. During initial setup, the user's voice is recorded and voice authentication technology is used to identify the user in subsequent interactions.

[0482] Example 1

[0483] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0484] Modern education demands personalized learning support. However, conventional learning systems face challenges in efficiently recognizing users' voice input and generating appropriate responses. Furthermore, they lack the ability to record and share individual learning progress. To address these challenges, an advanced system combining speech recognition and generative artificial intelligence models is needed.

[0485] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0486] In this invention, the server includes means for receiving a user's voice input, means for recognizing the voice input and converting it into text, means for analyzing the text and understanding the user's intent, means for generating a response according to the user's intent using a generative AI model, means for converting the generated response into voice data, means for outputting the voice data, and means for recording the voice input and the generated response. This enables personalized learning support, generates appropriate responses from the user's voice input, and makes it possible to effectively record and share learning progress.

[0487] "Means for receiving user voice input" refers to a device or system for capturing user-uttered speech and processing it as data.

[0488] "Means for recognizing and converting said voice input to text" refers to a technology or system that analyzes received voice data and converts it into corresponding text data.

[0489] "Means for analyzing the text and understanding the user's intent" refers to an algorithm or system that analyzes text data to determine what the user is looking for.

[0490] "Means for generating responses based on user intent using a generative artificial intelligence model" refers to the process of using technologies such as deep learning and machine learning to generate appropriate responses based on user input.

[0491] "Means for converting the generated response into voice data" refers to a system that converts a text-format response into voice data using voice synthesis technology.

[0492] "Means for outputting audio data" refers to a speaker or other audio device that allows the user to hear the generated audio data.

[0493] "Means for recording the voice input and generated responses" refers to a system that stores interactions with a user as a log.

[0494] "Means for storing recorded voice data in cloud storage" refers to technology that transfers and stores locally stored voice data in cloud-based storage via the Internet.

[0495] "Means for sharing data stored in cloud storage" refers to the functionality that allows other users or systems with access rights to the stored data to view and use the data.

[0496] "Voice authentication means for uniquely recognizing voice input" refers to technology that identifies and authenticates individuals based on their voice characteristics.

[0497] The present invention provides an educational support system that receives voice input from a user, converts the voice data into text data using a server and a terminal, analyzes the data, and generates an appropriate response. Detailed embodiments for carrying out the present invention will be described below.

[0498] Hardware Configuration

[0499] This system consists of a terminal used by the user and a server that communicates with it. The terminal is equipped with a high-sensitivity microphone and speaker, and includes a network communication module such as Wi-Fi or LTE as needed. The server has a high-performance processor and sufficient memory, and is connected to a cloud network.

[0500] Software Configuration

[0501] The software of this system consists of the following main components:

[0502] 1. Speech recognition engine (e.g., Google Cloud Speech-to-Text API)

[0503] 2. Natural Language Processing (NLP) engine (e.g., Spashitai)

[0504] 3. Generative AI models (e.g., GPT-4)

[0505] 4. Speech synthesis engine (e.g., Amazon Polly or Google Cloud Text-to-Speech API)

[0506] 5. Cloud storage services (e.g., Amazon S3 and Google Cloud Storage)

[0507] User voice input

[0508] Users speak into the device to ask questions or make requests about their studies, such as, "What are we going to study today?", and the device's sensitive microphone captures this voice.

[0509] Sending audio data

[0510] The device sends the captured audio data in binary format to the server, using Wi-Fi or LTE for network communication.

[0511] Speech recognition and text conversion

[0512] The server then calls the Google Cloud Speech-to-Text API to analyze the received voice data and converts it into text data, such as "What are you going to study today?"

[0513] Text analysis and intent understanding

[0514] The server uses a generative AI model (e.g., GPT-4) to analyze the text data and understand the user's intent. The generative AI model uses the following prompt:

[0515] "Here's a question about what you learned today. Please generate an appropriate response."

[0516] Response Generation

[0517] The server uses a generative AI model to generate a response based on the user's intentions, such as "Today's study will involve solving math problems."

[0518] Speech synthesis and response transmission

[0519] The server converts the generated text response into audio data using Amazon Polly or the Google Cloud Text-to-Speech API, and sends the audio data to the device, which then tells the user through the speaker, "Today's lesson will be solving math problems."

[0520] Recording and saving conversation logs

[0521] The server logs all user interactions and stores them in a cloud storage service (e.g., Amazon S3 or Google Cloud Storage), where the data can be later viewed by parents and educators.

[0522] Voice Authentication

[0523] The terminal uses the user's voice for voice authentication. The user's voice is recorded during initial setup and authentication is performed based on that data. Only authenticated users can use the system.

[0524] As described above, the system of the present invention provides personalized learning support to users by smoothly performing a series of processes from voice input to text conversion, response generation, voice output, and log recording. It also serves as a powerful tool for other users to improve the quality of their education.

[0525] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0526] Step 1:

[0527] The user speaks to the device, "What will I study today?" The device captures the voice using a built-in high-sensitivity microphone. The input is the user's voice input, and the output is the captured voice data.

[0528] Step 2:

[0529] The device transmits the captured audio data to the server using a network communication module (Wi-Fi or LTE). The input is the captured audio data, and the output is the audio data transmitted to the server.

[0530] Step 3:

[0531] The server inputs the received voice data into the Google Cloud Speech-to-Text API and converts the voice into text data. The input is the voice data received by the server, and the output is the converted text data, "What are you going to study today?"

[0532] Step 4:

[0533] The server inputs the text data into a generative AI model (e.g., GPT-4) for analysis. The prompt text is "We have received a question about today's learning content. Please generate an appropriate response." The input is the text data and the prompt text, and the output is the understood intent (understanding the intent of the question) as the result of analysis.

[0534] Step 5:

[0535] The server uses a generative AI model to generate a response based on the user's intent. The input is the analysis result and the user's intent, and the output is the generated text response, "Today's study is to solve math problems."

[0536] Step 6:

[0537] The server inputs the generated text response into a speech synthesis engine (e.g., Amazon Polly) to convert the text to speech data. The input is the generated text response, and the output is the converted speech data.

[0538] Step 7:

[0539] The server sends the converted voice data to the terminal, where the input is the converted voice data and the output is the voice data sent to the terminal.

[0540] Step 8:

[0541] The device plays the voice data received from the server over the speaker and tells the user, "Today's study will be solving math problems." The input is the received voice data, and the output is a voice response.

[0542] Step 9:

[0543] The server records all user interactions as log data and stores it in a cloud storage service (e.g., Amazon S3). The input is the collected voice input and generated response log data, and the output is the cloud-stored log data.

[0544] (Application example 1)

[0545] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0546] Currently, physical stores that sell educational products face the challenge of making it difficult for users to quickly and accurately obtain detailed product information. Furthermore, there is a lack of systems for managing user inquiry histories and comparison information about products, hindering efficient customer service and improving user satisfaction. Furthermore, individual user recognition through voice authentication is insufficient, making it difficult to provide personalized services.

[0547] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0548] In this invention, the server includes: means for receiving a user's voice input; means for recognizing the voice input and converting it into text; means for analyzing the text and understanding the user's intent; means for generating a response according to the user's intent; means for converting the generated response into voice data; means for outputting the voice data; means for allowing users to ask product-related questions at locations selling educational products; means for providing product information based on the questions; means for recording the user's voice input and the generated responses; means for storing the recorded voice data in cloud storage; means for sharing the data stored in the cloud storage; means for managing the user's question and answer history for future reference; voice authentication means for individually recognizing the voice input; and means for comparing the user's question with other products and suggesting recommended products based on the question content. This allows users to quickly and accurately obtain detailed information about educational products, and improves user satisfaction through the management of question history and product comparison suggestions. Furthermore, voice authentication enables individual user recognition, enabling personalized service provision.

[0549] "User's voice input" refers to words or sounds uttered by a user through a voice input device such as a microphone.

[0550] The "means for recognizing and converting speech input to text" refers to a device or method that uses a speech recognition engine to convert speech data into character string data.

[0551] "Means for analyzing said text and understanding the user's intent" refers to the process of using text analysis algorithms and generative AI models to understand what the user is saying and infer their intent.

[0552] The "means for generating a response according to the user's intention" is a device or method for generating an appropriate answer or suggestion based on the user's intention.

[0553] The "means for converting the generated response into voice data" is a voice synthesis engine that converts text data into voice data.

[0554] The "means for outputting audio data" is a device that allows the user to hear the audio data through a speaker or the like.

[0555] "Places that sell educational products" are physical stores that sell or provide learning materials, educational tools, etc.

[0556] The "means for providing product information based on the content of a question" refers to a device or method that responds to a user's question with detailed product information.

[0557] "Means for recording user voice input and generated responses" is a process for saving voice data and text data to a recording device.

[0558] The "means for storing the recorded voice data in cloud storage" is a technology for storing voice data on a remote server via the Internet.

[0559] The "means for sharing data stored in the cloud storage" is a technology for sharing stored data with other users or systems.

[0560] "Means for managing user question and answer history and making it available for future reference" refers to a system that stores question and answer data in an easily accessible format.

[0561] "Voice authentication means for individually recognizing voice input" refers to a technology that recognizes the individual voice characteristics of a user and identifies the user.

[0562] The "means for comparing products with other products and proposing recommended products based on the content of a question" is a technology for generating and presenting comparative information on related products and recommended products based on the content of a user's question.

[0563] This invention provides a system that allows users to ask questions about educational products through voice in a brick-and-mortar store. The system receives the user's voice input, analyzes it, generates an appropriate response, and replies by voice, enabling the provision of information quickly and accurately.

[0564] System configuration

[0565] The system includes the following main components:

[0566] 1. A means of receiving user voice input

[0567] The system receives voice data emitted by a user through a microphone, which the user does using a wearable device such as a smartphone or smart glasses.

[0568] 2. Speech recognition and text conversion methods

[0569] The server converts the received voice data into text using a voice recognition engine (e.g., Google Speech Recognition API).

[0570] 3. Text Analysis and Intention Understanding

[0571] The server uses a generative AI model to analyze text data and understand user intent. Text analysis and intent understanding are achieved by using OpenAI's API.

[0572] 4. Response Generation Method

[0573] The generative AI model generates a response in text format according to the user's intentions.

[0574] 5. Speech synthesis and output methods

[0575] The server converts the generated text response into audio data using a speech synthesis engine (e.g., the pyttsx3 library) and sends it to the user's device, where it is played back.

[0576] 6. Means of providing product information based on the content of the question

[0577] When a user asks a question about a product by voice, the system provides detailed information about the product based on the content of the question. For example, in response to the question, "What does this math workbook contain?", the system will respond, "This math workbook contains practice problems for basic arithmetic operations and applied problems."

[0578] 7. Voice input and response recording

[0579] It has the means to record the user's voice input and generated responses and store them in cloud storage for future reference and analysis.

[0580] 8. How to manage question and answer history

[0581] The system manages the user's question and answer history and allows them to refer to it as needed. Specifically, it also suggests comparison information with other products and recommended products based on the content of the question.

[0582] 9. Voice Authentication Methods

[0583] The system registers the user's voice during initial setup and performs voice authentication during subsequent use, enabling individual user recognition and providing personalized services.

[0584] Program processing

[0585] The program in this system processes and calculates data in the following steps:

[0586] 1. Capture and transmit voice input

[0587] When a user asks, "What's in this math workbook?" the device captures the audio and sends the data to the server.

[0588] 2. Voice Recognition

[0589] When the server receives the voice data, it uses a speech recognition engine (e.g., Google Speech Recognition API) to convert it into text such as "What's in this math workbook?"

[0590] 3. Text Analysis and Intention Understanding

[0591] The text data is analyzed using a generative AI model to understand that the user is asking about the product content, specifically OpenAI's GPT model.

[0592] 4. Response Generation

[0593] Based on the analysis results, the generative AI model generates a response such as, "This math drill contains practice problems and applied problems for basic arithmetic operations."

[0594] 5. Speech synthesis and output

[0595] The generated text response is converted into voice data using a voice synthesis engine, sent to the terminal, and answered to the user by voice.

[0596] 6. Logging and Storage

[0597] All conversations are logged and saved in cloud storage for future reference.

[0598] This allows users to quickly and accurately obtain detailed information about educational products and enjoy personalized services through voice authentication.

[0599] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0600] Step 1:

[0601] The user asks a question.

[0602] Users use the microphone on their smartphone or smart glasses to voice-inquire about a product, such as "What does this math workbook contain?"

[0603] Step 2:

[0604] Capture and transmit audio input

[0605] The device captures the user's voice, temporarily stores this voice data, and sends it to the server using the device's network function. The input is the user's voice data, and the output is the voice data sent to the server.

[0606] Step 3:

[0607] Voice Recognition

[0608] The server converts the received voice data into text using the Google Speech Recognition API. The input is the user's voice data, and the output is the converted text data. Specifically, it is converted into the text "What is included in this math workbook?"

[0609] Step 4:

[0610] Text analysis and intent understanding

[0611] The server uses a generative AI model (OpenAI's API) to analyze the text data and understand the user's intent. The input is text data, and the output is data that is interpreted as the user's intent. Specifically, it recognizes that the user is asking a question about the content of the drill.

[0612] Step 5:

[0613] Response Generation

[0614] The server uses a generative AI model to generate an appropriate response based on the user's intent. The input is data interpreted as the user's intent, and the output is the generated response text. For example, it generates a response such as, "This math workbook contains practice problems and applied problems for basic arithmetic operations."

[0615] Step 6:

[0616] Speech synthesis and transmission

[0617] The server converts the generated response into audio data using the pyttsx3 library. The input is the text response, and the output is the generated audio data. This audio data is then sent to the device.

[0618] Step 7:

[0619] Voice response output

[0620] The terminal plays the received audio data to the user through the speaker. The input is the audio data sent from the server, and the output is the audio heard by the user.

[0621] Step 8:

[0622] Logging and Storage

[0623] The server records the user's voice input and the generated response and stores them in cloud storage. The input is the voice input and response data, and the output is the log data stored in cloud storage. We store these data for future reference.

[0624] Step 9:

[0625] Question and answer history management

[0626] The server manages the user's questions and answers as a history and makes them available for reference as needed. The input is logged data, and the output is searchable historical data.

[0627] Step 10:

[0628] Voice Authentication

[0629] The server performs voice authentication to identify the individual user. The input is the user's voice data, and the output is the result of voice authentication. If the voice authentication is successful, the user can receive personalized services.

[0630] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0631] The present invention is an educational system that receives user voice input, analyzes it, and generates appropriate responses. Furthermore, by combining it with an emotion engine that recognizes the user's emotions and adjusts the response, it provides more personalized educational support.

[0632] System Overview

[0633] When a user asks a question or makes a request by voice, the device captures the voice and sends it to the server. The server uses a speech recognition engine to convert the voice into text, analyzes the text, generates an appropriate response based on a generative AI model, converts it into voice data, and sends it back to the device. The device responds to the user by voice, and records the interaction and emotional data in a log and saves it in cloud storage.

[0634] Capture and transmit audio input

[0635] Examples:

[0636] When a user speaks to the device, "What are you going to study today?", the device captures the voice input and sends the captured voice data to the server.

[0637] Speech recognition and text conversion

[0638] Examples:

[0639] The server analyzes the received voice data using a speech recognition engine and converts it into text such as "What are you going to study today?" This text data is sent to the next step for further analysis.

[0640] Text analysis and intent understanding

[0641] Examples:

[0642] The server uses a generative AI model to analyze the text data and understand that the user is asking a question about today's learning content.

[0643] emotion recognition

[0644] Examples:

[0645] The server uses an emotion engine to analyze the user's emotions from the voice data, for example, identifying emotions such as nervousness, excitement, sadness, etc.

[0646] Response Generation

[0647] Examples:

[0648] The server generates a text response such as, "Today's study will be solving math problems." It then adjusts the response based on the user's emotions. For example, if it detects that the user is nervous, it adds encouraging words such as, "It's okay, take your time."

[0649] Speech synthesis and output

[0650] Examples:

[0651] The server converts the generated text response into voice data using a speech synthesis engine. The converted voice data is sent to the device, which then tells the user, "Today's lesson is to solve math problems. It's okay, take your time."

[0652] Recording and storing conversation logs and emotional data

[0653] Examples:

[0654] The server logs user interactions and emotional analysis data and stores it in cloud storage, where parents and educators can access the logs to track learning progress and emotional changes.

[0655] Voice Authentication

[0656] Examples:

[0657] The device performs voice authentication to uniquely identify the user. The user's voice is recorded during initial setup, and voice authentication technology is used to identify the user during subsequent interactions.

[0658] In this way, the system of the present invention provides personalized learning support through a series of steps, from voice input to text conversion, response generation, voice output, emotion analysis, and log recording. It is also possible to share information with parents and educators, which increases the user's motivation to learn and provides effective educational support.

[0659] The processing flow will be explained below.

[0660] Step 1:

[0661] The user speaks to the device. For example, the user might say, "What are you going to study today?"

[0662] Step 2:

[0663] The device captures the user's voice. The device's built-in microphone records the user's voice data.

[0664] Step 3:

[0665] The device sends the captured audio data to the server, which then uploads the audio data to the server via the network.

[0666] Step 4:

[0667] The server receives the voice data. The server receives the voice data sent from the terminal.

[0668] Step 5:

[0669] The server uses a speech recognition engine to convert the voice data into text. For example, the server converts the voice data of "What are you going to study today?" into text format.

[0670] Step 6:

[0671] The server analyzes the text data. Using a generative AI model, the server analyzes the content of the text and understands the user's intent. The server determines that the user's question is related to today's learning content.

[0672] Step 7:

[0673] The server uses an emotion engine to analyze the user's emotion from the voice data, and determines that the user is, for example, "excited."

[0674] Step 8:

[0675] The server generates a response based on the user's intention and emotions. Based on the analysis results, the server generates a text response such as "Today's study is to solve math problems," and adds encouraging words such as "Have fun while you do it!", taking into account the user's excitement.

[0676] Step 9:

[0677] The server converts the text response into audio data. The server uses a speech synthesis engine to convert the generated text response into audio format.

[0678] Step 10:

[0679] The server transmits the voice data to the terminal, and the server sends the generated voice data to the terminal via the network.

[0680] Step 11:

[0681] The terminal receives the voice data. The terminal receives the voice data sent from the server.

[0682] Step 12:

[0683] The device plays back the audio data. Using the speaker, the device conveys to the user the following message: "Today's lesson is to solve a collection of math problems. Have fun doing it!"

[0684] Step 13:

[0685] The server records the user's speech input and the generated responses, and stores this interaction in a log file.

[0686] Step 14:

[0687] The server saves the logs in cloud storage, and then uploads the recorded voice data, text data, and emotion data to cloud storage for storage.

[0688] Step 15:

[0689] The server shares the data stored in the cloud storage, and sets up data sharing so that parents and educators can access it.

[0690] Step 16:

[0691] The device performs voice authentication to uniquely identify the user. The user's voice is recorded during initial setup, and voice authentication technology is used to identify the user during subsequent interactions.

[0692] In this way, the system of the present invention provides personalized learning support through a series of steps from user voice input to analysis, response generation, voice output, emotion analysis, and log recording.

[0693] Example 2

[0694] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0695] The main function of conventional educational support systems was to convert a user's voice input into text and generate an appropriate response. However, these systems were unable to provide responses that took the user's emotional state into account, making it difficult to provide fully personalized learning support. Furthermore, they lacked the authentication functionality to individually recognize each user's voice input, making operation difficult when multiple users were using the system. This made it impossible to provide efficient and effective educational support.

[0696] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0697] In this invention, the server includes means for receiving a user's voice input, means for recognizing the voice input and converting it into text, means for analyzing the text and understanding the user's intention, means for recognizing the user's emotions, means for generating a response according to the user's intention and emotion, means for converting the generated response into voice data, and means for outputting the voice data. This makes it possible to provide an appropriate response to the user's voice input that takes emotion into consideration. Furthermore, by further including authentication means for individually recognizing voice inputs, operation when multiple users use the system becomes smoother.

[0698] "Voice input" refers to the act of converting a user's speech into a digital signal and transmitting it to the system, or that digital signal.

[0699] "Text-to-text" refers to the process of converting digital data into text using speech recognition technology to represent spoken input as text.

[0700] "Intent understanding" refers to the process of analyzing input text data and recognizing what the user is looking for.

[0701] "Emotion recognition" refers to the process of analyzing and identifying a user's emotional state from voice or text data.

[0702] "Response generation" refers to the process of creating an appropriate reply based on the user's intent and feelings.

[0703] "Speech synthesis" refers to the process of converting text data into speech data and outputting it as natural-sounding speech.

[0704] "Network storage" refers to online storage space where data is stored on remote servers or cloud services and accessed and shared as needed.

[0705] "Authentication method" refers to a technology or process that identifies an individual user by analyzing the user's voice characteristics.

[0706] The present invention is an educational support system that receives user voice input, analyzes it using a speech recognition and emotion recognition engine, and generates an appropriate response. This system provides personalized educational support by combining a speech recognition engine, a text analysis engine, an emotion recognition engine, a generative AI model, and a speech synthesis engine.

[0707] When a user speaks, the device captures the speech and sends it to a server. The server then uses a speech recognition engine to convert the speech data into text. This speech recognition uses commonly available speech recognition technology (e.g., a cloud-based speech recognition service). The converted text data is then sent to a text analysis engine (e.g., a generative AI model) to analyze the user's intent.

[0708] Furthermore, the voice data is analyzed by an emotion recognition engine to identify the user's emotional state. Emotion recognition is performed using commonly available emotion analysis techniques (e.g., emotion recognition services and engines). Based on the analysis results, an appropriate response is generated in text format by a generative AI model.

[0709] Next, the generated text response is converted into voice data using a voice synthesis engine. This voice synthesis is performed using a commonly used voice synthesis technology (e.g., a voice synthesis service). The converted voice data is sent to the terminal and a voice response is provided to the user.

[0710] The system also has the ability to record interaction logs and emotional data and store them in network storage. Parents and educators can access these logs to check learning progress and emotional changes. For example, this data can be securely stored and shared using cloud storage such as AWS S3.

[0711] The system also uses voice authentication technology to identify individual users. During initial setup, the user's voice is recorded and saved in a profile. The system then matches each voice input with this profile to identify the user. This technology makes it possible to provide individual learning support when multiple users use the same system.

[0712] Specific examples

[0713] For example, if a user says to the device, "What are you going to study today?", the device captures the speech and sends it to the server. The server uses a speech recognition engine to convert the speech into text, "What are you going to study today?" and then uses a generative AI model to analyze the text and understand that the user is asking a question about what they're studying today. If the emotion recognition engine detects nervousness in the user's voice, the generated response might be something like, "Today's study is to solve math problems. It's okay, take your time." The server then converts this text response into audio data using a speech synthesis engine and sends it back to the device. Finally, the device tells the user, "Today's study is to solve math problems. It's okay, take your time."

[0714] Prompt Sentence Examples

[0715] User Question: "What are you studying today?"

[0716] User voice analysis results: "Nervous"

[0717] Generate Response: "Today's lesson is to do some math problems. It's okay, just take your time."

[0718] By combining these specific technologies, the system of the present invention provides advanced and personalized educational support that meets the specific needs of each user.

[0719] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0720] Step 1:

[0721] Capture and transmit audio input

[0722] When a user speaks into the device, the device uses its built-in microphone to capture the voice. Specifically, the user's voice (input) is converted into a digital signal, and the voice data is encoded and sent to the server (output). For example, if a user says, "What are we going to study today?", the voice is captured and digitized. This series of operations is performed in real time.

[0723] Step 2:

[0724] Speech recognition and text conversion

[0725] The server sends the voice data (input) received from the device to a voice recognition engine. Specifically, the voice data is converted into text data (output) using a voice recognition engine (for example, a cloud-based voice recognition service). For example, the voice data "What are you going to study today?" is converted into text as "What are you going to study today?" This converted text data is sent to the next step.

[0726] Step 3:

[0727] Text analysis and intent understanding

[0728] The server analyzes the text data (input) received from the speech recognition engine. Specifically, it uses a generative AI model (e.g., a generative AI model) to analyze the text data and understand the user's intent (output). For example, it analyzes the text "What are we going to study today?" and identifies the intent as "A question about today's learning content." This intent data is sent to the next step.

[0729] Step 4:

[0730] emotion recognition

[0731] The server sends the voice data and text data (input) from the speech recognition engine to the emotion recognition engine. The emotion recognition engine (e.g., emotion analysis service) is used to analyze the user's emotional state (output). For example, it identifies tension, excitement, happiness, etc. from the tone and speed of the voice. This emotion data is sent to the next step.

[0732] Step 5:

[0733] Response Generation

[0734] The server uses a generative AI model to generate an appropriate response based on the intent data and emotion data (input). Specifically, it creates a text response and adjusts it based on the emotion data (output). For example, if it recognizes that the user is nervous, it adds encouraging words such as, "Today's study is to solve your math problems. It's okay, take your time." This adjusted response is sent to the next step.

[0735] Step 6:

[0736] Speech synthesis and output

[0737] The server sends the generated text response (input) to a speech synthesis engine, which converts it into voice data. The speech synthesis engine (e.g., a speech synthesis service) generates the text as natural-sounding voice data (output). For example, voice data such as "Today's study is to solve the math problems. It's okay, take your time" is generated. This voice data is then sent to the device.

[0738] Step 7:

[0739] Recording and storing conversation logs and emotional data

[0740] The server logs user interactions and emotional data (input) and stores them in network storage, such as AWS S3, for later access by parents and educators (output). Specifically, the server compiles user questions, system responses, and emotional analysis results into a single log file.

[0741] Step 8:

[0742] Voice Authentication

[0743] During initial setup, the device records the user's voice and saves it in a profile. It then uses voice authentication technology (e.g., a voice authentication service) to identify the user with each interaction (input and output). For example, new voice input can be matched against the profile to authenticate the user. This allows multiple users to receive personalized learning assistance even when using the same system.

[0744] (Application example 2)

[0745] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0746] Conventional customer support and guidance systems in brick-and-mortar stores could only provide uniform answers to customer questions, making it difficult to respond taking into account the circumstances and emotions of each individual customer. They also lacked a mechanism for accumulating and analyzing customer responses and emotions. As a result, customer satisfaction declined, and improving the quality of store service became an issue.

[0747] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving a user's voice input, means for recognizing the voice input and converting it into text, means for analyzing the text and understanding the user's intention, means for generating a response according to the user's intention, means for recognizing the user's emotion and adjusting the response based on the emotion, means for converting the generated response into voice data, and means for outputting the voice data. This makes it possible to provide a personalized response to a customer's question according to the situation and emotion, thereby improving customer satisfaction and service quality.

[0748] The "means for receiving user voice input" is a device or module for capturing voice uttered by the user and incorporating it into the system.

[0749] The "means for recognizing the voice input and converting it into text" refers to an algorithm or engine for analyzing the received voice data and converting it into text information.

[0750] "Means for analyzing the text and understanding the user's intent" refers to AI models and natural language processing technologies that analyze text data to understand the content of questions and requests uttered by users.

[0751] "Means for generating a response according to the user's intent" refers to a generative AI model or response engine for generating appropriate and effective answers based on the user's intent.

[0752] "Means for recognizing the user's emotions and adjusting the response based on those emotions" refers to an algorithm or engine that analyzes voice data or text data to identify the user's emotions and optimizes the response content based on the results.

[0753] The "means for converting the generated response into voice data" is a voice synthesis engine for converting the response in text format into voice data.

[0754] The "means for outputting audio data" refers to an output device for audibly transmitting audio data to the user.

[0755] "Cloud storage" is a data storage service for storing data in a remote location accessible via the Internet.

[0756] "Voice authentication means" is a technology for individually identifying a speaker from voice data.

[0757] System Configuration

[0758] This invention is a system for providing advanced customer support in a customer support and guidance system, and is composed of the following main components:

[0759] 1. Voice input receiving devices: These include smart glasses and head-mounted displays. These devices capture the user's voice input.

[0760] 2. Speech recognition engine: An engine for converting voice data received on the server side into text data. As a specific example, Google Cloud Speech-to-Text API is used.

[0761] 3. Text analysis module: A generative AI model that analyzes the text data and understands the user's intent. OpenAI GPT-4 is used as a concrete example.

[0762] 4. Emotion Recognition Engine: An algorithm for analyzing user emotions from voice data, allowing for tailored responses.

[0763] 5. Speech synthesis engine: An engine for converting the generated text response into audio data. For example, Google Cloud Text-to-Speech API is used.

[0764] 6. Cloud Storage: Data storage service for storing and sharing generated response and emotion data.

[0765] System Operation

[0766] Capture and transmit audio input

[0767] When a user asks a question into the smart glasses or head-mounted display, the device captures the voice input and sends it to the server. For example, the user might say, "Please tell me the features of this product."

[0768] Speech recognition and text conversion

[0769] The server analyzes the received voice data using the Google Cloud Speech-to-Text API and converts the voice data into text data. This step generates the text "Please tell me the features of this product."

[0770] Text analysis and intent understanding

[0771] The server uses a generative AI model (OpenAI GPT-4) to analyze the text data and understand the intent of the user's question. For example, in this case, it interprets the user as "asking about the product's features."

[0772] emotion recognition

[0773] The voice data is analyzed using the emotion_recognition algorithm to identify the user's emotions, such as "nervous," "excited," or "sad."

[0774] Response Generation

[0775] The server generates an appropriate response based on the analysis results. For example, a text response such as "This product is made of high-quality materials and is particularly durable" is generated. If the user's emotion is identified as "tension," the response will include a phrase such as "Don't worry, take your time looking around."

[0776] Speech synthesis and output

[0777] The generated text response is converted to audio data using the Google Cloud Text-to-Speech API and sent to a smart eyeglasses or head-mounted display, where the user receives the answer via audio.

[0778] Specific prompt examples

[0779] A user asks, "What is this product?" How do you respond?

[0780] Data recording and storage

[0781] The server logs user interaction and emotional data and stores it in cloud storage, which is then analyzed and used to improve the store's services.

[0782] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0783] Step 1:

[0784] A user asks a question to smart glasses or a head-mounted display. The user's voice input is captured by the device and the voice data is sent to the server. The input is the user's voice data, and the output is the voice data sent to the server.

[0785] Step 2:

[0786] The server analyzes the received voice data using a speech recognition engine (specifically, the Google Cloud Speech-to-Text API). The input is the voice data sent to the server, and the output is the converted text data. The server then sends this data to the next step.

[0787] Step 3:

[0788] The server uses a text analysis module to analyze the text data and understand the user's intent using a generative AI model (OpenAI GPT-4). For example, if a user says, "Please tell me the features of this product," the server interprets the intent as "I am asking about the product's features." The input is the text data to be analyzed, and the output is the interpreted user intent.

[0789] Step 4:

[0790] The server analyzes the voice and text data using the emotion_recognition algorithm to identify the user's emotions. The input is voice or text data, and the output is the emotion recognition result, such as "nervous" or "excited."

[0791] Step 5:

[0792] The server uses a generative AI model to generate a response based on the user's understood intent and the identified emotion. For example, it generates a text response such as "This product is made of high-quality materials and is particularly durable." If the emotion is "tension," it adds "Don't worry, take your time looking around." The input is the user's intent and emotion, and the output is the generated text response.

[0793] Step 6:

[0794] The server converts the generated text response into audio data using a speech synthesis engine (Google Cloud Text-to-Speech API). The input is the text response, and the output is audio data. This audio data is sent to the device.

[0795] Step 7:

[0796] The terminal plays the received voice data and conveys the response to the user. The input is the voice data sent from the server, and the output is the voice response conveyed to the user.

[0797] Step 8:

[0798] The server logs user interaction and emotional data and stores them in cloud storage. The input is the user's voice input, response, and emotional data, and the output is the log data stored in cloud storage. The stored data is later analyzed and used to improve the service.

[0799] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0800] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0801] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0802] [Third embodiment]

[0803] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0804] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0805] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0806] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0807] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0808] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0809] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0810] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0811] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0812] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0813] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0814] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0815] The present invention is an educational system that receives voice input from a user (especially a child), analyzes it, and generates an appropriate response. This system includes a series of processes that operate between a server, a terminal, and the user.

[0816] System Overview

[0817] When a user asks a question or makes a request via voice, the device captures the speech and sends it to the server. The server uses a speech recognition engine to convert the speech to text and analyzes the text. It then generates an appropriate response based on a generative AI model, converts the response into audio data, and sends it back to the device. The device then responds to the user via voice, logs the interaction, and stores it in cloud storage.

[0818] Capture and transmit audio input

[0819] Examples:

[0820] When a user speaks to the device, "What are you going to study today?", the device captures the voice input and sends the captured voice data to the server.

[0821] Speech recognition and text conversion

[0822] Examples:

[0823] The server analyzes the received voice data with a speech recognition engine and converts it into text such as "What will you study today?" This text data is sent to the next step for further analysis.

[0824] Text analysis and intent understanding

[0825] Examples:

[0826] The server uses a generative AI model to analyze the text data and understand that the user is asking a question about today's learning content.

[0827] Response Generation

[0828] Examples:

[0829] The server generates a text response such as, "Today's study will be solving math problems." The generative AI model creates advice and suggestions based on the training data.

[0830] Speech synthesis and output

[0831] Examples:

[0832] The server converts the generated text response into voice data using a speech synthesis engine, which is then sent to the device, which then tells the user, "Today's study will be solving math problems."

[0833] Recording and saving conversation logs

[0834] Examples:

[0835] The server logs all user interactions and stores them in cloud storage, where parents and educators can access the logs to track learning progress.

[0836] Voice Authentication

[0837] Examples:

[0838] The device records the user's voice during initial setup and performs voice authentication during subsequent interactions, allowing only authenticated users to use the system.

[0839] In this way, the system of the present invention provides personalized learning support through a series of steps, from voice input to text conversion, response generation, voice output, and log recording. It is also possible to share information with parents and educators, which increases the user's motivation to learn and provides effective educational support.

[0840] The processing flow will be explained below.

[0841] Step 1:

[0842] The user speaks to the device. For example, the user might say, "What will I study today?"

[0843] Step 2:

[0844] The device captures the user's voice. The device's built-in microphone records the user's voice data.

[0845] Step 3:

[0846] The device sends the captured audio data to the server, which then uploads the audio data to the server via the network.

[0847] Step 4:

[0848] The server receives the voice data. The server receives the voice data sent from the terminal.

[0849] Step 5:

[0850] The server uses a speech recognition engine to convert the voice data into text. For example, the server converts the voice data of "What will you study today?" into text format.

[0851] Step 6:

[0852] The server analyzes the text data and uses a generative AI model to analyze the content of the text and understand the user's intent.

[0853] Step 7:

[0854] The server generates a response based on the user's intent. Based on the analysis results, the server generates an appropriate text response such as "Today's study will be solving math problems."

[0855] Step 8:

[0856] The server converts the text response into audio data. The server uses a speech synthesis engine to convert the text response into audio format.

[0857] Step 9:

[0858] The server transmits the generated voice data to the terminal via the network.

[0859] Step 10:

[0860] The terminal receives the voice data. The terminal receives the voice data sent from the server.

[0861] Step 11:

[0862] The device plays back the audio data. Using the speaker, the device plays back the audio, "Today's study will be solving math problems."

[0863] Step 12:

[0864] The server records the user's speech input and the generated responses, and stores this interaction in a log file.

[0865] Step 13:

[0866] The server saves the logs in cloud storage, and the server uploads the recorded audio and text data to cloud storage for storage.

[0867] Step 14:

[0868] The server shares the data stored in the cloud storage, and sets up data sharing so that parents and educators can access it.

[0869] Step 15:

[0870] The device performs voice authentication to uniquely identify the user. During initial setup, the user's voice is recorded and voice authentication technology is used to identify the user in subsequent interactions.

[0871] Example 1

[0872] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0873] Modern education demands personalized learning support. However, conventional learning systems face challenges in efficiently recognizing users' voice input and generating appropriate responses. Furthermore, they lack the ability to record and share individual learning progress. To address these challenges, an advanced system combining speech recognition and generative artificial intelligence models is needed.

[0874] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0875] In this invention, the server includes means for receiving a user's voice input, means for recognizing the voice input and converting it into text, means for analyzing the text and understanding the user's intent, means for generating a response according to the user's intent using a generative AI model, means for converting the generated response into voice data, means for outputting the voice data, and means for recording the voice input and the generated response. This enables personalized learning support, generates appropriate responses from the user's voice input, and makes it possible to effectively record and share learning progress.

[0876] "Means for receiving user voice input" refers to a device or system for capturing user-uttered speech and processing it as data.

[0877] "Means for recognizing and converting said voice input to text" refers to a technology or system that analyzes received voice data and converts it into corresponding text data.

[0878] "Means for analyzing the text and understanding the user's intent" refers to an algorithm or system that analyzes text data to determine what the user is looking for.

[0879] "Means for generating responses based on user intent using a generative artificial intelligence model" refers to the process of using technologies such as deep learning and machine learning to generate appropriate responses based on user input.

[0880] "Means for converting the generated response into voice data" refers to a system that converts a text-format response into voice data using voice synthesis technology.

[0881] "Means for outputting audio data" refers to a speaker or other audio device that allows the user to hear the generated audio data.

[0882] "Means for recording the voice input and generated responses" refers to a system that stores interactions with a user as a log.

[0883] "Means for storing recorded voice data in cloud storage" refers to technology that transfers and stores locally stored voice data in cloud-based storage via the Internet.

[0884] "Means for sharing data stored in cloud storage" refers to the functionality that allows other users or systems with access rights to the stored data to view and use the data.

[0885] "Voice authentication means for uniquely recognizing voice input" refers to technology that identifies and authenticates individuals based on their voice characteristics.

[0886] The present invention provides an educational support system that receives voice input from a user, converts the voice data into text data using a server and a terminal, analyzes the data, and generates an appropriate response. Detailed embodiments for carrying out the present invention will be described below.

[0887] Hardware Configuration

[0888] This system consists of a terminal used by the user and a server that communicates with it. The terminal is equipped with a high-sensitivity microphone and speaker, and includes a network communication module such as Wi-Fi or LTE as needed. The server has a high-performance processor and sufficient memory, and is connected to a cloud network.

[0889] Software Configuration

[0890] The software of this system consists of the following main components:

[0891] 1. Speech recognition engine (e.g., Google Cloud Speech-to-Text API)

[0892] 2. Natural Language Processing (NLP) engine (e.g., Spashitai)

[0893] 3. Generative AI models (e.g., GPT-4)

[0894] 4. Speech synthesis engine (e.g., Amazon Polly or Google Cloud Text-to-Speech API)

[0895] 5. Cloud storage services (e.g., Amazon S3 and Google Cloud Storage)

[0896] User voice input

[0897] Users speak into the device to ask questions or make requests about their studies, such as, "What are we going to study today?", and the device's sensitive microphone captures this voice.

[0898] Sending audio data

[0899] The device sends the captured audio data in binary format to the server, using Wi-Fi or LTE for network communication.

[0900] Speech recognition and text conversion

[0901] The server then calls the Google Cloud Speech-to-Text API to analyze the received voice data and converts it into text data, such as "What are you going to study today?"

[0902] Text analysis and intent understanding

[0903] The server uses a generative AI model (e.g., GPT-4) to analyze the text data and understand the user's intent. The generative AI model uses the following prompt:

[0904] "Here's a question about what you learned today. Please generate an appropriate response."

[0905] Response Generation

[0906] The server uses a generative AI model to generate a response based on the user's intentions, such as "Today's study will involve solving math problems."

[0907] Speech synthesis and response transmission

[0908] The server converts the generated text response into audio data using Amazon Polly or the Google Cloud Text-to-Speech API, and sends the audio data to the device, which then tells the user through the speaker, "Today's lesson will be solving math problems."

[0909] Recording and saving conversation logs

[0910] The server logs all user interactions and stores them in a cloud storage service (e.g., Amazon S3 or Google Cloud Storage), where the data can be later viewed by parents and educators.

[0911] Voice Authentication

[0912] The terminal uses the user's voice for voice authentication. The user's voice is recorded during initial setup and authentication is performed based on that data. Only authenticated users can use the system.

[0913] As described above, the system of the present invention provides personalized learning support to users by smoothly performing a series of processes from voice input to text conversion, response generation, voice output, and log recording. It also serves as a powerful tool for other users to improve the quality of their education.

[0914] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0915] Step 1:

[0916] The user speaks to the device, "What will I study today?" The device captures the voice using a built-in high-sensitivity microphone. The input is the user's voice input, and the output is the captured voice data.

[0917] Step 2:

[0918] The device transmits the captured audio data to the server using a network communication module (Wi-Fi or LTE). The input is the captured audio data, and the output is the audio data transmitted to the server.

[0919] Step 3:

[0920] The server inputs the received voice data into the Google Cloud Speech-to-Text API and converts the voice into text data. The input is the voice data received by the server, and the output is the converted text data, "What are you going to study today?"

[0921] Step 4:

[0922] The server inputs the text data into a generative AI model (e.g., GPT-4) for analysis. The prompt text is "We have received a question about today's learning content. Please generate an appropriate response." The input is the text data and the prompt text, and the output is the understood intent (understanding the intent of the question) as the result of analysis.

[0923] Step 5:

[0924] The server uses a generative AI model to generate a response based on the user's intent. The input is the analysis result and the user's intent, and the output is the generated text response, "Today's study is to solve math problems."

[0925] Step 6:

[0926] The server inputs the generated text response into a speech synthesis engine (e.g., Amazon Polly) to convert the text to speech data. The input is the generated text response, and the output is the converted speech data.

[0927] Step 7:

[0928] The server sends the converted voice data to the terminal, where the input is the converted voice data and the output is the voice data sent to the terminal.

[0929] Step 8:

[0930] The device plays the voice data received from the server over the speaker and tells the user, "Today's study will be solving math problems." The input is the received voice data, and the output is a voice response.

[0931] Step 9:

[0932] The server records all user interactions as log data and stores it in a cloud storage service (e.g., Amazon S3). The input is the collected voice input and generated response log data, and the output is the cloud-stored log data.

[0933] (Application example 1)

[0934] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0935] Currently, physical stores that sell educational products face the challenge of making it difficult for users to quickly and accurately obtain detailed product information. Furthermore, there is a lack of systems for managing user inquiry histories and comparison information about products, hindering efficient customer service and improving user satisfaction. Furthermore, individual user recognition through voice authentication is insufficient, making it difficult to provide personalized services.

[0936] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0937] In this invention, the server includes: means for receiving a user's voice input; means for recognizing the voice input and converting it into text; means for analyzing the text and understanding the user's intent; means for generating a response according to the user's intent; means for converting the generated response into voice data; means for outputting the voice data; means for allowing users to ask product-related questions at locations selling educational products; means for providing product information based on the questions; means for recording the user's voice input and the generated responses; means for storing the recorded voice data in cloud storage; means for sharing the data stored in the cloud storage; means for managing the user's question and answer history for future reference; voice authentication means for individually recognizing the voice input; and means for comparing the user's question with other products and suggesting recommended products based on the question content. This allows users to quickly and accurately obtain detailed information about educational products, and improves user satisfaction through the management of question history and product comparison suggestions. Furthermore, voice authentication enables individual user recognition, enabling personalized service provision.

[0938] "User's voice input" refers to words or sounds uttered by a user through a voice input device such as a microphone.

[0939] The "means for recognizing and converting speech input to text" refers to a device or method that uses a speech recognition engine to convert speech data into character string data.

[0940] "Means for analyzing said text and understanding the user's intent" refers to the process of using text analysis algorithms and generative AI models to understand what the user is saying and infer their intent.

[0941] The "means for generating a response according to the user's intention" is a device or method for generating an appropriate answer or suggestion based on the user's intention.

[0942] The "means for converting the generated response into voice data" is a voice synthesis engine that converts text data into voice data.

[0943] The "means for outputting audio data" is a device that allows the user to hear the audio data through a speaker or the like.

[0944] "Places that sell educational products" are physical stores that sell or provide learning materials, educational tools, etc.

[0945] The "means for providing product information based on the content of a question" refers to a device or method that responds to a user's question with detailed product information.

[0946] "Means for recording user voice input and generated responses" is a process for saving voice data and text data to a recording device.

[0947] The "means for storing the recorded voice data in cloud storage" is a technology for storing voice data on a remote server via the Internet.

[0948] The "means for sharing data stored in the cloud storage" is a technology for sharing stored data with other users or systems.

[0949] "Means for managing user question and answer history and making it available for future reference" refers to a system that stores question and answer data in an easily accessible format.

[0950] "Voice authentication means for individually recognizing voice input" refers to a technology that recognizes the individual voice characteristics of a user and identifies the user.

[0951] The "means for comparing products with other products and proposing recommended products based on the content of a question" is a technology for generating and presenting comparative information on related products and recommended products based on the content of a user's question.

[0952] This invention provides a system that allows users to ask questions about educational products through voice in a brick-and-mortar store. The system receives the user's voice input, analyzes it, generates an appropriate response, and replies by voice, enabling the provision of information quickly and accurately.

[0953] System configuration

[0954] The system includes the following main components:

[0955] 1. A means of receiving user voice input

[0956] The system receives voice data emitted by a user through a microphone, which the user does using a wearable device such as a smartphone or smart glasses.

[0957] 2. Speech recognition and text conversion methods

[0958] The server converts the received voice data into text using a voice recognition engine (e.g., Google Speech Recognition API).

[0959] 3. Text Analysis and Intention Understanding

[0960] The server uses a generative AI model to analyze text data and understand user intent. Text analysis and intent understanding are achieved by using OpenAI's API.

[0961] 4. Response Generation Method

[0962] The generative AI model generates a response in text format according to the user's intentions.

[0963] 5. Speech synthesis and output methods

[0964] The server converts the generated text response into audio data using a speech synthesis engine (e.g., the pyttsx3 library) and sends it to the user's device, where it is played back.

[0965] 6. Means of providing product information based on the content of the question

[0966] When a user asks a question about a product by voice, the system provides detailed information about the product based on the content of the question. For example, in response to the question, "What does this math workbook contain?", the system will respond, "This math workbook contains practice problems for basic arithmetic operations and applied problems."

[0967] 7. Voice input and response recording

[0968] It has the means to record the user's voice input and generated responses and store them in cloud storage for future reference and analysis.

[0969] 8. How to manage question and answer history

[0970] The system manages the user's question and answer history and allows them to refer to it as needed. Specifically, it also suggests comparison information with other products and recommended products based on the content of the question.

[0971] 9. Voice Authentication Methods

[0972] The system registers the user's voice during initial setup and performs voice authentication during subsequent use, enabling individual user recognition and providing personalized services.

[0973] Program processing

[0974] The program in this system processes and calculates data in the following steps:

[0975] 1. Capture and transmit voice input

[0976] When a user asks, "What's in this math workbook?" the device captures the audio and sends the data to the server.

[0977] 2. Voice Recognition

[0978] When the server receives the voice data, it uses a speech recognition engine (e.g., Google Speech Recognition API) to convert it into text such as "What's in this math workbook?"

[0979] 3. Text Analysis and Intention Understanding

[0980] The text data is analyzed using a generative AI model to understand that the user is asking about the product content, specifically OpenAI's GPT model.

[0981] 4. Response Generation

[0982] Based on the analysis results, the generative AI model generates a response such as, "This math drill contains practice problems and applied problems for basic arithmetic operations."

[0983] 5. Speech synthesis and output

[0984] The generated text response is converted into voice data using a voice synthesis engine, sent to the terminal, and answered to the user by voice.

[0985] 6. Logging and Storage

[0986] All conversations are logged and saved in cloud storage for future reference.

[0987] This allows users to quickly and accurately obtain detailed information about educational products and enjoy personalized services through voice authentication.

[0988] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0989] Step 1:

[0990] The user asks a question.

[0991] Users use the microphone on their smartphone or smart glasses to voice-inquire about a product, such as "What does this math workbook contain?"

[0992] Step 2:

[0993] Capture and transmit audio input

[0994] The device captures the user's voice, temporarily stores this voice data, and sends it to the server using the device's network function. The input is the user's voice data, and the output is the voice data sent to the server.

[0995] Step 3:

[0996] Voice Recognition

[0997] The server converts the received voice data into text using the Google Speech Recognition API. The input is the user's voice data, and the output is the converted text data. Specifically, it is converted into the text "What is included in this math workbook?"

[0998] Step 4:

[0999] Text analysis and intent understanding

[1000] The server uses a generative AI model (OpenAI's API) to analyze the text data and understand the user's intent. The input is text data, and the output is data that is interpreted as the user's intent. Specifically, it recognizes that the user is asking a question about the content of the drill.

[1001] Step 5:

[1002] Response Generation

[1003] The server uses a generative AI model to generate an appropriate response based on the user's intent. The input is data interpreted as the user's intent, and the output is the generated response text. For example, it generates a response such as, "This math workbook contains practice problems and applied problems for basic arithmetic operations."

[1004] Step 6:

[1005] Speech synthesis and transmission

[1006] The server converts the generated response into audio data using the pyttsx3 library. The input is the text response, and the output is the generated audio data. This audio data is then sent to the device.

[1007] Step 7:

[1008] Voice response output

[1009] The terminal plays the received audio data to the user through the speaker. The input is the audio data sent from the server, and the output is the audio heard by the user.

[1010] Step 8:

[1011] Logging and Storage

[1012] The server records the user's voice input and the generated response and stores them in cloud storage. The input is the voice input and response data, and the output is the log data stored in cloud storage. We store these data for future reference.

[1013] Step 9:

[1014] Question and answer history management

[1015] The server manages the user's questions and answers as a history and makes them available for reference as needed. The input is logged data, and the output is searchable historical data.

[1016] Step 10:

[1017] Voice Authentication

[1018] The server performs voice authentication to identify the individual user. The input is the user's voice data, and the output is the result of voice authentication. If the voice authentication is successful, the user can receive personalized services.

[1019] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1020] The present invention is an educational system that receives user voice input, analyzes it, and generates appropriate responses. Furthermore, by combining it with an emotion engine that recognizes the user's emotions and adjusts the response, it provides more personalized educational support.

[1021] System Overview

[1022] When a user asks a question or makes a request by voice, the device captures the voice and sends it to the server. The server uses a speech recognition engine to convert the voice into text, analyzes the text, generates an appropriate response based on a generative AI model, converts it into voice data, and sends it back to the device. The device responds to the user by voice, and records the interaction and emotional data in a log and saves it in cloud storage.

[1023] Capture and transmit audio input

[1024] Examples:

[1025] When a user speaks to the device, "What are you going to study today?", the device captures the voice input and sends the captured voice data to the server.

[1026] Speech recognition and text conversion

[1027] Examples:

[1028] The server analyzes the received voice data using a speech recognition engine and converts it into text such as "What are you going to study today?" This text data is sent to the next step for further analysis.

[1029] Text analysis and intent understanding

[1030] Examples:

[1031] The server uses a generative AI model to analyze the text data and understand that the user is asking a question about today's learning content.

[1032] emotion recognition

[1033] Examples:

[1034] The server uses an emotion engine to analyze the user's emotions from the voice data, for example, identifying emotions such as nervousness, excitement, sadness, etc.

[1035] Response Generation

[1036] Examples:

[1037] The server generates a text response such as, "Today's study will be solving math problems." It then adjusts the response based on the user's emotions. For example, if it detects that the user is nervous, it adds encouraging words such as, "It's okay, take your time."

[1038] Speech synthesis and output

[1039] Examples:

[1040] The server converts the generated text response into voice data using a speech synthesis engine. The converted voice data is sent to the device, which then tells the user, "Today's lesson is to solve math problems. It's okay, take your time."

[1041] Recording and storing conversation logs and emotional data

[1042] Examples:

[1043] The server logs user interactions and emotional analysis data and stores it in cloud storage, where parents and educators can access the logs to track learning progress and emotional changes.

[1044] Voice Authentication

[1045] Examples:

[1046] The device performs voice authentication to uniquely identify the user. The user's voice is recorded during initial setup, and voice authentication technology is used to identify the user during subsequent interactions.

[1047] In this way, the system of the present invention provides personalized learning support through a series of steps, from voice input to text conversion, response generation, voice output, emotion analysis, and log recording. It is also possible to share information with parents and educators, which increases the user's motivation to learn and provides effective educational support.

[1048] The processing flow will be explained below.

[1049] Step 1:

[1050] The user speaks to the device. For example, the user might say, "What are you going to study today?"

[1051] Step 2:

[1052] The device captures the user's voice. The device's built-in microphone records the user's voice data.

[1053] Step 3:

[1054] The device sends the captured audio data to the server, which then uploads the audio data to the server via the network.

[1055] Step 4:

[1056] The server receives the voice data. The server receives the voice data sent from the terminal.

[1057] Step 5:

[1058] The server uses a speech recognition engine to convert the voice data into text. For example, the server converts the voice data of "What are you going to study today?" into text format.

[1059] Step 6:

[1060] The server analyzes the text data. Using a generative AI model, the server analyzes the content of the text and understands the user's intent. The server determines that the user's question is related to today's learning content.

[1061] Step 7:

[1062] The server uses an emotion engine to analyze the user's emotion from the voice data, and determines that the user is, for example, "excited."

[1063] Step 8:

[1064] The server generates a response based on the user's intention and emotions. Based on the analysis results, the server generates a text response such as "Today's study is to solve math problems," and adds encouraging words such as "Have fun while you do it!", taking into account the user's excitement.

[1065] Step 9:

[1066] The server converts the text response into audio data. The server uses a speech synthesis engine to convert the generated text response into audio format.

[1067] Step 10:

[1068] The server transmits the voice data to the terminal, and the server sends the generated voice data to the terminal via the network.

[1069] Step 11:

[1070] The terminal receives the voice data. The terminal receives the voice data sent from the server.

[1071] Step 12:

[1072] The device plays back the audio data. Using the speaker, the device conveys to the user the following message: "Today's lesson is to solve a collection of math problems. Have fun doing it!"

[1073] Step 13:

[1074] The server records the user's speech input and the generated responses, and stores this interaction in a log file.

[1075] Step 14:

[1076] The server saves the logs in cloud storage, and then uploads the recorded voice data, text data, and emotion data to cloud storage for storage.

[1077] Step 15:

[1078] The server shares the data stored in the cloud storage, and sets up data sharing so that parents and educators can access it.

[1079] Step 16:

[1080] The device performs voice authentication to uniquely identify the user. The user's voice is recorded during initial setup, and voice authentication technology is used to identify the user during subsequent interactions.

[1081] In this way, the system of the present invention provides personalized learning support through a series of steps from user voice input to analysis, response generation, voice output, emotion analysis, and log recording.

[1082] Example 2

[1083] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1084] The main function of conventional educational support systems was to convert a user's voice input into text and generate an appropriate response. However, these systems were unable to provide responses that took the user's emotional state into account, making it difficult to provide fully personalized learning support. Furthermore, they lacked the authentication functionality to individually recognize each user's voice input, making operation difficult when multiple users were using the system. This made it impossible to provide efficient and effective educational support.

[1085] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1086] In this invention, the server includes means for receiving a user's voice input, means for recognizing the voice input and converting it into text, means for analyzing the text and understanding the user's intention, means for recognizing the user's emotions, means for generating a response according to the user's intention and emotion, means for converting the generated response into voice data, and means for outputting the voice data. This makes it possible to provide an appropriate response to the user's voice input that takes emotion into consideration. Furthermore, by further including authentication means for individually recognizing voice inputs, operation when multiple users use the system becomes smoother.

[1087] "Voice input" refers to the act of converting a user's speech into a digital signal and transmitting it to the system, or that digital signal.

[1088] "Text-to-text" refers to the process of converting digital data into text using speech recognition technology to represent spoken input as text.

[1089] "Intent understanding" refers to the process of analyzing input text data and recognizing what the user is looking for.

[1090] "Emotion recognition" refers to the process of analyzing and identifying a user's emotional state from voice or text data.

[1091] "Response generation" refers to the process of creating an appropriate reply based on the user's intent and feelings.

[1092] "Speech synthesis" refers to the process of converting text data into speech data and outputting it as natural-sounding speech.

[1093] "Network storage" refers to online storage space where data is stored on remote servers or cloud services and accessed and shared as needed.

[1094] "Authentication method" refers to a technology or process that identifies an individual user by analyzing the user's voice characteristics.

[1095] The present invention is an educational support system that receives user voice input, analyzes it using a speech recognition and emotion recognition engine, and generates an appropriate response. This system provides personalized educational support by combining a speech recognition engine, a text analysis engine, an emotion recognition engine, a generative AI model, and a speech synthesis engine.

[1096] When a user speaks, the device captures the speech and sends it to a server. The server then uses a speech recognition engine to convert the speech data into text. This speech recognition uses commonly available speech recognition technology (e.g., a cloud-based speech recognition service). The converted text data is then sent to a text analysis engine (e.g., a generative AI model) to analyze the user's intent.

[1097] Furthermore, the voice data is analyzed by an emotion recognition engine to identify the user's emotional state. Emotion recognition is performed using commonly available emotion analysis techniques (e.g., emotion recognition services and engines). Based on the analysis results, an appropriate response is generated in text format by a generative AI model.

[1098] Next, the generated text response is converted into voice data using a voice synthesis engine. This voice synthesis is performed using a commonly used voice synthesis technology (e.g., a voice synthesis service). The converted voice data is sent to the terminal and a voice response is provided to the user.

[1099] The system also has the ability to record interaction logs and emotional data and store them in network storage. Parents and educators can access these logs to check learning progress and emotional changes. For example, this data can be securely stored and shared using cloud storage such as AWS S3.

[1100] The system also uses voice authentication technology to identify individual users. During initial setup, the user's voice is recorded and saved in a profile. The system then matches each voice input with this profile to identify the user. This technology makes it possible to provide individual learning support when multiple users use the same system.

[1101] Specific examples

[1102] For example, if a user says to the device, "What are you going to study today?", the device captures the speech and sends it to the server. The server uses a speech recognition engine to convert the speech into text, "What are you going to study today?" and then uses a generative AI model to analyze the text and understand that the user is asking a question about what they're studying today. If the emotion recognition engine detects nervousness in the user's voice, the generated response might be something like, "Today's study is to solve math problems. It's okay, take your time." The server then converts this text response into audio data using a speech synthesis engine and sends it back to the device. Finally, the device tells the user, "Today's study is to solve math problems. It's okay, take your time."

[1103] Prompt Sentence Examples

[1104] User Question: "What are you studying today?"

[1105] User voice analysis results: "Nervous"

[1106] Generate Response: "Today's lesson is to do some math problems. It's okay, just take your time."

[1107] By combining these specific technologies, the system of the present invention provides advanced and personalized educational support that meets the specific needs of each user.

[1108] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1109] Step 1:

[1110] Capture and transmit audio input

[1111] When a user speaks into the device, the device uses its built-in microphone to capture the voice. Specifically, the user's voice (input) is converted into a digital signal, and the voice data is encoded and sent to the server (output). For example, if a user says, "What are we going to study today?", the voice is captured and digitized. This series of operations is performed in real time.

[1112] Step 2:

[1113] Speech recognition and text conversion

[1114] The server sends the voice data (input) received from the device to a voice recognition engine. Specifically, the voice data is converted into text data (output) using a voice recognition engine (for example, a cloud-based voice recognition service). For example, the voice data "What are you going to study today?" is converted into text as "What are you going to study today?" This converted text data is sent to the next step.

[1115] Step 3:

[1116] Text analysis and intent understanding

[1117] The server analyzes the text data (input) received from the speech recognition engine. Specifically, it uses a generative AI model (e.g., a generative AI model) to analyze the text data and understand the user's intent (output). For example, it analyzes the text "What are we going to study today?" and identifies the intent as "A question about today's learning content." This intent data is sent to the next step.

[1118] Step 4:

[1119] emotion recognition

[1120] The server sends the voice data and text data (input) from the speech recognition engine to the emotion recognition engine. The emotion recognition engine (e.g., emotion analysis service) is used to analyze the user's emotional state (output). For example, it identifies tension, excitement, happiness, etc. from the tone and speed of the voice. This emotion data is sent to the next step.

[1121] Step 5:

[1122] Response Generation

[1123] The server uses a generative AI model to generate an appropriate response based on the intent data and emotion data (input). Specifically, it creates a text response and adjusts it based on the emotion data (output). For example, if it recognizes that the user is nervous, it adds encouraging words such as, "Today's study is to solve your math problems. It's okay, take your time." This adjusted response is sent to the next step.

[1124] Step 6:

[1125] Speech synthesis and output

[1126] The server sends the generated text response (input) to a speech synthesis engine, which converts it into voice data. The speech synthesis engine (e.g., a speech synthesis service) generates the text as natural-sounding voice data (output). For example, voice data such as "Today's study is to solve the math problems. It's okay, take your time" is generated. This voice data is then sent to the device.

[1127] Step 7:

[1128] Recording and storing conversation logs and emotional data

[1129] The server logs user interactions and emotional data (input) and stores them in network storage, such as AWS S3, for later access by parents and educators (output). Specifically, the server compiles user questions, system responses, and emotional analysis results into a single log file.

[1130] Step 8:

[1131] Voice Authentication

[1132] During initial setup, the device records the user's voice and saves it in a profile. It then uses voice authentication technology (e.g., a voice authentication service) to identify the user with each interaction (input and output). For example, new voice input can be matched against the profile to authenticate the user. This allows multiple users to receive personalized learning assistance even when using the same system.

[1133] (Application example 2)

[1134] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1135] Conventional customer support and guidance systems in brick-and-mortar stores could only provide uniform answers to customer questions, making it difficult to respond taking into account the circumstances and emotions of each individual customer. They also lacked a mechanism for accumulating and analyzing customer responses and emotions. As a result, customer satisfaction declined, and improving the quality of store service became an issue.

[1136] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving a user's voice input, means for recognizing the voice input and converting it into text, means for analyzing the text and understanding the user's intention, means for generating a response according to the user's intention, means for recognizing the user's emotion and adjusting the response based on the emotion, means for converting the generated response into voice data, and means for outputting the voice data. This makes it possible to provide a personalized response to a customer's question according to the situation and emotion, thereby improving customer satisfaction and service quality.

[1137] The "means for receiving user voice input" is a device or module for capturing voice uttered by the user and incorporating it into the system.

[1138] The "means for recognizing the voice input and converting it into text" refers to an algorithm or engine for analyzing the received voice data and converting it into text information.

[1139] "Means for analyzing the text and understanding the user's intent" refers to AI models and natural language processing technologies that analyze text data to understand the content of questions and requests uttered by users.

[1140] "Means for generating a response according to the user's intent" refers to a generative AI model or response engine for generating appropriate and effective answers based on the user's intent.

[1141] "Means for recognizing the user's emotions and adjusting the response based on those emotions" refers to an algorithm or engine that analyzes voice data or text data to identify the user's emotions and optimizes the response content based on the results.

[1142] The "means for converting the generated response into voice data" is a voice synthesis engine for converting the response in text format into voice data.

[1143] The "means for outputting audio data" refers to an output device for audibly transmitting audio data to the user.

[1144] "Cloud storage" is a data storage service for storing data in a remote location accessible via the Internet.

[1145] "Voice authentication means" is a technology for individually identifying a speaker from voice data.

[1146] System Configuration

[1147] This invention is a system for providing advanced customer support in a customer support and guidance system, and is composed of the following main components:

[1148] 1. Voice input receiving devices: These include smart glasses and head-mounted displays. These devices capture the user's voice input.

[1149] 2. Speech recognition engine: An engine for converting voice data received on the server side into text data. As a specific example, Google Cloud Speech-to-Text API is used.

[1150] 3. Text analysis module: A generative AI model that analyzes the text data and understands the user's intent. OpenAI GPT-4 is used as a concrete example.

[1151] 4. Emotion Recognition Engine: An algorithm for analyzing user emotions from voice data, allowing for tailored responses.

[1152] 5. Speech synthesis engine: An engine for converting the generated text response into audio data. For example, Google Cloud Text-to-Speech API is used.

[1153] 6. Cloud Storage: Data storage service for storing and sharing generated response and emotion data.

[1154] System Operation

[1155] Capture and transmit audio input

[1156] When a user asks a question into the smart glasses or head-mounted display, the device captures the voice input and sends it to the server. For example, the user might say, "Please tell me the features of this product."

[1157] Speech recognition and text conversion

[1158] The server analyzes the received voice data using the Google Cloud Speech-to-Text API and converts the voice data into text data. This step generates the text "Please tell me the features of this product."

[1159] Text analysis and intent understanding

[1160] The server uses a generative AI model (OpenAI GPT-4) to analyze the text data and understand the intent of the user's question. For example, in this case, it interprets the user as "asking about the product's features."

[1161] emotion recognition

[1162] The voice data is analyzed using the emotion_recognition algorithm to identify the user's emotions, such as "nervous," "excited," or "sad."

[1163] Response Generation

[1164] The server generates an appropriate response based on the analysis results. For example, a text response such as "This product is made of high-quality materials and is particularly durable" is generated. If the user's emotion is identified as "tension," the response will include a phrase such as "Don't worry, take your time looking around."

[1165] Speech synthesis and output

[1166] The generated text response is converted to audio data using the Google Cloud Text-to-Speech API and sent to a smart eyeglasses or head-mounted display, where the user receives the answer via audio.

[1167] Specific prompt examples

[1168] A user asks, "What is this product?" How do you respond?

[1169] Data recording and storage

[1170] The server logs user interaction and emotional data and stores it in cloud storage, which is then analyzed and used to improve the store's services.

[1171] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1172] Step 1:

[1173] A user asks a question to smart glasses or a head-mounted display. The user's voice input is captured by the device and the voice data is sent to the server. The input is the user's voice data, and the output is the voice data sent to the server.

[1174] Step 2:

[1175] The server analyzes the received voice data using a speech recognition engine (specifically, the Google Cloud Speech-to-Text API). The input is the voice data sent to the server, and the output is the converted text data. The server then sends this data to the next step.

[1176] Step 3:

[1177] The server uses a text analysis module to analyze the text data and understand the user's intent using a generative AI model (OpenAI GPT-4). For example, if a user says, "Please tell me the features of this product," the server interprets the intent as "I am asking about the product's features." The input is the text data to be analyzed, and the output is the interpreted user intent.

[1178] Step 4:

[1179] The server analyzes the voice and text data using the emotion_recognition algorithm to identify the user's emotions. The input is voice or text data, and the output is the emotion recognition result, such as "nervous" or "excited."

[1180] Step 5:

[1181] The server uses a generative AI model to generate a response based on the user's understood intent and the identified emotion. For example, it generates a text response such as "This product is made of high-quality materials and is particularly durable." If the emotion is "tension," it adds "Don't worry, take your time looking around." The input is the user's intent and emotion, and the output is the generated text response.

[1182] Step 6:

[1183] The server converts the generated text response into audio data using a speech synthesis engine (Google Cloud Text-to-Speech API). The input is the text response, and the output is audio data. This audio data is sent to the device.

[1184] Step 7:

[1185] The terminal plays the received voice data and conveys the response to the user. The input is the voice data sent from the server, and the output is the voice response conveyed to the user.

[1186] Step 8:

[1187] The server logs user interaction and emotional data and stores them in cloud storage. The input is the user's voice input, response, and emotional data, and the output is the log data stored in cloud storage. The stored data is later analyzed and used to improve the service.

[1188] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1189] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1190] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1191] [Fourth embodiment]

[1192] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1193] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1194] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1195] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1196] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1197] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1198] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1199] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1200] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1201] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1202] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1203] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1204] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1205] The present invention is an educational system that receives voice input from a user (especially a child), analyzes it, and generates an appropriate response. This system includes a series of processes that operate between a server, a terminal, and the user.

[1206] System Overview

[1207] When a user asks a question or makes a request via voice, the device captures the speech and sends it to the server. The server uses a speech recognition engine to convert the speech to text and analyzes the text. It then generates an appropriate response based on a generative AI model, converts the response into audio data, and sends it back to the device. The device then responds to the user via voice, logs the interaction, and stores it in cloud storage.

[1208] Capture and transmit audio input

[1209] Examples:

[1210] When a user speaks to the device, "What are you going to study today?", the device captures the voice input and sends the captured voice data to the server.

[1211] Speech recognition and text conversion

[1212] Examples:

[1213] The server analyzes the received voice data with a speech recognition engine and converts it into text such as "What will you study today?" This text data is sent to the next step for further analysis.

[1214] Text analysis and intent understanding

[1215] Examples:

[1216] The server uses a generative AI model to analyze the text data and understand that the user is asking a question about today's learning content.

[1217] Response Generation

[1218] Examples:

[1219] The server generates a text response such as, "Today's study will be solving math problems." The generative AI model creates advice and suggestions based on the training data.

[1220] Speech synthesis and output

[1221] Examples:

[1222] The server converts the generated text response into voice data using a speech synthesis engine, which is then sent to the device, which then tells the user, "Today's study will be solving math problems."

[1223] Recording and saving conversation logs

[1224] Examples:

[1225] The server logs all user interactions and stores them in cloud storage, where parents and educators can access the logs to track learning progress.

[1226] Voice Authentication

[1227] Examples:

[1228] The device records the user's voice during initial setup and performs voice authentication during subsequent interactions, allowing only authenticated users to use the system.

[1229] In this way, the system of the present invention provides personalized learning support through a series of steps, from voice input to text conversion, response generation, voice output, and log recording. It is also possible to share information with parents and educators, which increases the user's motivation to learn and provides effective educational support.

[1230] The processing flow will be explained below.

[1231] Step 1:

[1232] The user speaks to the device. For example, the user might say, "What will I study today?"

[1233] Step 2:

[1234] The device captures the user's voice. The device's built-in microphone records the user's voice data.

[1235] Step 3:

[1236] The device sends the captured audio data to the server, which then uploads the audio data to the server via the network.

[1237] Step 4:

[1238] The server receives the voice data. The server receives the voice data sent from the terminal.

[1239] Step 5:

[1240] The server uses a speech recognition engine to convert the voice data into text. For example, the server converts the voice data of "What will you study today?" into text format.

[1241] Step 6:

[1242] The server analyzes the text data and uses a generative AI model to analyze the content of the text and understand the user's intent.

[1243] Step 7:

[1244] The server generates a response based on the user's intent. Based on the analysis results, the server generates an appropriate text response such as "Today's study will be solving math problems."

[1245] Step 8:

[1246] The server converts the text response into audio data. The server uses a speech synthesis engine to convert the text response into audio format.

[1247] Step 9:

[1248] The server transmits the generated voice data to the terminal via the network.

[1249] Step 10:

[1250] The terminal receives the voice data. The terminal receives the voice data sent from the server.

[1251] Step 11:

[1252] The device plays back the audio data. Using the speaker, the device plays back the audio, "Today's study will be solving math problems."

[1253] Step 12:

[1254] The server records the user's speech input and the generated responses, and stores this interaction in a log file.

[1255] Step 13:

[1256] The server saves the logs in cloud storage, and the server uploads the recorded audio and text data to cloud storage for storage.

[1257] Step 14:

[1258] The server shares the data stored in the cloud storage, and sets up data sharing so that parents and educators can access it.

[1259] Step 15:

[1260] The device performs voice authentication to uniquely identify the user. During initial setup, the user's voice is recorded and voice authentication technology is used to identify the user in subsequent interactions.

[1261] Example 1

[1262] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1263] Modern education demands personalized learning support. However, conventional learning systems face challenges in efficiently recognizing users' voice input and generating appropriate responses. Furthermore, they lack the ability to record and share individual learning progress. To address these challenges, an advanced system combining speech recognition and generative artificial intelligence models is needed.

[1264] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1265] In this invention, the server includes means for receiving a user's voice input, means for recognizing the voice input and converting it into text, means for analyzing the text and understanding the user's intent, means for generating a response according to the user's intent using a generative AI model, means for converting the generated response into voice data, means for outputting the voice data, and means for recording the voice input and the generated response. This enables personalized learning support, generates appropriate responses from the user's voice input, and makes it possible to effectively record and share learning progress.

[1266] "Means for receiving user voice input" refers to a device or system for capturing user-uttered speech and processing it as data.

[1267] "Means for recognizing and converting said voice input to text" refers to a technology or system that analyzes received voice data and converts it into corresponding text data.

[1268] "Means for analyzing the text and understanding the user's intent" refers to an algorithm or system that analyzes text data to determine what the user is looking for.

[1269] "Means for generating responses based on user intent using a generative artificial intelligence model" refers to the process of using technologies such as deep learning and machine learning to generate appropriate responses based on user input.

[1270] "Means for converting the generated response into voice data" refers to a system that converts a text-format response into voice data using voice synthesis technology.

[1271] "Means for outputting audio data" refers to a speaker or other audio device that allows the user to hear the generated audio data.

[1272] "Means for recording the voice input and generated responses" refers to a system that stores interactions with a user as a log.

[1273] "Means for storing recorded voice data in cloud storage" refers to technology that transfers and stores locally stored voice data in cloud-based storage via the Internet.

[1274] "Means for sharing data stored in cloud storage" refers to the functionality that allows other users or systems with access rights to the stored data to view and use the data.

[1275] "Voice authentication means for uniquely recognizing voice input" refers to technology that identifies and authenticates individuals based on their voice characteristics.

[1276] The present invention provides an educational support system that receives voice input from a user, converts the voice data into text data using a server and a terminal, analyzes the data, and generates an appropriate response. Detailed embodiments for carrying out the present invention will be described below.

[1277] Hardware Configuration

[1278] This system consists of a terminal used by the user and a server that communicates with it. The terminal is equipped with a high-sensitivity microphone and speaker, and includes a network communication module such as Wi-Fi or LTE as needed. The server has a high-performance processor and sufficient memory, and is connected to a cloud network.

[1279] Software Configuration

[1280] The software of this system consists of the following main components:

[1281] 1. Speech recognition engine (e.g., Google Cloud Speech-to-Text API)

[1282] 2. Natural Language Processing (NLP) engine (e.g., Spashitai)

[1283] 3. Generative AI models (e.g., GPT-4)

[1284] 4. Speech synthesis engine (e.g., Amazon Polly or Google Cloud Text-to-Speech API)

[1285] 5. Cloud storage services (e.g., Amazon S3 and Google Cloud Storage)

[1286] User voice input

[1287] Users speak into the device to ask questions or make requests about their studies, such as, "What are we going to study today?", and the device's sensitive microphone captures this voice.

[1288] Sending audio data

[1289] The device sends the captured audio data in binary format to the server, using Wi-Fi or LTE for network communication.

[1290] Speech recognition and text conversion

[1291] The server then calls the Google Cloud Speech-to-Text API to analyze the received voice data and converts it into text data, such as "What are you going to study today?"

[1292] Text analysis and intent understanding

[1293] The server uses a generative AI model (e.g., GPT-4) to analyze the text data and understand the user's intent. The generative AI model uses the following prompt:

[1294] "Here's a question about what you learned today. Please generate an appropriate response."

[1295] Response Generation

[1296] The server uses a generative AI model to generate a response based on the user's intentions, such as "Today's study will involve solving math problems."

[1297] Speech synthesis and response transmission

[1298] The server converts the generated text response into audio data using Amazon Polly or the Google Cloud Text-to-Speech API, and sends the audio data to the device, which then tells the user through the speaker, "Today's lesson will be solving math problems."

[1299] Recording and saving conversation logs

[1300] The server logs all user interactions and stores them in a cloud storage service (e.g., Amazon S3 or Google Cloud Storage), where the data can be later viewed by parents and educators.

[1301] Voice Authentication

[1302] The terminal uses the user's voice for voice authentication. The user's voice is recorded during initial setup and authentication is performed based on that data. Only authenticated users can use the system.

[1303] As described above, the system of the present invention provides personalized learning support to users by smoothly performing a series of processes from voice input to text conversion, response generation, voice output, and log recording. It also serves as a powerful tool for other users to improve the quality of their education.

[1304] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1305] Step 1:

[1306] The user speaks to the device, "What will I study today?" The device captures the voice using a built-in high-sensitivity microphone. The input is the user's voice input, and the output is the captured voice data.

[1307] Step 2:

[1308] The device transmits the captured audio data to the server using a network communication module (Wi-Fi or LTE). The input is the captured audio data, and the output is the audio data transmitted to the server.

[1309] Step 3:

[1310] The server inputs the received voice data into the Google Cloud Speech-to-Text API and converts the voice into text data. The input is the voice data received by the server, and the output is the converted text data, "What are you going to study today?"

[1311] Step 4:

[1312] The server inputs the text data into a generative AI model (e.g., GPT-4) for analysis. The prompt text is "We have received a question about today's learning content. Please generate an appropriate response." The input is the text data and the prompt text, and the output is the understood intent (understanding the intent of the question) as the result of analysis.

[1313] Step 5:

[1314] The server uses a generative AI model to generate a response based on the user's intent. The input is the analysis result and the user's intent, and the output is the generated text response, "Today's study is to solve math problems."

[1315] Step 6:

[1316] The server inputs the generated text response into a speech synthesis engine (e.g., Amazon Polly) to convert the text to speech data. The input is the generated text response, and the output is the converted speech data.

[1317] Step 7:

[1318] The server sends the converted voice data to the terminal, where the input is the converted voice data and the output is the voice data sent to the terminal.

[1319] Step 8:

[1320] The device plays the voice data received from the server over the speaker and tells the user, "Today's study will be solving math problems." The input is the received voice data, and the output is a voice response.

[1321] Step 9:

[1322] The server records all user interactions as log data and stores it in a cloud storage service (e.g., Amazon S3). The input is the collected voice input and generated response log data, and the output is the cloud-stored log data.

[1323] (Application example 1)

[1324] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1325] Currently, physical stores that sell educational products face the challenge of making it difficult for users to quickly and accurately obtain detailed product information. Furthermore, there is a lack of systems for managing user inquiry histories and comparison information about products, hindering efficient customer service and improving user satisfaction. Furthermore, individual user recognition through voice authentication is insufficient, making it difficult to provide personalized services.

[1326] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1327] In this invention, the server includes: means for receiving a user's voice input; means for recognizing the voice input and converting it into text; means for analyzing the text and understanding the user's intent; means for generating a response according to the user's intent; means for converting the generated response into voice data; means for outputting the voice data; means for allowing users to ask product-related questions at locations selling educational products; means for providing product information based on the questions; means for recording the user's voice input and the generated responses; means for storing the recorded voice data in cloud storage; means for sharing the data stored in the cloud storage; means for managing the user's question and answer history for future reference; voice authentication means for individually recognizing the voice input; and means for comparing the user's question with other products and suggesting recommended products based on the question content. This allows users to quickly and accurately obtain detailed information about educational products, and improves user satisfaction through the management of question history and product comparison suggestions. Furthermore, voice authentication enables individual user recognition, enabling personalized service provision.

[1328] "User's voice input" refers to words or sounds uttered by a user through a voice input device such as a microphone.

[1329] The "means for recognizing and converting speech input to text" refers to a device or method that uses a speech recognition engine to convert speech data into character string data.

[1330] "Means for analyzing said text and understanding the user's intent" refers to the process of using text analysis algorithms and generative AI models to understand what the user is saying and infer their intent.

[1331] The "means for generating a response according to the user's intention" is a device or method for generating an appropriate answer or suggestion based on the user's intention.

[1332] The "means for converting the generated response into voice data" is a voice synthesis engine that converts text data into voice data.

[1333] The "means for outputting audio data" is a device that allows the user to hear the audio data through a speaker or the like.

[1334] "Places that sell educational products" are physical stores that sell or provide learning materials, educational tools, etc.

[1335] The "means for providing product information based on the content of a question" refers to a device or method that responds to a user's question with detailed product information.

[1336] "Means for recording user voice input and generated responses" is a process for saving voice data and text data to a recording device.

[1337] The "means for storing the recorded voice data in cloud storage" is a technology for storing voice data on a remote server via the Internet.

[1338] The "means for sharing data stored in the cloud storage" is a technology for sharing stored data with other users or systems.

[1339] "Means for managing user question and answer history and making it available for future reference" refers to a system that stores question and answer data in an easily accessible format.

[1340] "Voice authentication means for individually recognizing voice input" refers to a technology that recognizes the individual voice characteristics of a user and identifies the user.

[1341] The "means for comparing products with other products and proposing recommended products based on the content of a question" is a technology for generating and presenting comparative information on related products and recommended products based on the content of a user's question.

[1342] This invention provides a system that allows users to ask questions about educational products through voice in a brick-and-mortar store. The system receives the user's voice input, analyzes it, generates an appropriate response, and replies by voice, enabling the provision of information quickly and accurately.

[1343] System configuration

[1344] The system includes the following main components:

[1345] 1. A means of receiving user voice input

[1346] The system receives voice data emitted by a user through a microphone, which the user does using a wearable device such as a smartphone or smart glasses.

[1347] 2. Speech recognition and text conversion methods

[1348] The server converts the received voice data into text using a voice recognition engine (e.g., Google Speech Recognition API).

[1349] 3. Text Analysis and Intention Understanding

[1350] The server uses a generative AI model to analyze text data and understand user intent. Text analysis and intent understanding are achieved by using OpenAI's API.

[1351] 4. Response Generation Method

[1352] The generative AI model generates a response in text format according to the user's intentions.

[1353] 5. Speech synthesis and output methods

[1354] The server converts the generated text response into audio data using a speech synthesis engine (e.g., the pyttsx3 library) and sends it to the user's device, where it is played back.

[1355] 6. Means of providing product information based on the content of the question

[1356] When a user asks a question about a product by voice, the system provides detailed information about the product based on the content of the question. For example, in response to the question, "What does this math workbook contain?", the system will respond, "This math workbook contains practice problems for basic arithmetic operations and applied problems."

[1357] 7. Voice input and response recording

[1358] It has the means to record the user's voice input and generated responses and store them in cloud storage for future reference and analysis.

[1359] 8. How to manage question and answer history

[1360] The system manages the user's question and answer history and allows them to refer to it as needed. Specifically, it also suggests comparison information with other products and recommended products based on the content of the question.

[1361] 9. Voice Authentication Methods

[1362] The system registers the user's voice during initial setup and performs voice authentication during subsequent use, enabling individual user recognition and providing personalized services.

[1363] Program processing

[1364] The program in this system processes and calculates data in the following steps:

[1365] 1. Capture and transmit voice input

[1366] When a user asks, "What's in this math workbook?" the device captures the audio and sends the data to the server.

[1367] 2. Voice Recognition

[1368] When the server receives the voice data, it uses a speech recognition engine (e.g., Google Speech Recognition API) to convert it into text such as "What's in this math workbook?"

[1369] 3. Text Analysis and Intention Understanding

[1370] The text data is analyzed using a generative AI model to understand that the user is asking about the product content, specifically OpenAI's GPT model.

[1371] 4. Response Generation

[1372] Based on the analysis results, the generative AI model generates a response such as, "This math drill contains practice problems and applied problems for basic arithmetic operations."

[1373] 5. Speech synthesis and output

[1374] The generated text response is converted into voice data using a voice synthesis engine, sent to the terminal, and answered to the user by voice.

[1375] 6. Logging and Storage

[1376] All conversations are logged and saved in cloud storage for future reference.

[1377] This allows users to quickly and accurately obtain detailed information about educational products and enjoy personalized services through voice authentication.

[1378] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1379] Step 1:

[1380] The user asks a question.

[1381] Users use the microphone on their smartphone or smart glasses to voice-inquire about a product, such as "What does this math workbook contain?"

[1382] Step 2:

[1383] Capture and transmit audio input

[1384] The device captures the user's voice, temporarily stores this voice data, and sends it to the server using the device's network function. The input is the user's voice data, and the output is the voice data sent to the server.

[1385] Step 3:

[1386] Voice Recognition

[1387] The server converts the received voice data into text using the Google Speech Recognition API. The input is the user's voice data, and the output is the converted text data. Specifically, it is converted into the text "What is included in this math workbook?"

[1388] Step 4:

[1389] Text analysis and intent understanding

[1390] The server uses a generative AI model (OpenAI's API) to analyze the text data and understand the user's intent. The input is text data, and the output is data that is interpreted as the user's intent. Specifically, it recognizes that the user is asking a question about the content of the drill.

[1391] Step 5:

[1392] Response Generation

[1393] The server uses a generative AI model to generate an appropriate response based on the user's intent. The input is data interpreted as the user's intent, and the output is the generated response text. For example, it generates a response such as, "This math workbook contains practice problems and applied problems for basic arithmetic operations."

[1394] Step 6:

[1395] Speech synthesis and transmission

[1396] The server converts the generated response into audio data using the pyttsx3 library. The input is the text response, and the output is the generated audio data. This audio data is then sent to the device.

[1397] Step 7:

[1398] Voice response output

[1399] The terminal plays the received audio data to the user through the speaker. The input is the audio data sent from the server, and the output is the audio heard by the user.

[1400] Step 8:

[1401] Logging and Storage

[1402] The server records the user's voice input and the generated response and stores them in cloud storage. The input is the voice input and response data, and the output is the log data stored in cloud storage. We store these data for future reference.

[1403] Step 9:

[1404] Question and answer history management

[1405] The server manages the user's questions and answers as a history and makes them available for reference as needed. The input is logged data, and the output is searchable historical data.

[1406] Step 10:

[1407] Voice Authentication

[1408] The server performs voice authentication to identify the individual user. The input is the user's voice data, and the output is the result of voice authentication. If the voice authentication is successful, the user can receive personalized services.

[1409] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1410] The present invention is an educational system that receives user voice input, analyzes it, and generates appropriate responses. Furthermore, by combining it with an emotion engine that recognizes the user's emotions and adjusts the response, it provides more personalized educational support.

[1411] System Overview

[1412] When a user asks a question or makes a request by voice, the device captures the voice and sends it to the server. The server uses a speech recognition engine to convert the voice into text, analyzes the text, generates an appropriate response based on a generative AI model, converts it into voice data, and sends it back to the device. The device responds to the user by voice, and records the interaction and emotional data in a log and saves it in cloud storage.

[1413] Capture and transmit audio input

[1414] Examples:

[1415] When a user speaks to the device, "What are you going to study today?", the device captures the voice input and sends the captured voice data to the server.

[1416] Speech recognition and text conversion

[1417] Examples:

[1418] The server analyzes the received voice data using a speech recognition engine and converts it into text such as "What are you going to study today?" This text data is sent to the next step for further analysis.

[1419] Text analysis and intent understanding

[1420] Examples:

[1421] The server uses a generative AI model to analyze the text data and understand that the user is asking a question about today's learning content.

[1422] emotion recognition

[1423] Examples:

[1424] The server uses an emotion engine to analyze the user's emotions from the voice data, for example, identifying emotions such as nervousness, excitement, sadness, etc.

[1425] Response Generation

[1426] Examples:

[1427] The server generates a text response such as, "Today's study will be solving math problems." It then adjusts the response based on the user's emotions. For example, if it detects that the user is nervous, it adds encouraging words such as, "It's okay, take your time."

[1428] Speech synthesis and output

[1429] Examples:

[1430] The server converts the generated text response into voice data using a speech synthesis engine. The converted voice data is sent to the device, which then tells the user, "Today's lesson is to solve math problems. It's okay, take your time."

[1431] Recording and storing conversation logs and emotional data

[1432] Examples:

[1433] The server logs user interactions and emotional analysis data and stores it in cloud storage, where parents and educators can access the logs to track learning progress and emotional changes.

[1434] Voice Authentication

[1435] Examples:

[1436] The device performs voice authentication to uniquely identify the user. The user's voice is recorded during initial setup, and voice authentication technology is used to identify the user during subsequent interactions.

[1437] In this way, the system of the present invention provides personalized learning support through a series of steps, from voice input to text conversion, response generation, voice output, emotion analysis, and log recording. It is also possible to share information with parents and educators, which increases the user's motivation to learn and provides effective educational support.

[1438] The processing flow will be explained below.

[1439] Step 1:

[1440] The user speaks to the device. For example, the user might say, "What are you going to study today?"

[1441] Step 2:

[1442] The device captures the user's voice. The device's built-in microphone records the user's voice data.

[1443] Step 3:

[1444] The device sends the captured audio data to the server, which then uploads the audio data to the server via the network.

[1445] Step 4:

[1446] The server receives the voice data. The server receives the voice data sent from the terminal.

[1447] Step 5:

[1448] The server uses a speech recognition engine to convert the voice data into text. For example, the server converts the voice data of "What are you going to study today?" into text format.

[1449] Step 6:

[1450] The server analyzes the text data. Using a generative AI model, the server analyzes the content of the text and understands the user's intent. The server determines that the user's question is related to today's learning content.

[1451] Step 7:

[1452] The server uses an emotion engine to analyze the user's emotion from the voice data, and determines that the user is, for example, "excited."

[1453] Step 8:

[1454] The server generates a response based on the user's intention and emotions. Based on the analysis results, the server generates a text response such as "Today's study is to solve math problems," and adds encouraging words such as "Have fun while you do it!", taking into account the user's excitement.

[1455] Step 9:

[1456] The server converts the text response into audio data. The server uses a speech synthesis engine to convert the generated text response into audio format.

[1457] Step 10:

[1458] The server transmits the voice data to the terminal, and the server sends the generated voice data to the terminal via the network.

[1459] Step 11:

[1460] The terminal receives the voice data. The terminal receives the voice data sent from the server.

[1461] Step 12:

[1462] The device plays back the audio data. Using the speaker, the device conveys to the user the following message: "Today's lesson is to solve a collection of math problems. Have fun doing it!"

[1463] Step 13:

[1464] The server records the user's speech input and the generated responses, and stores this interaction in a log file.

[1465] Step 14:

[1466] The server saves the logs in cloud storage, and then uploads the recorded voice data, text data, and emotion data to cloud storage for storage.

[1467] Step 15:

[1468] The server shares the data stored in the cloud storage, and sets up data sharing so that parents and educators can access it.

[1469] Step 16:

[1470] The device performs voice authentication to uniquely identify the user. The user's voice is recorded during initial setup, and voice authentication technology is used to identify the user during subsequent interactions.

[1471] In this way, the system of the present invention provides personalized learning support through a series of steps from user voice input to analysis, response generation, voice output, emotion analysis, and log recording.

[1472] Example 2

[1473] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1474] The main function of conventional educational support systems was to convert a user's voice input into text and generate an appropriate response. However, these systems were unable to provide responses that took the user's emotional state into account, making it difficult to provide fully personalized learning support. Furthermore, they lacked the authentication functionality to individually recognize each user's voice input, making operation difficult when multiple users were using the system. This made it impossible to provide efficient and effective educational support.

[1475] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1476] In this invention, the server includes means for receiving a user's voice input, means for recognizing the voice input and converting it into text, means for analyzing the text and understanding the user's intention, means for recognizing the user's emotions, means for generating a response according to the user's intention and emotion, means for converting the generated response into voice data, and means for outputting the voice data. This makes it possible to provide an appropriate response to the user's voice input that takes emotion into consideration. Furthermore, by further including authentication means for individually recognizing voice inputs, operation when multiple users use the system becomes smoother.

[1477] "Voice input" refers to the act of converting a user's speech into a digital signal and transmitting it to the system, or that digital signal.

[1478] "Text-to-text" refers to the process of converting digital data into text using speech recognition technology to represent spoken input as text.

[1479] "Intent understanding" refers to the process of analyzing input text data and recognizing what the user is looking for.

[1480] "Emotion recognition" refers to the process of analyzing and identifying a user's emotional state from voice or text data.

[1481] "Response generation" refers to the process of creating an appropriate reply based on the user's intent and feelings.

[1482] "Speech synthesis" refers to the process of converting text data into speech data and outputting it as natural-sounding speech.

[1483] "Network storage" refers to online storage space where data is stored on remote servers or cloud services and accessed and shared as needed.

[1484] "Authentication method" refers to a technology or process that identifies an individual user by analyzing the user's voice characteristics.

[1485] The present invention is an educational support system that receives user voice input, analyzes it using a speech recognition and emotion recognition engine, and generates an appropriate response. This system provides personalized educational support by combining a speech recognition engine, a text analysis engine, an emotion recognition engine, a generative AI model, and a speech synthesis engine.

[1486] When a user speaks, the device captures the speech and sends it to a server. The server then uses a speech recognition engine to convert the speech data into text. This speech recognition uses commonly available speech recognition technology (e.g., a cloud-based speech recognition service). The converted text data is then sent to a text analysis engine (e.g., a generative AI model) to analyze the user's intent.

[1487] Furthermore, the voice data is analyzed by an emotion recognition engine to identify the user's emotional state. Emotion recognition is performed using commonly available emotion analysis techniques (e.g., emotion recognition services and engines). Based on the analysis results, an appropriate response is generated in text format by a generative AI model.

[1488] Next, the generated text response is converted into voice data using a voice synthesis engine. This voice synthesis is performed using a commonly used voice synthesis technology (e.g., a voice synthesis service). The converted voice data is sent to the terminal and a voice response is provided to the user.

[1489] The system also has the ability to record interaction logs and emotional data and store them in network storage. Parents and educators can access these logs to check learning progress and emotional changes. For example, this data can be securely stored and shared using cloud storage such as AWS S3.

[1490] The system also uses voice authentication technology to identify individual users. During initial setup, the user's voice is recorded and saved in a profile. The system then matches each voice input with this profile to identify the user. This technology makes it possible to provide individual learning support when multiple users use the same system.

[1491] Specific examples

[1492] For example, if a user says to the device, "What are you going to study today?", the device captures the speech and sends it to the server. The server uses a speech recognition engine to convert the speech into text, "What are you going to study today?" and then uses a generative AI model to analyze the text and understand that the user is asking a question about what they're studying today. If the emotion recognition engine detects nervousness in the user's voice, the generated response might be something like, "Today's study is to solve math problems. It's okay, take your time." The server then converts this text response into audio data using a speech synthesis engine and sends it back to the device. Finally, the device tells the user, "Today's study is to solve math problems. It's okay, take your time."

[1493] Prompt Sentence Examples

[1494] User Question: "What are you studying today?"

[1495] User voice analysis results: "Nervous"

[1496] Generate Response: "Today's lesson is to do some math problems. It's okay, just take your time."

[1497] By combining these specific technologies, the system of the present invention provides advanced and personalized educational support that meets the specific needs of each user.

[1498] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1499] Step 1:

[1500] Capture and transmit audio input

[1501] When a user speaks into the device, the device uses its built-in microphone to capture the voice. Specifically, the user's voice (input) is converted into a digital signal, and the voice data is encoded and sent to the server (output). For example, if a user says, "What are we going to study today?", the voice is captured and digitized. This series of operations is performed in real time.

[1502] Step 2:

[1503] Speech recognition and text conversion

[1504] The server sends the voice data (input) received from the device to a voice recognition engine. Specifically, the voice data is converted into text data (output) using a voice recognition engine (for example, a cloud-based voice recognition service). For example, the voice data "What are you going to study today?" is converted into text as "What are you going to study today?" This converted text data is sent to the next step.

[1505] Step 3:

[1506] Text analysis and intent understanding

[1507] The server analyzes the text data (input) received from the speech recognition engine. Specifically, it uses a generative AI model (e.g., a generative AI model) to analyze the text data and understand the user's intent (output). For example, it analyzes the text "What are we going to study today?" and identifies the intent as "A question about today's learning content." This intent data is sent to the next step.

[1508] Step 4:

[1509] emotion recognition

[1510] The server sends the voice data and text data (input) from the speech recognition engine to the emotion recognition engine. The emotion recognition engine (e.g., emotion analysis service) is used to analyze the user's emotional state (output). For example, it identifies tension, excitement, happiness, etc. from the tone and speed of the voice. This emotion data is sent to the next step.

[1511] Step 5:

[1512] Response Generation

[1513] The server uses a generative AI model to generate an appropriate response based on the intent data and emotion data (input). Specifically, it creates a text response and adjusts it based on the emotion data (output). For example, if it recognizes that the user is nervous, it adds encouraging words such as, "Today's study is to solve your math problems. It's okay, take your time." This adjusted response is sent to the next step.

[1514] Step 6:

[1515] Speech synthesis and output

[1516] The server sends the generated text response (input) to a speech synthesis engine, which converts it into voice data. The speech synthesis engine (e.g., a speech synthesis service) generates the text as natural-sounding voice data (output). For example, voice data such as "Today's study is to solve the math problems. It's okay, take your time" is generated. This voice data is then sent to the device.

[1517] Step 7:

[1518] Recording and storing conversation logs and emotional data

[1519] The server logs user interactions and emotional data (input) and stores them in network storage, such as AWS S3, for later access by parents and educators (output). Specifically, the server compiles user questions, system responses, and emotional analysis results into a single log file.

[1520] Step 8:

[1521] Voice Authentication

[1522] During initial setup, the device records the user's voice and saves it in a profile. It then uses voice authentication technology (e.g., a voice authentication service) to identify the user with each interaction (input and output). For example, new voice input can be matched against the profile to authenticate the user. This allows multiple users to receive personalized learning assistance even when using the same system.

[1523] (Application example 2)

[1524] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1525] Conventional customer support and guidance systems in brick-and-mortar stores could only provide uniform answers to customer questions, making it difficult to respond taking into account the circumstances and emotions of each individual customer. They also lacked a mechanism for accumulating and analyzing customer responses and emotions. As a result, customer satisfaction declined, and improving the quality of store service became an issue.

[1526] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving a user's voice input, means for recognizing the voice input and converting it into text, means for analyzing the text and understanding the user's intention, means for generating a response according to the user's intention, means for recognizing the user's emotion and adjusting the response based on the emotion, means for converting the generated response into voice data, and means for outputting the voice data. This makes it possible to provide a personalized response to a customer's question according to the situation and emotion, thereby improving customer satisfaction and service quality.

[1527] The "means for receiving user voice input" is a device or module for capturing voice uttered by the user and incorporating it into the system.

[1528] The "means for recognizing the voice input and converting it into text" refers to an algorithm or engine for analyzing the received voice data and converting it into text information.

[1529] "Means for analyzing the text and understanding the user's intent" refers to AI models and natural language processing technologies that analyze text data to understand the content of questions and requests uttered by users.

[1530] "Means for generating a response according to the user's intent" refers to a generative AI model or response engine for generating appropriate and effective answers based on the user's intent.

[1531] "Means for recognizing the user's emotions and adjusting the response based on those emotions" refers to an algorithm or engine that analyzes voice data or text data to identify the user's emotions and optimizes the response content based on the results.

[1532] The "means for converting the generated response into voice data" is a voice synthesis engine for converting the response in text format into voice data.

[1533] The "means for outputting audio data" refers to an output device for audibly transmitting audio data to the user.

[1534] "Cloud storage" is a data storage service for storing data in a remote location accessible via the Internet.

[1535] "Voice authentication means" is a technology for individually identifying a speaker from voice data.

[1536] System Configuration

[1537] This invention is a system for providing advanced customer support in a customer support and guidance system, and is composed of the following main components:

[1538] 1. Voice input receiving devices: These include smart glasses and head-mounted displays. These devices capture the user's voice input.

[1539] 2. Speech recognition engine: An engine for converting voice data received on the server side into text data. As a specific example, Google Cloud Speech-to-Text API is used.

[1540] 3. Text analysis module: A generative AI model that analyzes the text data and understands the user's intent. OpenAI GPT-4 is used as a concrete example.

[1541] 4. Emotion Recognition Engine: An algorithm for analyzing user emotions from voice data, allowing for tailored responses.

[1542] 5. Speech synthesis engine: An engine for converting the generated text response into audio data. For example, Google Cloud Text-to-Speech API is used.

[1543] 6. Cloud Storage: Data storage service for storing and sharing generated response and emotion data.

[1544] System Operation

[1545] Capture and transmit audio input

[1546] When a user asks a question into the smart glasses or head-mounted display, the device captures the voice input and sends it to the server. For example, the user might say, "Please tell me the features of this product."

[1547] Speech recognition and text conversion

[1548] The server analyzes the received voice data using the Google Cloud Speech-to-Text API and converts the voice data into text data. This step generates the text "Please tell me the features of this product."

[1549] Text analysis and intent understanding

[1550] The server uses a generative AI model (OpenAI GPT-4) to analyze the text data and understand the intent of the user's question. For example, in this case, it interprets the user as "asking about the product's features."

[1551] emotion recognition

[1552] The voice data is analyzed using the emotion_recognition algorithm to identify the user's emotions, such as "nervous," "excited," or "sad."

[1553] Response Generation

[1554] The server generates an appropriate response based on the analysis results. For example, a text response such as "This product is made of high-quality materials and is particularly durable" is generated. If the user's emotion is identified as "tension," the response will include a phrase such as "Don't worry, take your time looking around."

[1555] Speech synthesis and output

[1556] The generated text response is converted to audio data using the Google Cloud Text-to-Speech API and sent to a smart eyeglasses or head-mounted display, where the user receives the answer via audio.

[1557] Specific prompt examples

[1558] A user asks, "What is this product?" How do you respond?

[1559] Data recording and storage

[1560] The server logs user interaction and emotional data and stores it in cloud storage, which is then analyzed and used to improve the store's services.

[1561] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1562] Step 1:

[1563] A user asks a question to smart glasses or a head-mounted display. The user's voice input is captured by the device and the voice data is sent to the server. The input is the user's voice data, and the output is the voice data sent to the server.

[1564] Step 2:

[1565] The server analyzes the received voice data using a speech recognition engine (specifically, the Google Cloud Speech-to-Text API). The input is the voice data sent to the server, and the output is the converted text data. The server then sends this data to the next step.

[1566] Step 3:

[1567] The server uses a text analysis module to analyze the text data and understand the user's intent using a generative AI model (OpenAI GPT-4). For example, if a user says, "Please tell me the features of this product," the server interprets the intent as "I am asking about the product's features." The input is the text data to be analyzed, and the output is the interpreted user intent.

[1568] Step 4:

[1569] The server analyzes the voice and text data using the emotion_recognition algorithm to identify the user's emotions. The input is voice or text data, and the output is the emotion recognition result, such as "nervous" or "excited."

[1570] Step 5:

[1571] The server uses a generative AI model to generate a response based on the user's understood intent and the identified emotion. For example, it generates a text response such as "This product is made of high-quality materials and is particularly durable." If the emotion is "tension," it adds "Don't worry, take your time looking around." The input is the user's intent and emotion, and the output is the generated text response.

[1572] Step 6:

[1573] The server converts the generated text response into audio data using a speech synthesis engine (Google Cloud Text-to-Speech API). The input is the text response, and the output is audio data. This audio data is sent to the device.

[1574] Step 7:

[1575] The terminal plays the received voice data and conveys the response to the user. The input is the voice data sent from the server, and the output is the voice response conveyed to the user.

[1576] Step 8:

[1577] The server logs user interaction and emotional data and stores them in cloud storage. The input is the user's voice input, response, and emotional data, and the output is the log data stored in cloud storage. The stored data is later analyzed and used to improve the service.

[1578] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1579] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1580] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1581] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1582] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1583] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1584] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1585] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1586] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1587] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1588] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1589] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1590] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1591] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1592] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1593] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1594] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1595] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1596] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1597] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1598] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1599] The following is further disclosed regarding the above embodiment.

[1600] (Claim 1)

[1601] means for receiving a user's voice input;

[1602] means for recognizing and converting said speech input into text;

[1603] means for analyzing the text and understanding the user's intent;

[1604] A means for generating a response according to a user's intention;

[1605] means for converting the generated response into audio data;

[1606] A system including means for outputting audio data.

[1607] (Claim 2)

[1608] means for recording said user's voice input and generated responses;

[1609] means for storing the recorded voice data in cloud storage;

[1610] The system according to claim 1, further comprising means for sharing data stored in the cloud storage.

[1611] (Claim 3)

[1612] 10. The system of claim 1, further comprising voice authentication means for individually recognizing said voice input.

[1613] "Example 1"

[1614] (Claim 1)

[1615] means for receiving a user's voice input;

[1616] means for recognizing and converting said speech input into text;

[1617] means for analyzing the text and understanding the user's intent;

[1618] A means for generating a response according to a user's intention using a generative artificial intelligence model;

[1619] means for converting the generated response into audio data;

[1620] means for outputting audio data;

[1621] means for recording said speech input and generated responses;

[1622] A system including:

[1623] (Claim 2)

[1624] means for storing the recorded voice data in cloud storage;

[1625] A means for sharing data stored in the cloud storage;

[1626] 10. The system of claim 1, comprising:

[1627] (Claim 3)

[1628] 10. The system of claim 1, further comprising voice authentication means for individually recognizing said voice input.

[1629] "Application Example 1"

[1630] (Claim 1)

[1631] means for receiving a user's voice input;

[1632] means for recognizing and converting said speech input into text;

[1633] means for analyzing the text and understanding the user's intent;

[1634] A means for generating a response according to a user's intention;

[1635] means for converting the generated response into audio data;

[1636] means for outputting audio data;

[1637] a means for users to ask questions about educational products at a location that sells educational products;

[1638] The system includes a means for providing product information based on the content of the question.

[1639] (Claim 2)

[1640] means for recording said user's voice input and generated responses;

[1641] means for storing the recorded voice data in cloud storage;

[1642] A means for sharing data stored in the cloud storage;

[1643] 10. The system of claim 1, further comprising means for managing a user's question and answer history for future reference.

[1644] (Claim 3)

[1645] voice authentication means for individually recognizing said voice input;

[1646] 2. The system according to claim 1, further comprising means for comparing with other products and suggesting recommended products based on the content of the question.

[1647] "Example 2: Combining Emotion Engines"

[1648] (Claim 1)

[1649] means for receiving a user's voice input;

[1650] means for recognizing and converting said speech input into text;

[1651] means for analyzing the text and understanding the user's intent;

[1652] means for recognizing a user's emotion;

[1653] A means for generating a response according to the user's intention and emotion;

[1654] means for converting the generated response into audio data;

[1655] A system including means for outputting audio data.

[1656] (Claim 2)

[1657] means for recording said user's voice input and generated responses;

[1658] means for storing the recorded voice data and emotion data in a network storage;

[1659] The system according to claim 1, further comprising a means for sharing data stored in the network storage.

[1660] (Claim 3)

[1661] 10. The system of claim 1, further comprising authentication means for uniquely recognizing said voice input.

[1662] "Application example 2 when combining emotion engines"

[1663] (Claim 1)

[1664] means for receiving a user's voice input;

[1665] means for recognizing and converting said speech input into text;

[1666] means for analyzing the text and understanding the user's intent;

[1667] A means for generating a response according to a user's intention;

[1668] means for recognizing a user's emotion and adjusting a response based on the emotion;

[1669] means for converting the generated response into audio data;

[1670] A system including means for outputting audio data.

[1671] (Claim 2)

[1672] means for recording said user's voice input and generated responses;

[1673] means for storing the recorded voice data and emotion data in cloud storage;

[1674] The system according to claim 1, further comprising means for sharing data stored in the cloud storage.

[1675] (Claim 3)

[1676] 10. The system of claim 1, further comprising voice authentication means for individually recognizing said voice input. [Explanation of symbols]

[1677] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving a user's voice input; means for recognizing and converting said speech input into text; means for analyzing the text and understanding the user's intent; A means for generating a response according to a user's intention; means for converting the generated response into audio data; A system including means for outputting audio data.

2. means for recording said user's voice input and generated responses; means for storing the recorded voice data in cloud storage; The system according to claim 1 , further comprising means for sharing data stored in the cloud storage.

3. 10. The system of claim 1, further comprising voice authentication means for uniquely recognizing said voice input.

Images