System

Abstract

To provide a system.SOLUTION: A system comprises: means for receiving input of problem information and points of uncertainty from a learner; means for generating a prompt requesting an answer and explanation for the input; means for inputting the prompt into a generative AI and generating the answer and explanation for the input; means for analyzing the emotional state of the learner using an emotion analysis engine; and means for generating an answer that provides appropriate support based on the generated answer and explanation for the input and analysis results of the emotional state of the learner.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, there are various systems available, such as systems that allow elementary, junior high, and high school students and the elderly to interactively consult about their concerns, systems that allow students to input problem information and points of uncertainty like a private tutor and provide answers and advice, and systems that provide English conversation education, but there is no system that provides all of these services. [Means for solving the problem]

[0005] The present invention provides a system that includes a means for providing conversational information, a means for interactively offering advice on worries and other issues as part of mental care for elementary, junior high, and high school students and the elderly, and a means for conducting real-time conversations using voice recognition and output. Furthermore, the system also includes a means for providing answers and advice to students taking exams by inputting problem information and points of uncertainty, similar to a private tutor, a means for conducting English conversation and providing English education, and a means for providing scalability that allows options to be added. This makes it possible to provide services that meet the various needs of elementary, junior high, and high school students, the elderly, and students taking exams, all in one system. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment Example 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] One embodiment of the present invention is a system that provides interactive advice to elementary, junior high, and high school students and the elderly about their concerns, etc., as a means of providing conversational information. This system receives input information from the user and generates a response in response. Specifically, it provides advice and solutions to the concerns and problems the user is facing. It is also possible to have real-time conversations using voice recognition and output. This allows the user to talk about their concerns in their own words, and the system then provides appropriate advice.

[0029] "Example 2"

[0030] Furthermore, another embodiment of the present invention is a system that provides answers and advice to test-takers, much like a private tutor, by inputting problem information and unclear points. This system receives problem information and unclear points from the user and generates answers and advice accordingly. Specifically, it provides answers and explanations for problems and unclear points that the user encounters while studying. This allows the user to study at their own pace and receive the necessary support when needed.

[0031] "Example 3"

[0032] A further embodiment of the present invention is a system for conducting English conversation and providing English language education. The system receives English utterances from a user and generates English responses in response to the utterances. Specifically, the user conducts a conversation in English, and the system provides responses in English in response. This allows the user to gain experience using English in a real conversational situation. Furthermore, the system provides feedback on the user's utterances to help improve the user's English ability.

[0033] The processing flow of each embodiment will be described below.

[0034] "Example 1"

[0035] Step 1: The user speaks their concerns or problems into the system.

[0036] Step 2: The system uses voice recognition technology to convert the user's input into text information.

[0037] Step 3: The system analyzes the text information and generates appropriate advice or solutions. Step 4: The system outputs the generated advice or solutions to the user aloud.

[0038] "Example 2"

[0039] Step 1: The test-taker enters the problem information and any questions they have about the problem into the system.

[0040] Step 2: The system analyzes the entered problem information and unclear points and generates appropriate answers and explanations.

[0041] Step 3: Output the generated answers and explanations to the test-taker.

[0042] "Example 3"

[0043] Step 1: The user speaks to the system in English.

[0044] Step 2: The system analyzes the user's utterance and generates an appropriate English response.

[0045] Step 3: Output the generated English response to the user.

[0046] Step 4: The system provides feedback on the user's speech to help improve their English skills.

[0047] Example 1

[0048] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0049] In modern society, elementary, junior high, and high school students, as well as the elderly, have a wide range of worries and problems, and there is a need for appropriate mental care for them. However, with conventional systems, it is difficult to provide consultation using real-time dialogue or voice recognition and output, making it difficult for users to talk about their worries in their own words and receive appropriate advice. In addition, there is a lack of support, such as private tutoring for students taking exams, or English conversation education. A system that can solve these issues is needed.

[0050] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0051] In this invention, the server includes means for acquiring user input information, means for converting voice data into text data using voice recognition technology, means for generating a response based on the text data using a generative AI model, means for converting the generated response into voice data using voice output technology and providing it, means for providing interactive advice on worries and other issues as mental care to elementary, junior high, and high school students and the elderly, and means for conducting real-time conversations. This allows users to talk about their worries in their own words and receive appropriate advice in real time. It can also provide support like a private tutor for students taking exams and English conversation lessons.

[0052] The "means for acquiring user input information" refers to a device or software that provides an interface for the user to input information to the system in the form of voice, text, or the like.

[0053] "Speech recognition technology" is a technology that converts a user's voice into a digital signal, analyzes that signal, and converts it into text data.

[0054] A "generative AI model" is an artificial intelligence algorithm that generates appropriate responses based on input text data.

[0055] "Voice output technology" is a technology that converts text data into voice data and provides it to the user as voice.

[0056] "A means of providing interactive consultation on worries and problems as part of mental care" is a system that receives consultations in an interactive format about the worries and problems that users are facing, and provides appropriate advice and solutions.

[0057] "Means for real-time conversation" refers to technologies and devices that allow users and systems to exchange information instantly and engage in dialogue without delay.

[0058] "Tutor-like support for test takers" is a system that provides appropriate answers and advice to test takers regarding their learning problems and questions.

[0059] "Providing English conversation education" is a system that provides educational support to help users practice English conversation and improve their English skills.

[0060] "Extensibility to add options" refers to a design that is flexible enough to allow new functions and services to be added to the system.

[0061] MODE FOR CARRYING OUT THE INVENTION

[0062] This invention is a system that provides interactive advice to elementary, junior high, and high school students and the elderly about their concerns, etc. It acquires user input information and generates responses to that information. Specifically, it can conduct real-time conversations using voice recognition technology, a generative AI model, and voice output technology.

[0063] Hardware and software used

[0064] The system uses the following hardware and software:

[0065] Device: The device used by the user, such as a smartphone, tablet, or computer

[0066] Speech recognition technology: Speech recognition software such as Google® Cloud Speech-to-Text

[0067] Generative AI models: Generative AI models such as OpenAI's GPT-4

[0068] Speech output technology: Text-to-speech software such as Amazon Polly

[0069] Server: A server for processing data and running generative AI models

[0070] System Operation

[0071] The user speaks to the system using the device. For example, an elementary school student might ask, "I had a fight with a friend. How can I make up?" This voice input is captured through the device's microphone.

[0072] The device uses Google Cloud Speech-to-Text to convert the voice data into text data, which contains the user's concerns and problems.

[0073] The server receives the text data sent from the device and generates a prompt sentence to be input to the generative AI model (GPT-4). The prompt sentence has the following format:

[0074] "A primary school student is asking for advice about a fight he had with a friend. Can you offer some advice on how to make up?"

[0075] The generative AI model generates appropriate advice based on this prompt, such as, "First, it's important to apologize to your friend. Then, it's important to talk about the cause of the argument and come to a mutual understanding."

[0076] The server sends the generated response to the device, which then uses Amazon Polly to convert the response into audio data and provides it to the user, who can receive the system's response through the audio played from the device.

[0077] This system allows users to talk about their concerns in their own words and receive appropriate advice in real time. It can also provide support like a private tutor for students taking exams, or English conversation lessons.

[0078] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0079] Step 1:

[0080] The user speaks into the device's microphone to describe their worries or problems. For example, they might say, "I had a fight with a friend. How can we make up?" The input is captured in the device as voice data.

[0081] Step 2:

[0082] The device uses speech recognition technology to convert the voice data into text data. Specifically, it uses Google Cloud Speech-to-Text to analyze the voice data and generate text data such as "I had a fight with my friend. How can I make up?" The input is voice data, and the output is text data.

[0083] Step 3:

[0084] The device sends the generated text data to the server. The server receives this text data and generates a prompt to be input into the generative AI model. For example, the prompt might read, "An elementary school student has had a fight with a friend and is asking for advice. Please provide some advice on how to make up." The input is text data, and the output is a prompt.

[0085] Step 4:

[0086] The server inputs a prompt into a generative AI model (GPT-4) and generates a response to the user's concerns or problems. The generative AI model generates text data containing appropriate advice based on the prompt. For example, it generates a response such as, "First, it's important to apologize to your friend. After that, it's important to discuss the cause of the fight and come to a mutual understanding." The input is the prompt, and the output is the response text.

[0087] Step 5:

[0088] The server sends the generated response text to the device. The device then uses voice output technology to convert the response text into voice data. Specifically, Amazon Polly is used to convert the text data into voice data, generating voice data such as, "First, it's important to apologize to your friend. Then, it's important to talk about the cause of the argument and come to a mutual understanding." The input is the response text, and the output is voice data.

[0089] Step 6:

[0090] The terminal plays the generated voice data and provides it to the user. The user receives the system's response through the voice played from the terminal. This allows the user to receive appropriate advice in real time. The input is voice data, and the output is a voice response to the user.

[0091] (Application example 1)

[0092] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0093] In modern society, mental health issues faced by elementary, junior high, and high school students, as well as the elderly, are serious, and appropriate support is needed. However, conventional mental care systems have difficulty providing appropriate advice to users in real time regarding their concerns, and few systems have advanced dialogue functions that utilize voice recognition or generative AI models. For this reason, there is a need for a system that allows users to talk about their concerns in their own words and receive appropriate advice in real time.

[0094] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0095] In this invention, the server includes means for providing conversation information, means for interactively offering advice on worries and the like as part of mental care to elementary, junior high, and high school students and the elderly, means for conducting conversations in real time using voice recognition and output, means for providing advice and solutions to users' worries using a generative AI model, means for generating prompt sentences and generating appropriate responses, and means for providing feedback in real time using a voice output function. This enables users to talk about their worries in their own words and receive appropriate advice in response to them in real time.

[0096] "Means for providing conversational information" is a function for collecting information through dialogue with the user and generating an appropriate response.

[0097] "A means of providing interactive consultation to elementary, junior high, and high school students and the elderly regarding their worries and problems as part of mental care" is a function that allows elementary, junior high, and high school students and the elderly to discuss their worries and problems in an interactive format and provide appropriate advice and support.

[0098] "Means for conducting real-time conversations using voice recognition and output" refers to a function that enables real-time conversations by recognizing the user's voice, converting it into text, and outputting the generated response as voice.

[0099] "Means of providing advice and solutions to users' concerns using generative AI models" refers to a function that uses generative AI models to generate appropriate advice and solutions to users' concerns.

[0100] The "means for generating a prompt sentence and generating an appropriate response" is a function for generating a prompt sentence based on a user's input and generating an appropriate response based on the prompt sentence.

[0101] The "means for providing feedback in real time using a voice output function" is a function for outputting the generated response by voice and providing feedback to the user in real time.

[0102] A system for carrying out this invention is configured as follows: The server includes means for providing conversation information, means for interactively offering advice on worries and the like as part of mental care to elementary, junior high, and high school students and the elderly, means for conducting conversations in real time using voice recognition and output, means for providing advice and solutions to users' worries using a generative AI model, means for generating prompt sentences to generate appropriate responses, and means for providing feedback in real time using a voice output function.

[0103] Program processing explanation

[0104] The server uses speech recognition software (e.g., the speech_recognition library) to convert the user's voice into text. What the user says through the microphone is sent to the server as voice data. The server converts this voice data into text data.

[0105] Next, a generative AI model (e.g., OpenAI's API) is used to generate appropriate advice or solutions based on the text data. The generative AI model generates prompts based on the user's input and then generates responses based on those prompts.

[0106] The generated response is converted into voice data using voice output software (e.g., gTTS library) and fed back to the user in real time. The voice output is provided to the user through a speaker.

[0107] Specific examples

[0108] For example, if a user says, "I'm having trouble making friends at school," the server converts this speech to text and generates the following prompt:

[0109] Example prompt sentence:

[0110] User's problem: I'm having trouble making friends at school

[0111] Please provide advice.

[0112] The generative AI model generates appropriate advice based on the prompt, and then converts the advice into voice data using voice output software, allowing the user to receive the advice in real time through a speaker.

[0113] In this way, users can talk about their concerns in their own words and receive appropriate advice in real time.

[0114] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0115] Step 1:

[0116] The user speaks about their concerns through a microphone. The device collects this voice data and sends it to the server. The input is the user's voice data, and the output is the voice data sent to the server.

[0117] Step 2:

[0118] The server uses speech recognition software (e.g., the speech_recognition library) to convert the received voice data into text data. The input is voice data, and the output is text data. Specifically, the speech recognition software analyzes the voice data and generates the corresponding text.

[0119] Step 3:

[0120] The server uses a generative AI model (e.g., OpenAI's API) to generate a prompt based on the text data. The input is the user's text data, and the output is the prompt. Specifically, the generative AI model analyzes the text data and generates an appropriate prompt.

[0121] Step 4:

[0122] The server uses the generated prompt sentence to have the generative AI model generate appropriate advice or solutions. The input is the prompt sentence, and the output is the generated advice or solution. In concrete terms, the generative AI model generates a response based on the prompt sentence.

[0123] Step 5:

[0124] The server converts the generated advice or solution into voice data using voice output software (e.g., gTTS library). The input is the generated text data, and the output is voice data. Specifically, the voice output software converts the text data into voice data.

[0125] Step 6:

[0126] The server transmits the voice data to the terminal, and the terminal provides real-time feedback to the user through the speaker. The input is the voice data, and the output is the voice feedback provided to the user. In concrete terms, the terminal plays the voice data and provides feedback to the user.

[0127] Example 2

[0128] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0129] Conventional learning support systems have had the problem of being unable to provide immediate, appropriate answers or advice to specific problems or unclear points that learners encounter during their studies. Another problem is that learners cannot receive the support they need when they need it, as they progress through their studies at their own pace. Furthermore, there is a lack of means for providing mental care or interactive consultations, and comprehensive support for learners is insufficient.

[0130] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0131] In this invention, the server includes means for providing conversation information, means for interactively offering advice to learners about their worries and other concerns as part of mental care, means for conducting conversations in real time using voice recognition and output, means for receiving problem information and unclear points entered by the learner and generating answers and advice for them, means for generating answers and advice using a generative AI model, and means for providing the generated answers and advice to the learner. This allows learners to progress through their studies at their own pace and receive the support they need when they need it.

[0132] "Conversational information" refers to the content of text and voice communication exchanged between a user and a system.

[0133] "Mental care" refers to interactive support to reduce learners' psychological worries and stress and support their mental health.

[0134] "Speech recognition" is a technology that converts speech into text data.

[0135] "Audio output" is a technology that reproduces text data as audio.

[0136] "Real-time" refers to immediate processing or response without delay.

[0137] "Learner" refers to an individual engaged in an education or learning activity.

[0138] "Problem information" refers to the content of specific problems or issues that a learner encounters during learning.

[0139] "Unclear points" refer to parts that learners do not understand or have questions about.

[0140] An "answer" is a correct answer or solution to a problem submitted by a learner.

[0141] "Advice" is advice or guidance to help learners solve problems.

[0142] A "generative AI model" is an algorithm or system that uses artificial intelligence to generate text or speech.

[0143] A "prompt" is text that contains instructions or questions that are input to a generative AI model.

[0144] The present invention is a system that instantly provides appropriate answers and advice to specific problems and unclear points that a learner encounters during learning. A specific embodiment of this system will be described below.

[0145] Hardware and software used

[0146] 1. Terminal

[0147] These are devices such as computers and smartphones used by learners. These devices are connected to the Internet and are used to input problem information and any questions through a user interface.

[0148] 2. Server

[0149] A computer system for processing data and running a generative AI model. The server receives data sent by users and inputs it into the generative AI model.

[0150] 3. Generative AI Models

[0151] For example, advanced natural language processing models such as OpenAI's GPT-4 generate appropriate answers and advice based on the problem information and questions users provide.

[0152] Data processing and calculation

[0153] 1. User Input

[0154] The user uses the terminal to input information about the problems and points of confusion they encounter while studying, for example, "I don't know how to solve differential equations in mathematics."

[0155] 2. Data transmission

[0156] The device sends the problem information and uncertainties entered by the user to the server, which converts the received data into an appropriate format and inputs it into the generative AI model.

[0157] 3. Use of generative AI models

[0158] The server inputs data to the generative AI model as a prompt sentence. For example, it sends the prompt sentence "Please tell me how to solve a differential equation" to the generative AI model.

[0159] 4. Generating Answers and Advice

[0160] The generative AI model generates appropriate answers and advice based on the input prompt. For example, it might generate an answer like, "To solve a differential equation, first transform the equation into standard form, then solve the characteristic equation."

[0161] 5. User Feedback

[0162] The server sends the generated answers and advice to the terminal. The terminal displays the received answers and advice to the user. For example, the user's screen may display a message such as, "Here's how to solve a differential equation: First, transform the equation into standard form, and then solve the characteristic equation."

[0163] Specific examples

[0164] Example user input

[0165] "I don't know how to graph a linear function."

[0166] Example prompts for generative AI models

[0167] "How do I draw the graph of a linear function?"

[0168] Example of a generated answer

[0169] "To graph a linear function, first determine the shape of the function. For example, in y = 2x + 3, the y-intercept is 3 and the slope is 2. Next, plot the y-intercept and draw a line according to the slope."

[0170] This way, users can learn at their own pace and get the support they need when they need it.

[0171] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0172] Step 1:

[0173] Users use the terminal to input information about problems and questions they encounter while studying.

[0174] Specifically, the user opens a dedicated application or web browser on their device and enters the problem information and any points of uncertainty into the input form. For example, they might enter, "I don't know how to solve a differential equation in mathematics."

[0175] Input: User-entered problem information and questions.

[0176] Output: Problem information and unknown data entered into the terminal.

[0177] Step 2:

[0178] The terminal sends the problem information and unclear points entered by the user to the server.

[0179] Specifically, the terminal transmits the user's input data to the server via the Internet.

[0180] Input: Problem information and question data entered into the terminal.

[0181] Output: Problem information and unknown data sent to the server.

[0182] Step 3:

[0183] The server converts the received problem information and uncertainties into a suitable format for input into the generative AI model.

[0184] Specifically, the server converts the data into a prompt, for example, "Please tell me how to solve a differential equation."

[0185] Input: Problem information and unknown data sent to the server.

[0186] Output: A prompt to be fed into the generative AI model.

[0187] Step 4:

[0188] The server inputs a prompt sentence into the generative AI model.

[0189] Specifically, the server sends a prompt to the generative AI model and waits for the model to generate an answer or advice.

[0190] Input: A prompt to be used as input to the generative AI model.

[0191] Output: The answer or advice returned by the generative AI model.

[0192] Step 5:

[0193] The generative AI model generates answers and advice based on the input prompt.

[0194] Specifically, the generative AI model uses natural language processing technology to generate appropriate answers and advice. For example, it might generate an answer like, "Here's how to solve a differential equation: First, transform the equation into standard form, and then solve the characteristic equation."

[0195] Input: The prompt text fed into the generative AI model.

[0196] Output: The generated answer or advice.

[0197] Step 6:

[0198] The server sends the generated answers and advice to the terminal.

[0199] Specifically, the server sends the answers and advice received from the generative AI model to the user's device.

[0200] Input: Generated answers and advice.

[0201] Output: Answers and advice sent to your device.

[0202] Step 7:

[0203] The device displays the answers and advice received to the user.

[0204] Specifically, the device displays the answer or advice on the user's screen. For example, the device displays a message on the user's screen saying, "Here's how to solve the differential equation: First, transform the equation into standard form, then solve the characteristic equation."

[0205] Input: Answers and advice sent to the device.

[0206] Output: The answer or advice displayed to the user.

[0207] (Application example 2)

[0208] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0209] Conventional learning support systems have difficulty providing immediate answers and advice to problems and unclear points that students encounter while studying. Furthermore, they lack the scalability to accommodate diverse needs, such as mental health care and English education. This makes it difficult for users to progress through their studies at their own pace, and they are unable to receive the support they need when they need it.

[0210] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for providing conversation information, means for interactively offering advice on worries and the like as part of mental care to elementary, junior high, and high school students and the elderly, means for conducting real-time conversations using voice recognition and output, means for inputting problem information and unclear points from the user and generating answers and advice, means for providing answers and advice using a generative AI model, and means including an application to be installed on a smartphone. This allows users to study at their own pace and receive the support they need when they need it.

[0211] "Conversational information" is information provided through dialogue with the user.

[0212] "Mental care" is care to support the user's mental health.

[0213] "Speech recognition" is a technology that converts speech into text data.

[0214] "Audio output" is a technology that reproduces text data as audio.

[0215] "Real-time" refers to processing occurring immediately without delay.

[0216] "Problem information" refers to the specific problems or questions that users encounter while studying.

[0217] "Unclear points" are parts that users do not understand or have questions about.

[0218] An "answer" is the correct answer to a question.

[0219] "Advice" is advice on how to solve problems or proceed with learning.

[0220] A "generative AI model" is a model that uses artificial intelligence to generate answers and advice from data.

[0221] A "smartphone" is a type of mobile phone, a device that can connect to the Internet and use applications.

[0222] An "application" is software that provides specific functions or services.

[0223] To implement the present invention, the following system configuration and program are required.

[0224] System Configuration

[0225] 1. Hardware:

[0226] Smartphone: A device where users can enter problem information or questions.

[0227] Server: A computer system that runs generative AI models and generates answers and advice.

[0228] 2. Software:

[0229] Python: A programming language used to implement the program.

[0230] OpenAI API: An API for using generative AI models.

[0231] Program processing

[0232] 1. User input:

[0233] Users enter any problems or questions they encounter while studying in text format through a smartphone application.

[0234] 2. Data transmission:

[0235] The smartphone sends the entered problem information and any unclear points to the server.

[0236] 3. Run the generative AI model:

[0237] The server uses the OpenAI API to run the generative AI model and generate answers and advice for the input problem information and unknown points.

[0238] 4. Return of answers and advice:

[0239] The server sends the generated answers and advice back to the smartphone and displays them to the user.

[0240] Specific examples

[0241] For example, if a user types, "Please tell me the formula for solving a quadratic equation," the following prompt sentence is sent to the generative AI model:

[0242] Example prompt sentence:

[0243] A candidate is asking the following question: "What is the formula for solving a quadratic equation?"

[0244] Please provide answers and advice.

[0245] The generative AI model generates answers and advice based on this prompt, returning the following results:

[0246] Answers and advice:

[0247] The formula for solving the quadratic equation ax^2 + bx + c = 0 is:

[0248] x = (-b ± √(b^2 - 4ac)) / 2a

[0249] When using this formula, it is important to substitute the values ​​of a, b, and c correctly. Also, note that the number of solutions changes depending on the value of the discriminant (b^2 - 4ac): if the discriminant is positive, you get two real solutions, if it is zero, you get one real solution, and if it is negative, you get two imaginary solutions.

[0250] This way, users can learn at their own pace and get the support they need when they need it.

[0251] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0252] Step 1:

[0253] The user starts the smartphone application and inputs any problems or questions they encounter while studying in text format. The input text is temporarily saved in the application.

[0254] Input: Text data of the problem information or questions entered by the user

[0255] Output: Temporarily saved text data

[0256] Step 2:

[0257] The device (smartphone) sends the entered text data to the server via an internet connection.

[0258] Input: Temporarily saved text data

[0259] Output: Text data sent to the server

[0260] Step 3:

[0261] The server analyzes the received text data and generates a prompt suitable for the generative AI model, which is then formatted to clarify the user's question.

[0262] Input: Text data sent to the server

[0263] Output: Generated prompt statement

[0264] Step 4:

[0265] The server sends the generated prompt to the OpenAI API and executes the generative AI model, which generates answers and advice based on the prompt.

[0266] Input: Generated prompt text

[0267] Output: The answer or advice generated by the generative AI model

[0268] Step 5:

[0269] The server receives the answers and advice returned by the generative AI model, formats them in a user-friendly format, and then sends the formatted answers and advice back to the smartphone.

[0270] Input: Answers and advice generated by a generative AI model

[0271] Output: Formatted answers and advice

[0272] Step 6:

[0273] The terminal (smartphone) displays the answers and advice received from the server to the user, allowing the user to obtain answers and advice for problems and unclear points during study.

[0274] Input: Formatted answers and advice

[0275] Output: The answer or advice displayed to the user

[0276] Example 3

[0277] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0278] Conventional English language education systems and mental health care systems have difficulty providing real-time dialogue and feedback, and have been unable to quickly provide appropriate responses or advice to users' utterances. It has also been difficult to accurately understand the content of users' utterances and generate natural responses that fit the context. This has led to issues such as insufficient improvement in users' English ability and in the effectiveness of mental health care.

[0279] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0280] In this invention, the server includes means for providing conversation information, means for interactively consulting with a user about worries and the like as part of mental care, means for conducting conversations in real time using voice recognition and output, means for receiving the user's speech as voice data and converting it into text data using voice recognition software, means for inputting prompt sentences into a generative AI model and generating an appropriate response, means for converting the generated response into voice data and providing it to the user, and means for generating feedback on the user's speech and providing it to the user. This enables users to improve their English ability and receive the benefits of mental care while engaging in natural conversations in real time.

[0281] "Conversation information" is information about the dialogue that users have through the system.

[0282] "Mental care" means providing interactive consultation and advice to support the mental health of users.

[0283] "Speech recognition" is a technology that converts voice data into text data.

[0284] "Voice output" is a technology that converts text data into voice data and provides it to the user as voice.

[0285] "Real-time" refers to providing immediate responses and feedback to user utterances.

[0286] "Voice data" refers to data that is a digital recording of a user's speech.

[0287] "Text data" is character information converted from voice data using voice recognition technology.

[0288] A "generative AI model" is an artificial intelligence model that understands the content of a user's speech and generates natural responses according to the context.

[0289] A "prompt sentence" is text containing contextual information that is input into a generative AI model.

[0290] "Feedback" refers to providing evaluations and advice on the user's speech.

[0291] The present invention is a system that allows a user to make an utterance in English, generates an appropriate response to the utterance, and provides feedback. A specific embodiment of this system will be described below.

[0292] First, a user accesses the system using a device (e.g., a smartphone or PC). When the user speaks in English, the device captures the speech as voice data. This voice data is then sent to a server via the Internet.

[0293] The server converts the received voice data into text data using speech recognition software (e.g., a speech recognition API). The speech recognition software analyzes the voice data and generates corresponding text data. This text data is used as part of a prompt sentence that is input into the generative AI model.

[0294] Next, the server inputs a prompt to the generative AI model (e.g., a natural language processing model), which looks like this:

[0295] User: What is the weather like today?

[0296] system:

[0297] The generative AI model generates an appropriate response based on this prompt sentence. For example, it might generate a response such as, "The weather today is sunny with a high of 25 degrees Celsius." This response is then converted into voice data by the server using speech synthesis software (e.g., a speech synthesis API).

[0298] The generated voice data is then sent back to the terminal via the Internet, where it is played back and a response in English is provided to the user, allowing the user to have a natural conversation in real time.

[0299] Additionally, the server generates feedback on the user's speech, assessing things like pronunciation accuracy and grammar, and including suggestions to help the user improve their English proficiency. For example, feedback might be generated such as, "Your pronunciation was clear, but you could improve your intonation on the word 'weather'."

[0300] The generated feedback is then sent back to the terminal via the Internet, where it is displayed to the user, who can then receive specific advice on their own speech and improve their English ability.

[0301] This system combines technologies such as speech recognition software, generative AI models, and speech synthesis software to provide users with natural dialogue and feedback in real time. This allows users to gain experience using English in real conversation situations, thereby improving their English ability. The flow of the identification process in Example 3 will be described with reference to Figure 15.

[0302] Step 1:

[0303] The user speaks in English.

[0304] The user asks questions or speaks to the system in English, for example, "What is the weather like today?" The input is the user's voice, and the output is the voice data captured on the device.

[0305] Step 2:

[0306] The terminal transmits the voice data to the server.

[0307] The device (such as a smartphone or PC) captures the user's speech as voice data and sends this voice data to a server via the Internet. The input is the user's voice data, and the output is the voice data sent to the server.

[0308] Step 3:

[0309] The server converts the voice data into text data.

[0310] The server converts the received voice data into text data using voice recognition software (e.g., a voice recognition API). The voice recognition software analyzes the voice data and generates corresponding text data. The input is voice data, and the output is text data.

[0311] Step 4:

[0312] The server inputs a prompt sentence into the generative AI model and generates a response.

[0313] The server inputs a prompt sentence to the generative AI model (e.g., a natural language processing model). The prompt sentence looks like this:

[0314] User: What is the weather like today?

[0315] system:

[0316] The generative AI model generates an appropriate response based on this prompt. For example, it generates a response such as, "The weather today is sunny with a high of 25 degrees Celsius." The input is the text data and the prompt, and the output is the generated response text.

[0317] Step 5:

[0318] The server converts the generated response into audio data.

[0319] The server converts the generated response into audio data using speech synthesis software (e.g., a speech synthesis API), where the input is the generated response text and the output is the audio data.

[0320] Step 6:

[0321] The server sends the audio data to the terminal.

[0322] The server transmits the generated voice data to the terminal via the Internet. The input is the voice data, and the output is the voice data transmitted to the terminal.

[0323] Step 7:

[0324] The terminal plays the response to the user.

[0325] The terminal plays the received voice data and provides a response in English to the user. The input is the voice data and the output is the voice played to the user.

[0326] Step 8:

[0327] The server generates feedback in response to the user's utterance.

[0328] The server generates feedback on the user's speech, evaluating the accuracy of pronunciation and grammar, and providing advice to help improve the user's English skills. The input is the user's speech text, and the output is the feedback text.

[0329] Step 9:

[0330] The server sends the feedback to the device.

[0331] The server transmits the generated feedback to the terminal via the Internet. The input is the feedback text, and the output is the feedback transmitted to the terminal.

[0332] Step 10:

[0333] The device displays the feedback to the user.

[0334] The device displays the received feedback to the user, for example, advice such as "Your pronunciation was clear, but you could improve your intonation on the word 'weather'." The input is the feedback text, and the output is the feedback displayed to the user.

[0335] (Application example 3)

[0336] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0337] Conventional English education systems and mental health care systems make it difficult for users to gain experience using English in real conversational situations, and do not provide real-time feedback, limiting the improvement of English proficiency. Furthermore, virtual stores also lack real-time responses when users ask questions about products or make purchases in English, resulting in an inadequate user experience. To address these issues, a system that provides real-time English responses and feedback is needed.

[0338] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0339] In this invention, the server includes a means for providing conversational information, a means for interactively providing advice to elementary, junior high, and high school students and the elderly about their worries as part of mental care, a means for conducting conversations in real time using voice recognition and output, a means for providing real-time English responses when a user asks a question about a product in English or makes a purchase in a virtual store, and a means for providing feedback on the user's speech to help improve their English ability. This allows users to gain experience using English in actual conversational situations and improve their English ability through real-time feedback. This also improves the user experience in the virtual store.

[0340] "Conversation information" refers to information about what the user says and the system's response.

[0341] "Mental care" refers to supporting mental health by alleviating users' psychological worries and stress through dialogue.

[0342] "Speech recognition" is a technology that collects a user's speech through a microphone and converts it into text data.

[0343] "Voice output" is a technology that converts text data into voice and provides it to the user through a speaker.

[0344] "Real-time" refers to providing instant responses and feedback to user utterances.

[0345] A "virtual store" is a virtual store that exists on the Internet, where users can browse and purchase products online.

[0346] "Feedback" is information that provides evaluation and advice on a user's speech or behavior, encouraging improvement.

[0347] "English proficiency" refers collectively to a user's ability to understand, speak, and write English.

[0348] "User experience" refers to the overall satisfaction and convenience that users feel when using a system or service.

[0349] A system for carrying out this invention has the following configuration: The server includes means for providing conversation information, means for interactively offering advice on worries and the like as part of mental care for elementary, junior high, and high school students and the elderly, means for conducting conversations in real time using voice recognition and output, means for providing real-time English responses when a user asks a question about a product in English or goes through the purchasing process in a virtual store, and means for providing feedback on the user's speech and supporting the improvement of English ability.

[0350] Hardware and software used

[0351] Hardware: smartphone, head-mounted display, microphone, speaker

[0352] Software: OpenAI API (generative AI model), SpeechRecognition (voice recognition), pyttsx3 (voice synthesis)

[0353] Data processing and calculation

[0354] The server first collects the user's speech through a microphone and converts the speech data into text using the SpeechRecognition library. It then uses the OpenAI API to generate an appropriate English response to the user's speech. This response is generated based on the prompt sentence using a generative AI model. The generated response is then converted into audio using the pyttsx3 library and provided to the user through the speaker.

[0355] Additionally, feedback is generated based on the user's speech and the generated response, and is provided as audio or text to help improve the user's English proficiency.

[0356] Specific examples

[0357] For example, if a user asks in English, "Tell me about this product," in a virtual store, the server responds, "Sure, this product is a high-quality, eco-friendly item made from sustainable materials. Would you like to know more details?" It also provides feedback, "Your pronunciation was good, but try to use more complex sentences."

[0358] Prompt Sentence Examples

[0359] User utterance: "Can you tell me about this product?"

[0360] Prompt to generative AI model: "The user asked about a product. Provide a detailed description in English."

[0361] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0362] Step 1:

[0363] The user speaks into the microphone in English.

[0364] Input: User's voice data

[0365] Output: Audio data collected by microphone

[0366] Step 2:

[0367] The device uses speech recognition software (SpeechRecognition) to convert the collected voice data into text data.

[0368] Input: Audio data

[0369] Output: Text data

[0370] Specific operation: The device launches voice recognition software, analyzes the voice data, and generates corresponding text data.

[0371] Step 3:

[0372] The server uses a generative AI model (OpenAI API) to generate appropriate English responses based on the text data.

[0373] Input: User's text data

[0374] Output: Generated English response

[0375] Specific operation: The server sends the text data as a prompt to the generative AI model, which then generates a response.

[0376] Step 4:

[0377] The server converts the generated English response into voice data using speech synthesis software (pyttsx3).

[0378] Input: Generated English responses (text data)

[0379] Output: Audio data

[0380] Specific operation: The server starts the speech synthesis software and converts the text data into speech data.

[0381] Step 5:

[0382] The terminal plays the generated audio data to the user through a speaker.

[0383] Input: Audio data

[0384] Output: The audio the user hears

[0385] Specific operation: The device turns on the speaker and plays audio data.

[0386] Step 6:

[0387] The server generates feedback for the user's speech and provides it in the form of voice or text.

[0388] Input: User utterance text data, generated English responses

[0389] Output: Feedback (audio or text)

[0390] What it does: The server analyzes the user's speech and the generated responses, and generates an evaluation and advice. The generated feedback is converted into audio data using speech synthesis software and played through the speaker or displayed as text.

[0391] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0392] "Example 1"

[0393] One embodiment of the present invention is a conversation service system incorporating an emotion engine. This system receives a user's voice input and converts it into text information using speech recognition technology. The emotion engine then analyzes this text information to understand the user's emotional state. For example, if a user says, "I got a bad score on the test today. I'm feeling depressed," the emotion engine recognizes the emotional state as "depressed." Based on this emotional state, the system then generates appropriate advice or solutions and outputs them in voice.

[0394] "Example 2"

[0395] Another embodiment is a tutoring service system incorporating an emotion engine. In this system, when a test-taker inputs problem information or points of uncertainty during study, the emotion engine analyzes the user's emotional state. For example, if the user says, "I can't solve this math problem and I'm frustrated," the emotion engine recognizes the user's emotional state as "frustrated." Based on this emotional state, the system generates appropriate answers and explanations, and even provides calming advice.

[0396] "Example 3"

[0397] Yet another embodiment is an English conversation education service system incorporating an emotion engine. In this system, when a user speaks in English, the emotion engine analyzes the user's emotional state. For example, if the user says, "I'm having trouble pronouncing this English phrase," the emotion engine recognizes the user's emotional state as "trouble." Based on this emotional state, the system then generates appropriate pronunciation guidance and advice to support the user's English learning.

[0398] The processing flow of each embodiment will be described below.

[0399] "Example 1"

[0400] Step 1: Receive voice input from the user.

[0401] Step 2: Use speech recognition technology to convert the voice input into text information.

[0402] Step 3: The emotion engine analyzes the text information and understands the user's emotional state.

[0403] Step 4: Based on the emotional state, the system generates appropriate advice and solutions.

[0404] Step 5: Output the generated advice or solution aloud.

[0405] "Example 2"

[0406] Step 1: The test-taker enters the problem information and any questions they have about the test into the system.

[0407] Step 2: The emotion engine analyzes the user's emotional state.

[0408] Step 3: Based on the emotional state, the system generates an appropriate answer or explanation.

[0409] Step 4: Provide calming advice if needed.

[0410] "Example 3"

[0411] Step 1: The user inputs an English utterance into the system.

[0412] Step 2: The emotion engine analyzes the user's emotional state.

[0413] Step 3: Based on the emotional state, the system generates appropriate pronunciation guidance and advice. Step 4: The system provides the generated pronunciation guidance and advice to the user to support English learning.

[0414] Example 1

[0415] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0416] In modern society, there is a demand for providing appropriate advice and solutions to the worries and problems faced by elementary, junior high, and high school students, as well as the elderly. However, there are only a limited number of systems that allow these users to consult in real time, and in particular, there are few systems that provide responses that take emotional state into consideration. In addition, systems that combine speech recognition and generative AI models are complex, making them difficult to put into practical use.

[0417] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes means for receiving a user's voice input, means for converting the voice data into text data, means for analyzing the text data to grasp the user's emotional state, means for generating an appropriate response using a generative AI model, means for converting the generated response into voice, and means for outputting the voice response to the user. This enables real-time interactive consultation that takes the user's emotional state into consideration.

[0418] "Means for receiving user voice input" refers to devices or software that capture the voice uttered by the user and input it into the system.

[0419] A "means for converting voice data to text data" is a process or device that uses voice recognition technology to convert a voice signal into corresponding text information.

[0420] "Means for analyzing text data to understand a user's emotional state" refers to algorithms or software for analyzing text information and identifying a user's emotional or psychological state.

[0421] "Means for generating appropriate responses using generative AI models" refers to models or systems that use artificial intelligence technology to generate appropriate responses or advice in response to user input.

[0422] The "means for converting the generated response into speech" refers to speech synthesis technology or a device that converts the text response into a speech signal and plays it aloud to the user.

[0423] The "means for outputting a voice response to a user" refers to a device or system for transmitting the generated voice to a user through an output device such as a speaker.

[0424] This invention is a system that provides real-time interactive advice to elementary, junior high, and high school students and the elderly regarding their worries and problems. This system can receive user voice input, generate appropriate responses, and output them as voice.

[0425] Hardware and software used

[0426] Hardware: microphones, speakers, servers

[0427] Software: speech recognition software (e.g., Google Speech-to-Text), sentiment analysis engines (e.g., IBM Watson® Tone Analyzer), generative AI models (e.g., OpenAI GPT-3®), and speech synthesis software

[0428] System configuration

[0429] 1. A means of receiving user voice input:

[0430] The user talks about their worries and problems through the microphone. For example, the user might say, "I got a bad score on the test today. I'm feeling down."

[0431] 2. How to convert audio data to text data:

[0432] The device uses speech recognition software to convert the user's speech into text, specifically using Google Speech-to-Text to generate the text "I got a bad score on the test today. I'm feeling down."

[0433] 3. Analyzing text data to understand the user's emotional state:

[0434] The server uses an emotion analysis engine to analyze the text information and understand the user's emotional state. Specifically, it uses IBM Watson Tone Analyzer to recognize the emotion "depressed."

[0435] 4. Using generative AI models to generate appropriate responses:

[0436] The server uses a generative AI model to generate appropriate advice and solutions based on the user's emotional state. Specifically, it uses OpenAI GPT-3 to generate a response such as, "Even if you got a bad score on the test, you'll be fine if you try harder next time. Let me know if there's anything I can help you with."

[0437] 5. Means for converting the generated response into speech:

[0438] The server then uses speech synthesis software to convert the generated text response into speech, specifically using speech synthesis technology to generate a voice message that says, "Even if you got a bad score on the test, you can do better next time. Let me know if there's anything I can help you with."

[0439] 6. A means of outputting a voice response to the user:

[0440] The device receives the voice data from the server and outputs it to the user through the speaker. Specifically, the speaker plays a message saying, "Even if you got a bad score on the test, you'll be fine if you try harder next time. Let me know if there's anything I can help you with."

[0441] Specific examples

[0442] User Type: "I had a fight with a friend today. How can I make up?"

[0443] System response: "I see you had a fight with your friend. The first thing you need to do is stay calm and try to understand the other person's feelings. Choose the right time to apologize and honestly express your feelings."

[0444] Prompt Sentence Examples

[0445] "I had a fight with a friend. How can I make up?"

[0446] "I got a bad score on the test today. I'm feeling down."

[0447] "I feel like I'm being bullied at school lately. What should I do?"

[0448] In this way, the system provides appropriate advice to the user's concerns and problems.

[0449] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0450] Step 1:

[0451] The user inputs voice. The user talks about their worries and problems through the microphone. A specific example of this is when the user says, "I got a bad score on the test today. I'm feeling down." The input is the user's voice data, and the output is the audio signal captured by the microphone.

[0452] Step 2:

[0453] The device sends voice data to the server. The device captures the user's voice data and sends it to the server. In concrete terms, the device uploads the voice data to the server in real time. The input is the captured voice signal, and the output is the voice data sent to the server.

[0454] Step 3:

[0455] The server performs speech recognition. Using speech recognition software, the server converts the speech data into text data. Specifically, the server uses Google Speech-to-Text to generate the text "I got a bad score on the test today. I'm feeling depressed." The input is speech data, and the output is text data.

[0456] Step 4:

[0457] The server performs emotion analysis. Using an emotion analysis engine, the server analyzes text data and understands the user's emotional state. Specifically, the server uses IBM Watson Tone Analyzer to recognize the emotion "depressed." The input is text data, and the output is emotional state data.

[0458] Step 5:

[0459] The server generates a response. Using a generative AI model, the server generates appropriate advice and solutions based on the user's emotional state. Specifically, the server uses OpenAI GPT-3 to generate a response such as, "Even if you got a bad score on the test, you'll be fine if you try harder next time. Let me know if there's anything I can help you with." The input is emotional state data and text data, and the output is the generated response text data.

[0460] Step 6:

[0461] The server converts the response into speech. The server then uses speech synthesis software to convert the generated text response into speech. Specifically, the server uses speech synthesis technology to generate speech data that says, "Even if you got a bad score on the test, you'll be fine if you try harder next time. Let me know if there's anything I can help you with." The input is the text data of the generated response, and the output is speech data.

[0462] Step 7:

[0463] The device outputs a voice response to the user. The device outputs the voice data received from the server to the user through the speaker. Specifically, the device plays a voice message from the speaker saying, "Even if you got a bad score on the test, you'll be fine if you try harder next time. Let me know if there's anything I can help you with." The input is voice data, and the output is the voice that the user can hear.

[0464] (Application example 1)

[0465] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0466] In modern society, mental health issues faced by elementary, junior high, and high school students, as well as the elderly, are serious, and it is difficult to receive appropriate support, especially when they feel anxious or scared. Security services are also required to grasp the user's psychological state in real time and respond appropriately. Furthermore, there is a lack of technology for smartphone applications that can analyze the user's emotional state and provide appropriate advice.

[0467] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes a means for providing conversation information, a means for interactively offering advice on worries and the like as mental care to elementary, junior high, and high school students and the elderly, a means for conducting conversations in real time using voice recognition and output, a means for analyzing the user's emotional state using an emotion engine, and a means for generating advice for the user's worries using a generative AI model. This makes it possible to offer real-time advice and appropriate advice and solutions when the user feels anxious or scared.

[0468] "Means for providing conversational information" refers to technologies and devices for providing information through dialogue with users.

[0469] "Means of providing interactive consultation to elementary, junior high, and high school students and the elderly as part of mental care" refers to technologies and devices that provide dialogue-based consultation to elementary, junior high, and high school students and the elderly about the worries and problems they have, and provide mental care.

[0470] "Means for real-time conversations using speech recognition and output" refers to technologies and devices that recognize speech, convert it into text, and output the generated text as speech, allowing for real-time conversations.

[0471] "Means for analyzing a user's emotional state using an emotion engine" refers to technology or devices that analyze emotions from a user's statements and actions and grasp that emotional state.

[0472] "Means for generating advice for a user's concerns using a generative AI model" refers to technology or devices for using a generative AI model to generate appropriate advice or solutions for a user's concerns or problems.

[0473] "In the context of security services, a means of providing real-time consultation and appropriate advice and solutions when users feel anxious or scared" refers to technology or devices that, as part of a security service, provide real-time consultation and appropriate advice and solutions when users feel anxious or scared.

[0474] "A means for providing advice to users on their concerns using voice recognition, emotion analysis, and generative AI models as an application installed on a smartphone" refers to technology or devices for providing advice to users on their concerns using voice recognition, emotion analysis, and generative AI models as an application installed on a smartphone.

[0475] The following system configuration is used as an embodiment of the present invention: The system is composed of three main elements: a server, a terminal, and a user.

[0476] server

[0477] The server includes the following means:

[0478] 1. Means of providing conversational information: Technologies and devices for providing information through dialogue with users.

[0479] 2. Mental care tools: These are techniques and devices that provide mental care by providing advice through dialogue on the worries and problems faced by elementary, junior high, and high school students and the elderly.

[0480] 3. Speech recognition and output means: Technology and devices that recognize speech, convert it into text, and output the generated text as speech, allowing for real-time conversations.

[0481] 4. Emotion engine: A technology or device that analyzes emotions from a user's words and actions and grasps their emotional state.

[0482] 5. Generative AI model: A technology or device that generates appropriate advice or solutions to users' concerns or problems.

[0483] Terminal

[0484] The device used is primarily a smartphone, with the following applications installed:

[0485] 1. Speech recognition application: Uses a speech recognition library (e.g. speech_recognition) to convert the user's speech into text.

[0486] 2. Sentiment analysis application: Analyze the sentiment from the user's text using a sentiment analysis model (e.g., the transformers library).

[0487] 3. Generative AI model application: Using a generative AI model (e.g., GPT-3), advice is generated to address the user's concerns.

[0488] User

[0489] The user interacts with the system through their smartphone. Specifically, the system is used in the following steps:

[0490] 1. The user talks about their worries or problems into their smartphone.

[0491] 2. The smartphone's voice recognition application converts the speech into text.

[0492] 3. The text is sent to the server and the emotion engine performs emotion analysis.

[0493] 4. Based on the sentiment analysis results, the generative AI model generates appropriate advice.

[0494] 5. The generated advice is provided to the user through a voice output means.

[0495] Specific examples

[0496] For example, if a user says, "I've been afraid to walk on the street at night lately," the emotion analysis model will recognize "fear," and the generative AI model will generate a prompt that "offers advice to the fearful user about safety measures and ways to stay calm when walking on the street at night."

[0497] Prompt Sentence Examples

[0498] The user is feeling scared. Please provide advice for the following problem: Lately, I've been afraid to walk on the street at night.

[0499] In this way, it is possible to provide appropriate advice in real time according to the user's emotional state.

[0500] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0501] Step 1:

[0502] The user speaks about their worries and problems into the smartphone. The input is the user's voice, and the output is the voice data captured through the smartphone's microphone.

[0503] Step 2:

[0504] The device's speech recognition application converts the voice data into text. Specifically, it uses a speech recognition library (e.g., speech_recognition) to analyze the voice data and generate text data. The input is voice data, and the output is text data.

[0505] Step 3:

[0506] The terminal sends the generated text data to the server. The input is the text data, and the output is the text data sent to the server.

[0507] Step 4:

[0508] The server's emotion engine analyzes the text data to understand the user's emotional state. Specifically, it analyzes the text data using an emotion analysis model (e.g., the transformers library) to identify the emotional state. The input is text data, and the output is the emotional state (e.g., fear, sadness, joy, etc.).

[0509] Step 5:

[0510] The server's generative AI model generates appropriate advice based on the emotional state and text data. Specifically, a generative AI model (e.g., GPT-3) is used to generate prompt sentences according to the emotional state, and advice is generated based on those prompt sentences. The input is the emotional state and text data, and the output is the generated advice.

[0511] Step 6:

[0512] The server sends the generated advice to the terminal. The input is the generated advice, and the output is the advice sent to the terminal.

[0513] Step 7:

[0514] The audio output means of the terminal provides the advice to the user as audio. Specifically, the text data is converted into audio data and played back through a speaker. The input is the advice as text data, and the output is the advice as audio data.

[0515] In this way, it is possible to provide appropriate advice in real time according to the user's emotional state.

[0516] Example 2

[0517] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0518] Conventional learning support systems struggle to provide appropriate answers and advice to specific problems and unclear points that learners face. Furthermore, they lack support that takes into account the learner's emotional state, which often leads to learners feeling stressed. Furthermore, limited means of providing real-time dialogue and mental care make it difficult for learners to progress effectively.

[0519] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0520] In this invention, the server includes means for providing conversation information, means for interactively offering advice to learners about their worries and other concerns as part of mental care, means for conducting conversations in real time using voice recognition and output, means for receiving problem information and unclear points from learners and generating answers and advice for them, means for analyzing the learner's emotional state using an emotion analysis engine and providing appropriate answers and advice based on the results, and means for generating answers and explanations using a generative AI model. This allows learners to receive appropriate answers and advice for specific problems and unclear points, and by receiving support tailored to their emotional state, they can effectively progress with their studies while reducing stress.

[0521] "Conversational information" refers to the content of text or voice communication exchanged between a user and a system.

[0522] "Mental care" refers to interactive support to support learners' mental health and reduce stress and distress.

[0523] "Speech recognition and output" refers to the technology that converts the user's speech into text and the technology that converts text from the system into speech and outputs it.

[0524] "Learner" refers to those receiving education, especially those who use this system to further their studies.

[0525] "Problem information" refers to the content of specific problems or tasks that learners face during their studies.

[0526] "Unclear points" are parts that learners do not understand or have questions about.

[0527] An "answer" is a correct answer or solution to a problem entered by a learner.

[0528] "Advice" is advice or guidance to help learners solve problems.

[0529] An "emotion analysis engine" is a technology that analyzes the emotional state of a user based on their input and outputs the results.

[0530] A "generative AI model" is an artificial intelligence model that performs natural language processing based on user input and generates appropriate answers and explanations.

[0531] This invention is a system that allows learners to input problem information and points of uncertainty, much like a private tutor, and provides answers and advice. Furthermore, by incorporating an emotion analysis engine, it is possible to analyze the learner's emotional state and provide appropriate support based on the results.

[0532] Hardware and software used

[0533] Hardware: Servers, devices (PCs, tablets, smartphones)

[0534] Software: Generative AI models (e.g., GPT-4), sentiment analysis engines (e.g., IBM Watson Tone Analyzer)

[0535] Program processing flow

[0536] User Input

[0537] The user uses the terminal to input any problems or uncertainties they encounter while studying into the system. For example, the user might input, "I can't solve this math problem. What should I do?"

[0538] Receiving server input

[0539] The server receives input from the user, parses the input data, and stores it.

[0540] Sending prompts to the server's generated AI model

[0541] The server sends the received input data to the generative AI model, and then sends a prompt to the generative AI model saying, "Please generate answers and explanations for the questions and unclear points entered by the user."

[0542] Generative AI model answer generation

[0543] The generative AI model generates an answer or explanation based on the prompt, for example, "This problem uses quadratic equation solving techniques. First, transform the equation into standard form."

[0544] Sending input to the server's sentiment analysis engine (in another embodiment)

[0545] If the user's input contains an emotion, the server sends the input to the emotion analysis engine, for example, "I'm frustrated because I can't solve this math problem."

[0546] Receiving the analysis results of the emotion analysis engine

[0547] The emotion analysis engine analyzes the user's emotional state and returns the results to the server. For example, it recognizes the emotional state of "irritated."

[0548] Server final answer generation

[0549] The server generates a final answer that provides appropriate support to the user based on the answers and explanations from the generative AI model and the results from the sentiment analysis engine. For example, it generates a final answer such as, "This problem uses the quadratic equation solving method. First, transform the equation into standard form. Please stay calm and try it slowly."

[0550] Server response

[0551] The server sends the generated final answer to the user's terminal.

[0552] Receiving user answers

[0553] The user receives the answer sent from the server on their device and displays it. For example, the user's device receives a response from the server and displays the answer on the screen: "This problem uses the quadratic equation solution method. First, transform the equation into standard form. Please stay calm and try this slowly."

[0554] Examples and prompts

[0555] Example 1: Answers to math problems

[0556] User input: "I can't solve this math problem. What should I do?"

[0557] The generative AI model's answer: "This problem uses quadratic equation solving techniques. First, transform the equation into standard form."

[0558] Example 2: Support including sentiment analysis

[0559] User input: "I'm frustrated because I can't solve this math problem."

[0560] Sentiment analysis engine result: "I'm frustrated"

[0561] The generative AI model's answer: "This problem uses quadratic equation solving techniques. First, transform the equation into standard form. Please stay calm and take your time."

[0562] Prompt Sentence Examples

[0563] "Generate answers and explanations for problems and uncertainties entered by the user. Also, analyze the user's emotional state and provide appropriate advice."

[0564] This system allows users to receive appropriate answers and advice for specific problems and points of uncertainty, and by receiving support tailored to their emotional state, they can study effectively while reducing stress.

[0565] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0566] Step 1:

[0567] The user uses the device to input any problems or uncertainties they encounter while studying into the system. Specifically, the user uses the device's keyboard or touchscreen to type, "I can't solve this math problem. What should I do?" The input data is sent to the server in text format.

[0568] Step 2:

[0569] The server receives input from the user. Specifically, the server receives an HTTP request, analyzes the input data, and saves it. The input data is text data such as "I can't solve this math problem. What should I do?"

[0570] Step 3:

[0571] The server sends the received input data to the generative AI model. Specifically, the server sends the generative AI model a prompt message saying, "Please generate answers and explanations for the questions and unclear points entered by the user." The input data is the user's problem information, and the output data is the answer provided by the generative AI model.

[0572] Step 4:

[0573] The generative AI model generates an answer and explanation based on the prompt. Specifically, the generative AI model uses an internal natural language processing algorithm to generate the answer, "This problem uses a quadratic equation solution. First, transform the equation into standard form." The input data is the prompt, and the output data is the answer text.

[0574] Step 5:

[0575] If the user's input contains emotion, the server sends the input to the emotion analysis engine. Specifically, the server sends the input "I'm frustrated because I can't solve this math problem." to the emotion analysis engine. The input data is text containing the user's emotion, and the output data is the emotion analysis result.

[0576] Step 6:

[0577] The emotion analysis engine analyzes the user's emotional state and returns the results to the server. Specifically, the emotion analysis engine recognizes the emotional state of "irritated" and returns the results to the server. The input data is text containing emotions, and the output data is the emotion analysis results.

[0578] Step 7:

[0579] The server generates a final answer that provides appropriate support to the user based on the answer and explanation from the generative AI model and the results from the sentiment analysis engine. Specifically, the server generates the final answer: "This problem uses the quadratic equation solution method. First, transform the equation into standard form. Please stay calm and try this slowly." The input data are the answer from the generative AI model and the sentiment analysis results, and the output data is the final answer.

[0580] Step 8:

[0581] The server sends the generated final answer to the user's device. Specifically, the server generates an HTTP response and sends it to the user's device. The input data is the final answer, and the output data is the HTTP response.

[0582] Step 9:

[0583] The user receives the answer sent from the server on their device and displays it. Specifically, the user's device receives the response from the server and displays the answer on the screen: "This problem uses the quadratic equation solution method. First, transform the equation into standard form. Please stay calm and try this slowly." The input data is the HTTP response, and the output data is the displayed answer text.

[0584] (Application example 2)

[0585] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0586] Conventional systems have had difficulty providing appropriate answers and advice in real time to problems and unclear points that workers encounter while working. Furthermore, they lacked support that took into account the emotional state of workers, and were unable to provide appropriate responses to reduce stress and anxiety. This led to concerns about reduced work efficiency and the occurrence of mistakes.

[0587] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0588] In this invention, the server includes means for providing conversation information, means for interactively offering advice on worries and the like as part of mental care to elementary, junior high, and high school students and the elderly, means for conducting conversations in real time using voice recognition and output, means for providing answers and advice to problems and unclear points that workers encounter, and means for incorporating an emotion engine to analyze the user's emotional state and provide appropriate support based on that. This makes it possible to provide appropriate answers and advice in real time to problems and unclear points that workers encounter while working, and to provide support that takes into account their emotional state.

[0589] "Conversational information" is information obtained through dialogue with the user.

[0590] "Mental care" is care to support the user's mental health.

[0591] "Speech recognition" is a technology that converts speech into text data.

[0592] "Real-time" refers to immediate processing or response without delay.

[0593] "Workers" refers to personnel who perform work in factories or on-site.

[0594] "Problem information" is information about the challenges or questions users are facing.

[0595] "Unclear points" are points that users don't understand or have questions about.

[0596] An "answer" is a response to a problem or question.

[0597] "Advice" is advice or suggestions for solving problems or making improvements.

[0598] An "emotion engine" is a technology for analyzing a user's emotional state.

[0599] "Emotional state" refers to the user's current psychological state.

[0600] "Support" refers to solving users' problems and providing emotional support.

[0601] The system for carrying out this invention consists of three main elements: a server, a terminal, and a user. The server includes a means for providing conversational information, a means for providing interactive advice on worries and other issues as part of mental care for elementary, junior high, and high school students and the elderly, a means for conducting real-time conversations using voice recognition and output, a means for providing answers and advice to problems and unclear points that workers encounter, and a means for incorporating an emotion engine to analyze the user's emotional state and providing appropriate support based on that analysis.

[0602] The server uses OpenAI's API and emotion recognition engine (EmotionRecognizer) to generate answers and advice based on the problem information and emotional state entered by the user. Specifically, when the user enters problem information, the emotion recognition engine analyzes the user's emotional state from that information. Then, based on the problem information and emotional state, the server uses the OpenAI API to generate appropriate answers and advice.

[0603] The terminal provides an interface for users to input problem information. Input devices include a keyboard and a voice input device. The information entered by the user is sent in real time to the server, where it is analyzed and a response is made.

[0604] Users input problems or questions they encounter while working into the device. For example, if they input "I'm frustrated because I'm having trouble setting up this machine," the emotion recognition engine recognizes the emotional state of "frustrated." Next, it uses the OpenAI API to generate appropriate answers and advice based on the problem information ("I'm having trouble setting up this machine") and the emotional state ("frustrated").

[0605] For example, if a user types "I'm having trouble setting up this machine and I'm frustrated," the following prompt will be sent to the generative AI model:

[0606] Problem: I'm having trouble setting up this machine and I'm frustrated.

[0607] Emotional state: Irritated

[0608] Please provide appropriate answers and advice.

[0609] Based on this prompt, the OpenAI API generates answers and advice such as, "Please recheck the machine setup procedures and follow them carefully. We also recommend taking a short break to refresh yourself."

[0610] In this way, users can receive appropriate answers and advice in real time, and can also receive support that takes their emotional state into account.

[0611] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0612] Step 1:

[0613] The user inputs the problem information and emotional state into the terminal. For example, the user might input, "I'm frustrated because the settings on this machine aren't working properly." A keyboard or voice input device is used as the input device. The input information is sent from the terminal to the server.

[0614] Step 2:

[0615] The server passes the received problem information to the emotion recognition engine. The emotion recognition engine (EmotionRecognizer) analyzes the user's emotional state from the input text. For example, it recognizes the emotional state "irritated." The input is the problem information, and the output is the emotional state.

[0616] Step 3:

[0617] The server generates a prompt based on the problem information and the user's emotional state. The generated prompt has the following format:

[0618] Problem: I'm having trouble setting up this machine and I'm frustrated.

[0619] Emotional state: Irritated

[0620] Please provide appropriate answers and advice.

[0621] The input is problem information and emotional state, and the output is a prompt sentence.

[0622] Step 4:

[0623] The server sends the generated prompt to the OpenAI API, which then generates an appropriate answer or advice based on the prompt. For example, the OpenAI API might generate an answer or advice such as, "Please recheck the machine's setup procedures and follow them carefully. We also recommend taking a short break to refresh yourself." The input is the prompt, and the output is the answer or advice.

[0624] Step 5:

[0625] The server sends the answers and advice received from the OpenAI API to the device. The device displays the received answers and advice to the user. The user can check the displayed answers and advice and use them to solve the problem. The input is the answers and advice, and the output is what is displayed to the user.

[0626] Example 3

[0627] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0628] Conventional English language education systems and mental health care systems have had difficulty fully analyzing the user's speech content and emotional state, and providing appropriate responses and feedback. Furthermore, there was a lack of technology for real-time conversation and emotional analysis, limiting the effectiveness of learning and mental health care for users. Furthermore, there was a need for flexible responses tailored to the individual needs of users.

[0629] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0630] In this invention, the server includes means for providing conversation information, means for interactively consulting with users about their worries as part of mental care, means for conducting conversations in real time using voice recognition and output, means for analyzing the user's speech and using a generative AI model to generate an appropriate response, and means for analyzing the user's emotional state and using an emotion analysis engine to provide feedback based on the analysis. This makes it possible to analyze the content of the user's speech and emotional state in real time and provide appropriate responses and feedback.

[0631] "Conversation information" refers to information about what the user says and the responses to that speech.

[0632] "Mental care" means providing interactive consultation and advice to support the mental health of users.

[0633] "Speech recognition" is a technology that captures a user's speech as voice data and converts it into text data.

[0634] A "generative AI model" is an artificial intelligence model that analyzes the content of a user's speech and generates an appropriate response.

[0635] An "emotion analysis engine" is a technology for analyzing a user's emotional state from their speech content and tone of voice.

[0636] "Feedback" is advice or guidance provided based on the user's speech or emotional state.

[0637] "Real-time" means that user utterances and responses are processed immediately without delay.

[0638] "Expandability" refers to the flexibility to add new functions and options to a system.

[0639] This invention is a system for supporting users' English education and mental care. This system operates among three parties: a server, a terminal, and a user.

[0640] The server is a computer equipped with a high-performance CPU and GPU, and operates using a generative AI model (e.g., GPT-4) and a sentiment analysis engine (e.g., IBM Watson). The terminal is a device used by the user, equipped with a microphone and an internet connection.

[0641] When a user speaks in English into the device, the device captures the voice data and sends it to the server. The server uses a generative AI model to convert the voice data into text and analyze the content of the speech. The generative AI model then generates an appropriate English response based on the analysis results. It also uses an emotion analysis engine to analyze the user's emotional state and generates feedback based on that. The server sends the generated response and feedback to the device, which then provides it to the user.

[0642] As a concrete example, consider the case where a user utters, "I find it difficult to pronounce this phrase." In this case, the device sends the voice data to the server. The server uses a generative AI model to generate and analyze the text "I find it difficult to pronounce this phrase." The generative AI model generates a response saying, "Try breaking the phrase into smaller parts and practice each part separately." The sentiment analysis engine analyzes the emotional state of "I'm having trouble" and generates feedback saying, "It seems you are having trouble. Don't worry, with practice, it will get easier." The server sends this response and feedback to the device, which then provides it to the user.

[0643] An example of a prompt is as follows:

[0644] User utterance: "I find it difficult to pronounce this phrase"

[0645] Prompt for generative AI models: "Provide a helpful response to the user's difficulty in pronouncing the phrase."

[0646] Prompt to sentiment analysis engine: "Analyze the user's emotional state based on their speech and provide appropriate feedback."

[0647] This system allows users to gain experience using English in real conversation situations, and by receiving emotion-based feedback, they can improve their English ability. Furthermore, mental health consultations and advice are provided in real time, which can support the user's mental health. The flow of the identification process in Example 3 will be described with reference to FIG. 21.

[0648] Step 1:

[0649] The user speaks in English to the terminal. For example, they say, "I find it difficult to pronounce this phrase." The terminal captures this speech with a microphone. The input is the user's speech data, and the output is the captured speech data.

[0650] Step 2:

[0651] The device sends the captured audio data to the server. Specifically, the device application uploads the audio data to the server via the Internet. The input is the captured audio data, and the output is the audio data sent to the server.

[0652] Step 3:

[0653] The server converts the received voice data into text using a generative AI model. For example, it uses speech recognition technology to generate the text "I find it difficult to pronounce this phrase." The input is voice data, and the output is the converted text data.

[0654] Step 4:

[0655] The server uses a generative AI model to analyze text data and understand the user's intent and content. For example, it analyzes the content "The pronunciation is difficult." The input is text data, and the output is the analysis result.

[0656] Step 5:

[0657] Based on the analysis results, the server uses a generative AI model to generate an appropriate English response, for example, "Try breaking the phrase into smaller parts and practice each part separately." The input is the analysis results, and the output is the generated response.

[0658] Step 6:

[0659] The server uses an emotion analysis engine to analyze the user's emotional state from their speech and tone of voice. For example, it can recognize the emotion "I'm in trouble." The input is text data and tone of voice data, and the output is the emotion analysis result.

[0660] Step 7:

[0661] The server generates feedback for the user based on the sentiment analysis results. For example, it generates feedback such as "It seems you are having trouble. Don't worry, with practice, it will get easier." The input is the sentiment analysis results, and the output is the generated feedback.

[0662] Step 8:

[0663] The server sends the generated response and feedback to the terminal. Specifically, the server's application sends the response and feedback to the terminal over the Internet. The input is the generated response and feedback, and the output is the response and feedback sent to the terminal.

[0664] Step 9:

[0665] The terminal provides the received responses and feedback to the user. Specifically, an application on the terminal displays the feedback as a text message or plays it as an audio message. The input is the response and feedback sent from the server, and the output is the response and feedback provided to the user.

[0666] (Application example 3)

[0667] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0668] Conventional English conversation training systems do not provide feedback that takes into account the user's emotional state, limiting the effectiveness of learning. Furthermore, in training English customer service in brick-and-mortar stores, real-time feedback and emotion analysis are not provided, making it difficult to improve staff's English skills.

[0669] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes means for providing conversation information, means for interactively offering advice to users about their worries and other issues as part of mental care, means for conducting real-time conversations using voice recognition and output, means for performing emotion analysis and providing feedback based on the user's emotional state, means for generating responses to the user's utterances using a generative AI model, and means including an application to be installed on a smart device. This enables real-time feedback that takes into account the user's emotional state, making it possible to effectively improve English ability and provide English customer service training in physical stores.

[0670] "Conversation information" refers to information about what the user says and the system responds to.

[0671] "Mental care" means providing interactive consultation and advice to support the mental health of users.

[0672] "Speech recognition" is a technology that analyzes a user's speech as digital data and converts it into text data.

[0673] "Real-time" refers to providing instant responses and feedback to user utterances.

[0674] "Emotion analysis" is a technology that analyzes a user's emotional state from the content of their speech and the tone of their voice.

[0675] "Feedback" refers to evaluations and advice provided in response to a user's speech or actions.

[0676] A "generative AI model" is an algorithm or system that uses artificial intelligence to generate responses to user utterances.

[0677] A "smart device" is a portable electronic device with internet connectivity and advanced computing capabilities.

[0678] An "application" is software that is installed on a smart device to provide specific functions or services.

[0679] A system for carrying out this invention has the following configuration: a server includes means for providing conversation information, means for interactively offering advice to users about their worries as part of mental care, means for conducting real-time conversations using voice recognition and output, means for performing emotion analysis and providing feedback based on the user's emotional state, means for generating responses to user utterances using a generative AI model, and means including an application to be installed on a smart device.

[0680] Hardware and Software Configuration

[0681] Hardware: Smart devices (e.g., smart glasses, smartphones)

[0682] software:

[0683] speech_recognition library: software for converting speech to text

[0684] The transformers library: software for generating responses using generative AI models (e.g., DialoGPT).

[0685] emotion_recognition library: Software for analyzing a user's emotional state

[0686] Data processing and calculation

[0687] The server receives voice input from the smart device and converts the speech to text using a speech recognition library. It then analyzes the user's emotional state using an emotion analysis library. It uses a generative AI model to generate an appropriate response to the user's utterance. It also generates and provides feedback to the user based on the emotional state.

[0688] Specific examples

[0689] Consider a scenario in which a store staff member wears smart glasses and serves customers in English. When a customer asks, "Can you help me find a specific product?", the staff member responds, "Sure, what are you looking for?" The system then analyzes the staff member's speech and provides appropriate feedback.

[0690] Example prompt sentence:

[0691] User: "Can you help me find a specific product?"

[0692] System: "Sure, what are you looking for?"

[0693] Feedback: "If you have difficulty pronouncing something, try speaking more slowly."

[0694] In this way, real-time feedback that takes into account the user's emotional state becomes possible, enabling effective improvement of English ability and English customer service training in physical stores.

[0695] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0696] Step 1:

[0697] A smart device (terminal) receives voice input. When a user speaks into the smart device, the device's microphone captures the voice. The input is the user's voice, and the output is digital voice data.

[0698] Step 2:

[0699] The device uses a speech recognition library to convert digital audio data into text data. The speech recognition library (e.g., speech_recognition) analyzes the audio data and generates corresponding text. The input is digital audio data, and the output is text data.

[0700] Step 3:

[0701] The server uses an emotion analysis library to analyze the user's emotional state from the text data. The emotion analysis library (e.g., emotion_recognition) analyzes the text data and identifies the user's emotional state (e.g., confusion, joy, etc.). The input is text data, and the output is emotional state data.

[0702] Step 4:

[0703] The server uses a generative AI model to generate a response to the user's utterance. The generative AI model (e.g., DialoGPT from the transformers library) takes text data as input and generates an appropriate response text. The input is the user's utterance text, and the output is the response text.

[0704] Step 5:

[0705] The server generates feedback for the user based on the emotional state data. It analyzes the emotional state data and generates feedback such as "If the pronunciation is difficult, please try speaking more slowly" if the user is confused. The input is the emotional state data and the output is the feedback text.

[0706] Step 6:

[0707] The terminal provides the generated response text and feedback text to the user. The terminal's display or audio output function is used to display or audio-output the response and feedback to the user. The input is the response text and feedback text, and the output is the display or audio output to the user.

[0708] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0709] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0710] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0711] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0712] [Second embodiment]

[0713] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0714] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0715] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0716] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0717] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0718] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0719] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0720] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0721] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0722] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0723] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0724] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0725] "Example 1"

[0726] One embodiment of the present invention is a system that provides interactive advice to elementary, junior high, and high school students and the elderly about their concerns, etc., as a means of providing conversational information. This system receives input information from the user and generates a response in response. Specifically, it provides advice and solutions to the concerns and problems the user is facing. It is also possible to have real-time conversations using voice recognition and output. This allows the user to talk about their concerns in their own words, and the system then provides appropriate advice.

[0727] "Example 2"

[0728] Furthermore, another embodiment of the present invention is a system that provides answers and advice to test-takers, much like a private tutor, by inputting problem information and unclear points. This system receives problem information and unclear points from the user and generates answers and advice accordingly. Specifically, it provides answers and explanations for problems and unclear points that the user encounters while studying. This allows the user to study at their own pace and receive the necessary support when needed.

[0729] "Example 3"

[0730] A further embodiment of the present invention is a system for conducting English conversation and providing English language education. The system receives English utterances from a user and generates English responses in response to the utterances. Specifically, the user conducts a conversation in English, and the system provides responses in English in response. This allows the user to gain experience using English in a real conversational situation. Furthermore, the system provides feedback on the user's utterances to help improve the user's English ability.

[0731] The processing flow of each embodiment will be described below.

[0732] "Example 1"

[0733] Step 1: The user speaks their concerns or problems into the system.

[0734] Step 2: The system uses voice recognition technology to convert the user's input into text information.

[0735] Step 3: The system analyzes the text information and generates appropriate advice or solutions. Step 4: The system outputs the generated advice or solutions to the user aloud.

[0736] "Example 2"

[0737] Step 1: The test-taker enters the problem information and any questions they have about the problem into the system.

[0738] Step 2: The system analyzes the entered problem information and unclear points, and generates appropriate answers and explanations.

[0739] do.

[0740] Step 3: Output the generated answers and explanations to the test-taker.

[0741] "Example 3"

[0742] Step 1: The user speaks to the system in English.

[0743] Step 2: The system analyzes the user's utterance and generates an appropriate English response.

[0744] Step 3: Output the generated English response to the user.

[0745] Step 4: The system provides feedback on the user's speech to help improve their English skills.

[0746] Example 1

[0747] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0748] In modern society, elementary, junior high, and high school students, as well as the elderly, have a wide range of worries and problems, and there is a need for appropriate mental care for them. However, with conventional systems, it is difficult to provide consultation using real-time dialogue or voice recognition and output, making it difficult for users to talk about their worries in their own words and receive appropriate advice. In addition, there is a lack of support, such as private tutoring for students taking exams, or English conversation education. A system that can solve these issues is needed.

[0749] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0750] In this invention, the server includes means for acquiring user input information, means for converting voice data into text data using voice recognition technology, means for generating a response based on the text data using a generative AI model, means for converting the generated response into voice data using voice output technology and providing it, means for providing interactive advice on worries and other issues as mental care to elementary, junior high, and high school students and the elderly, and means for conducting real-time conversations. This allows users to talk about their worries in their own words and receive appropriate advice in real time. It can also provide support like a private tutor for students taking exams and English conversation lessons.

[0751] The "means for acquiring user input information" refers to a device or software that provides an interface for users to input information to the system in the form of voice, text, or the like.

[0752] "Speech recognition technology" is a technology that converts a user's voice into a digital signal, analyzes that signal, and converts it into text data.

[0753] A "generative AI model" is an artificial intelligence algorithm that generates appropriate responses based on input text data.

[0754] "Voice output technology" is a technology that converts text data into voice data and provides it to the user as voice.

[0755] "A means of providing interactive consultation on worries and problems as part of mental care" is a system that receives consultations in an interactive format about the worries and problems that users are facing, and provides appropriate advice and solutions.

[0756] "Means for real-time conversation" refers to technologies and devices that allow users and systems to exchange information instantly and engage in dialogue without delay.

[0757] "Tutor-like support for test takers" is a system that provides appropriate answers and advice to test takers regarding their learning problems and questions.

[0758] "Providing English conversation education" is a system that provides educational support to help users practice English conversation and improve their English skills.

[0759] "Extensibility to add options" refers to a design that is flexible enough to allow new functions and services to be added to the system.

[0760] MODE FOR CARRYING OUT THE INVENTION

[0761] This invention is a system that provides interactive advice to elementary, junior high, and high school students and the elderly about their concerns, etc. It acquires user input information and generates responses to that information. Specifically, it can conduct real-time conversations using voice recognition technology, a generative AI model, and voice output technology.

[0762] Hardware and software used

[0763] The system uses the following hardware and software:

[0764] Device: The device used by the user, such as a smartphone, tablet, or computer

[0765] Speech recognition technology: Speech recognition software such as Google Cloud Speech-to-Text

[0766] Generative AI models: Generative AI models such as OpenAI's GPT-4

[0767] Speech output technology: Text-to-speech software such as Amazon Polly

[0768] Server: A server for processing data and running generative AI models

[0769] System Operation

[0770] The user speaks to the system using the device. For example, an elementary school student might ask, "I had a fight with a friend. How can I make up?" This voice input is captured through the device's microphone.

[0771] The device uses Google Cloud Speech-to-Text to convert the voice data into text data, which contains the user's concerns and problems.

[0772] The server receives the text data sent from the device and generates a prompt sentence to be input to the generative AI model (GPT-4). The prompt sentence has the following format:

[0773] "A primary school student is asking for advice about a fight he had with a friend. Can you offer some advice on how to make up?"

[0774] The generative AI model generates appropriate advice based on this prompt, such as, "First, it's important to apologize to your friend. Then, it's important to talk about the cause of the argument and come to a mutual understanding."

[0775] The server sends the generated response to the device, which then uses Amazon Polly to convert the response into audio data and provides it to the user, who can receive the system's response through the audio played from the device.

[0776] This system allows users to talk about their concerns in their own words and receive appropriate advice in real time. It can also provide support like a private tutor for students taking exams, or English conversation lessons.

[0777] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0778] Step 1:

[0779] The user speaks into the device's microphone to describe their worries or problems. For example, they might say, "I had a fight with a friend. How can we make up?" The input is captured in the device as voice data.

[0780] Step 2:

[0781] The device uses speech recognition technology to convert the voice data into text data. Specifically, it uses Google Cloud Speech-to-Text to analyze the voice data and generate text data such as "I had a fight with my friend. How can I make up?" The input is voice data, and the output is text data.

[0782] Step 3:

[0783] The device sends the generated text data to the server. The server receives this text data and generates a prompt to be input into the generative AI model. For example, the prompt might read, "An elementary school student has had a fight with a friend and is asking for advice. Please provide some advice on how to make up." The input is text data, and the output is a prompt.

[0784] Step 4:

[0785] The server inputs a prompt into a generative AI model (GPT-4) and generates a response to the user's concerns or problems. The generative AI model generates text data containing appropriate advice based on the prompt. For example, it generates a response such as, "First, it's important to apologize to your friend. After that, it's important to discuss the cause of the fight and come to a mutual understanding." The input is the prompt, and the output is the response text.

[0786] Step 5:

[0787] The server sends the generated response text to the device. The device then uses voice output technology to convert the response text into voice data. Specifically, Amazon Polly is used to convert the text data into voice data, generating voice data such as, "First, it's important to apologize to your friend. Then, it's important to talk about the cause of the argument and come to a mutual understanding." The input is the response text, and the output is voice data.

[0788] Step 6:

[0789] The terminal plays the generated voice data and provides it to the user. The user receives the system's response through the voice played from the terminal. This allows the user to receive appropriate advice in real time. The input is voice data, and the output is a voice response to the user.

[0790] (Application example 1)

[0791] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0792] In modern society, mental health issues faced by elementary, junior high, and high school students, as well as the elderly, are serious, and appropriate support is needed. However, conventional mental care systems have difficulty providing appropriate advice to users in real time regarding their concerns, and few systems have advanced dialogue functions that utilize voice recognition or generative AI models. For this reason, there is a need for a system that allows users to talk about their concerns in their own words and receive appropriate advice in real time.

[0793] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0794] In this invention, the server includes means for providing conversation information, means for interactively offering advice on worries and the like as part of mental care to elementary, junior high, and high school students and the elderly, means for conducting conversations in real time using voice recognition and output, means for providing advice and solutions to users' worries using a generative AI model, means for generating prompt sentences and generating appropriate responses, and means for providing feedback in real time using a voice output function. This enables users to talk about their worries in their own words and receive appropriate advice in response to them in real time.

[0795] "Means for providing conversational information" is a function for collecting information through dialogue with the user and generating an appropriate response.

[0796] "A means of providing interactive consultation to elementary, junior high, and high school students and the elderly regarding their worries and problems as part of mental care" is a function that allows elementary, junior high, and high school students and the elderly to discuss their worries and problems in an interactive format and provide appropriate advice and support.

[0797] "Means for conducting real-time conversations using voice recognition and output" refers to a function that enables real-time conversations by recognizing the user's voice, converting it into text, and outputting the generated response as voice.

[0798] "Means of providing advice and solutions to users' concerns using generative AI models" refers to a function that uses generative AI models to generate appropriate advice and solutions to users' concerns.

[0799] The "means for generating a prompt sentence and generating an appropriate response" is a function for generating a prompt sentence based on a user's input and generating an appropriate response based on the prompt sentence.

[0800] The "means for providing feedback in real time using a voice output function" is a function for outputting the generated response by voice and providing feedback to the user in real time.

[0801] A system for carrying out this invention is configured as follows: The server includes means for providing conversation information, means for interactively offering advice on worries and the like as part of mental care to elementary, junior high, and high school students and the elderly, means for conducting conversations in real time using voice recognition and output, means for providing advice and solutions to users' worries using a generative AI model, means for generating prompt sentences to generate appropriate responses, and means for providing feedback in real time using a voice output function.

[0802] Program processing explanation

[0803] The server uses speech recognition software (e.g., the speech_recognition library) to convert the user's voice into text. What the user says through the microphone is sent as voice data to the server, which then converts this voice data into text data.

[0804] Next, a generative AI model (e.g., OpenAI's API) is used to generate appropriate advice or solutions based on the text data. The generative AI model generates prompts based on the user's input and then generates responses based on those prompts.

[0805] The generated response is converted into voice data using voice output software (e.g., gTTS library) and fed back to the user in real time. The voice output is provided to the user through a speaker.

[0806] Specific examples

[0807] For example, if a user says, "I'm having trouble making friends at school," the server converts this speech to text and generates the following prompt:

[0808] Example prompt sentence:

[0809] User's problem: I'm having trouble making friends at school

[0810] Please provide advice.

[0811] The generative AI model generates appropriate advice based on the prompt, and then converts the advice into voice data using voice output software, allowing the user to receive the advice in real time through a speaker.

[0812] In this way, users can talk about their concerns in their own words and receive appropriate advice in real time.

[0813] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0814] Step 1:

[0815] The user speaks about their concerns through a microphone. The device collects this voice data and sends it to the server. The input is the user's voice data, and the output is the voice data sent to the server.

[0816] Step 2:

[0817] The server uses speech recognition software (e.g., the speech_recognition library) to convert the received voice data into text data. The input is voice data, and the output is text data. Specifically, the speech recognition software analyzes the voice data and generates the corresponding text.

[0818] Step 3:

[0819] The server uses a generative AI model (e.g., OpenAI's API) to generate a prompt based on the text data. The input is the user's text data, and the output is the prompt. Specifically, the generative AI model analyzes the text data and generates an appropriate prompt.

[0820] Step 4:

[0821] The server uses the generated prompt sentence to have the generative AI model generate appropriate advice or solutions. The input is the prompt sentence, and the output is the generated advice or solution. In concrete terms, the generative AI model generates a response based on the prompt sentence.

[0822] Step 5:

[0823] The server converts the generated advice or solution into voice data using voice output software (e.g., gTTS library). The input is the generated text data, and the output is voice data. Specifically, the voice output software converts the text data into voice data.

[0824] Step 6:

[0825] The server transmits the voice data to the terminal, and the terminal provides real-time feedback to the user through the speaker. The input is the voice data, and the output is the voice feedback provided to the user. In concrete terms, the terminal plays the voice data and provides feedback to the user.

[0826] Example 2

[0827] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0828] Conventional learning support systems have had the problem of being unable to provide immediate, appropriate answers or advice to specific problems or unclear points that learners encounter during their studies. Another problem is that learners cannot receive the support they need when they need it, as they progress through their studies at their own pace. Furthermore, there is a lack of means for providing mental care or interactive consultations, and comprehensive support for learners is insufficient.

[0829] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0830] In this invention, the server includes means for providing conversation information, means for interactively offering advice to learners about their worries and other concerns as part of mental care, means for conducting conversations in real time using voice recognition and output, means for receiving problem information and unclear points entered by the learner and generating answers and advice for them, means for generating answers and advice using a generative AI model, and means for providing the generated answers and advice to the learner. This allows learners to progress through their studies at their own pace and receive the support they need when they need it.

[0831] "Conversational information" refers to the content of text and voice communication exchanged between a user and a system.

[0832] "Mental care" refers to interactive support to reduce learners' psychological worries and stress and support their mental health.

[0833] "Speech recognition" is a technology that converts speech into text data.

[0834] "Audio output" is a technology that reproduces text data as audio.

[0835] "Real-time" refers to immediate processing or response without delay.

[0836] "Learner" refers to an individual engaged in an education or learning activity.

[0837] "Problem information" refers to the content of specific problems or issues that a learner encounters during learning.

[0838] "Unclear points" refer to parts that learners do not understand or have questions about.

[0839] An "answer" is a correct answer or solution to a problem submitted by a learner.

[0840] "Advice" is advice or guidance to help learners solve problems.

[0841] A "generative AI model" is an algorithm or system that uses artificial intelligence to generate text or speech.

[0842] A "prompt" is text that contains instructions or questions that are input to a generative AI model.

[0843] The present invention is a system that instantly provides appropriate answers and advice to specific problems and unclear points that a learner encounters during learning. A specific embodiment of this system will be described below.

[0844] Hardware and software used

[0845] 1. Terminal

[0846] These are devices such as computers and smartphones used by learners. These devices are connected to the Internet and are used to input problem information and any questions through a user interface.

[0847] 2. Server

[0848] A computer system for processing data and running a generative AI model. The server receives data sent by users and inputs it into the generative AI model.

[0849] 3. Generative AI Models

[0850] For example, advanced natural language processing models such as OpenAI's GPT-4 generate appropriate answers and advice based on the problem information and questions users provide.

[0851] Data processing and calculation

[0852] 1. User Input

[0853] The user uses the terminal to input information about the problems and points of confusion they encounter while studying, for example, "I don't know how to solve differential equations in mathematics."

[0854] 2. Data transmission

[0855] The device sends the problem information and uncertainties entered by the user to the server, which converts the received data into an appropriate format and inputs it into the generative AI model.

[0856] 3. Use of generative AI models

[0857] The server inputs data to the generative AI model as a prompt sentence. For example, it sends the prompt sentence "Please tell me how to solve a differential equation" to the generative AI model.

[0858] 4. Generating Answers and Advice

[0859] The generative AI model generates appropriate answers and advice based on the input prompt. For example, it might generate an answer like, "To solve a differential equation, first transform the equation into standard form, then solve the characteristic equation."

[0860] 5. User Feedback

[0861] The server sends the generated answers and advice to the terminal. The terminal displays the received answers and advice to the user. For example, the user's screen may display a message such as, "Here's how to solve a differential equation: First, transform the equation into standard form, and then solve the characteristic equation."

[0862] Specific examples

[0863] Example user input

[0864] "I don't know how to graph a linear function."

[0865] Example prompts for generative AI models

[0866] "How do I draw the graph of a linear function?"

[0867] Example of a generated answer

[0868] "To graph a linear function, first determine the shape of the function. For example, in y = 2x + 3, the y-intercept is 3 and the slope is 2. Next, plot the y-intercept and draw a line according to the slope."

[0869] This way, users can learn at their own pace and get the support they need when they need it.

[0870] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0871] Step 1:

[0872] Users use the terminal to input information about problems and questions they encounter while studying.

[0873] Specifically, the user opens a dedicated application or web browser on their device and enters the problem information and any points of uncertainty into the input form. For example, they might enter, "I don't know how to solve a differential equation in mathematics."

[0874] Input: User-entered problem information and questions.

[0875] Output: Problem information and unknown data entered into the terminal.

[0876] Step 2:

[0877] The terminal sends the problem information and unclear points entered by the user to the server.

[0878] Specifically, the terminal transmits the user's input data to the server via the Internet.

[0879] Input: Problem information and question data entered into the terminal.

[0880] Output: Problem information and unknown data sent to the server.

[0881] Step 3:

[0882] The server converts the received problem information and uncertainties into a suitable format for input into the generative AI model.

[0883] Specifically, the server converts the data into a prompt, for example, "Please tell me how to solve a differential equation."

[0884] Input: Problem information and unknown data sent to the server.

[0885] Output: A prompt to be fed into the generative AI model.

[0886] Step 4:

[0887] The server inputs a prompt sentence into the generative AI model.

[0888] Specifically, the server sends a prompt to the generative AI model and waits for the model to generate an answer or advice.

[0889] Input: A prompt to be used as input to the generative AI model.

[0890] Output: The answer or advice returned by the generative AI model.

[0891] Step 5:

[0892] The generative AI model generates answers and advice based on the input prompt.

[0893] Specifically, the generative AI model uses natural language processing technology to generate appropriate answers and advice. For example, it might generate an answer like, "Here's how to solve a differential equation: First, transform the equation into standard form, and then solve the characteristic equation."

[0894] Input: The prompt text fed into the generative AI model.

[0895] Output: The generated answer or advice.

[0896] Step 6:

[0897] The server sends the generated answers and advice to the terminal.

[0898] Specifically, the server sends the answers and advice received from the generative AI model to the user's device.

[0899] Input: Generated answers and advice.

[0900] Output: Answers and advice sent to your device.

[0901] Step 7:

[0902] The device displays the answers and advice received to the user.

[0903] Specifically, the device displays the answer or advice on the user's screen. For example, the device displays a message on the user's screen saying, "Here's how to solve the differential equation: First, transform the equation into standard form, then solve the characteristic equation."

[0904] Input: Answers and advice sent to the device.

[0905] Output: The answer or advice displayed to the user.

[0906] (Application example 2)

[0907] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0908] Conventional learning support systems have difficulty providing immediate answers and advice to problems and unclear points that students encounter while studying. Furthermore, they lack the scalability to accommodate diverse needs, such as mental health care and English education. This makes it difficult for users to progress through their studies at their own pace, and they are unable to receive the support they need when they need it.

[0909] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for providing conversation information, means for interactively offering advice on worries and the like as part of mental care to elementary, junior high, and high school students and the elderly, means for conducting real-time conversations using voice recognition and output, means for inputting problem information and unclear points from the user and generating answers and advice, means for providing answers and advice using a generative AI model, and means including an application to be installed on a smartphone. This allows users to study at their own pace and receive the necessary support when they need it.

[0910] "Conversational information" is information provided through dialogue with the user.

[0911] "Mental care" is care to support the user's mental health.

[0912] "Speech recognition" is a technology that converts speech into text data.

[0913] "Audio output" is a technology that reproduces text data as audio.

[0914] "Real-time" refers to processing occurring immediately without delay.

[0915] "Problem information" refers to the specific problems or questions that users encounter while studying.

[0916] "Unclear points" are parts that users do not understand or have questions about.

[0917] An "answer" is the correct answer to a question.

[0918] "Advice" is advice on how to solve problems or proceed with learning.

[0919] A "generative AI model" is a model that uses artificial intelligence to generate answers and advice from data.

[0920] A "smartphone" is a type of mobile phone, a device that can connect to the Internet and use applications.

[0921] An "application" is software that provides specific functions or services.

[0922] To implement the present invention, the following system configuration and program are required.

[0923] System Configuration

[0924] 1. Hardware:

[0925] Smartphone: A device where users can enter problem information or questions.

[0926] Server: A computer system that runs generative AI models and generates answers and advice.

[0927] 2. Software:

[0928] Python: A programming language used to implement the program.

[0929] OpenAI API: An API for using generative AI models.

[0930] Program processing

[0931] 1. User input:

[0932] Users enter any problems or questions they encounter while studying in text format through a smartphone application.

[0933] 2. Data transmission:

[0934] The smartphone sends the entered problem information and any unclear points to the server.

[0935] 3. Run the generative AI model:

[0936] The server uses the OpenAI API to run the generative AI model and generate answers and advice for the input problem information and unknown points.

[0937] 4. Return of answers and advice:

[0938] The server sends the generated answers and advice back to the smartphone and displays them to the user.

[0939] Specific examples

[0940] For example, if a user types, "Please tell me the formula for solving a quadratic equation," the following prompt sentence is sent to the generative AI model:

[0941] Example prompt sentence:

[0942] A candidate is asking the following question: "What is the formula for solving a quadratic equation?"

[0943] Please provide answers and advice.

[0944] The generative AI model generates answers and advice based on this prompt, returning the following results:

[0945] Answers and advice:

[0946] The formula for solving the quadratic equation ax^2 + bx + c = 0 is:

[0947] x = (-b ± √(b^2 - 4ac)) / 2a

[0948] When using this formula, it is important to substitute the values ​​of a, b, and c correctly. Also, note that the number of solutions changes depending on the value of the discriminant (b^2 - 4ac): if the discriminant is positive, you get two real solutions, if it is zero, you get one real solution, and if it is negative, you get two imaginary solutions.

[0949] This way, users can learn at their own pace and get the support they need when they need it.

[0950] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0951] Step 1:

[0952] The user starts the smartphone application and inputs any problems or questions they encounter while studying in text format. The input text is temporarily saved in the application.

[0953] Input: Text data of the problem information or questions entered by the user

[0954] Output: Temporarily saved text data

[0955] Step 2:

[0956] The device (smartphone) sends the entered text data to the server via an internet connection.

[0957] Input: Temporarily saved text data

[0958] Output: Text data sent to the server

[0959] Step 3:

[0960] The server analyzes the received text data and generates a prompt suitable for the generative AI model, which is then formatted to clarify the user's question.

[0961] Input: Text data sent to the server

[0962] Output: Generated prompt statement

[0963] Step 4:

[0964] The server sends the generated prompt to the OpenAI API and executes the generative AI model, which generates answers and advice based on the prompt.

[0965] Input: Generated prompt text

[0966] Output: The answer or advice generated by the generative AI model

[0967] Step 5:

[0968] The server receives the answers and advice returned by the generative AI model, formats them in a user-friendly format, and then sends the formatted answers and advice back to the smartphone.

[0969] Input: Answers and advice generated by a generative AI model

[0970] Output: Formatted answers and advice

[0971] Step 6:

[0972] The terminal (smartphone) displays the answers and advice received from the server to the user, allowing the user to obtain answers and advice for problems and unclear points during study.

[0973] Input: Formatted answers and advice

[0974] Output: The answer or advice displayed to the user

[0975] Example 3

[0976] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0977] Conventional English language education systems and mental health care systems have difficulty providing real-time dialogue and feedback, and have been unable to quickly provide appropriate responses or advice to users' utterances. It has also been difficult to accurately understand the content of users' utterances and generate natural responses that fit the context. This has led to issues such as insufficient improvement in users' English ability and in the effectiveness of mental health care.

[0978] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0979] In this invention, the server includes means for providing conversation information, means for interactively consulting with a user about worries and the like as part of mental care, means for conducting conversations in real time using voice recognition and output, means for receiving the user's speech as voice data and converting it into text data using voice recognition software, means for inputting prompt sentences into a generative AI model and generating an appropriate response, means for converting the generated response into voice data and providing it to the user, and means for generating feedback on the user's speech and providing it to the user. This enables users to improve their English ability and receive the benefits of mental care while engaging in natural conversations in real time.

[0980] "Conversation information" is information about the dialogue that users have through the system.

[0981] "Mental care" means providing interactive consultation and advice to support the mental health of users.

[0982] "Speech recognition" is a technology that converts voice data into text data.

[0983] "Voice output" is a technology that converts text data into voice data and provides it to the user as voice.

[0984] "Real-time" refers to providing immediate responses and feedback to user utterances.

[0985] "Voice data" refers to data that is a digital recording of a user's speech.

[0986] "Text data" is character information converted from voice data using voice recognition technology.

[0987] A "generative AI model" is an artificial intelligence model that understands the content of a user's speech and generates natural responses according to the context.

[0988] A "prompt sentence" is text containing contextual information that is input into a generative AI model.

[0989] "Feedback" refers to providing evaluations and advice on the user's speech.

[0990] The present invention is a system that allows a user to make an utterance in English, generates an appropriate response to the utterance, and provides feedback. A specific embodiment of this system will be described below.

[0991] First, a user accesses the system using a device (e.g., a smartphone or PC). When the user speaks in English, the device captures the speech as voice data. This voice data is then sent to a server via the Internet.

[0992] The server converts the received voice data into text data using speech recognition software (e.g., a speech recognition API). The speech recognition software analyzes the voice data and generates corresponding text data. This text data is used as part of a prompt sentence that is input into the generative AI model.

[0993] Next, the server inputs a prompt sentence to the generative AI model (e.g., a natural language processing model), which looks like this:

[0994] User: What is the weather like today?

[0995] system:

[0996] The generative AI model generates an appropriate response based on this prompt sentence. For example, it might generate a response such as, "The weather today is sunny with a high of 25 degrees Celsius." This response is then converted into voice data by the server using speech synthesis software (e.g., a speech synthesis API).

[0997] The generated voice data is then sent back to the terminal via the Internet, where it is played back and a response in English is provided to the user, allowing the user to have a natural conversation in real time.

[0998] Additionally, the server generates feedback on the user's speech, assessing things like pronunciation accuracy and grammar, and including suggestions to help the user improve their English proficiency. For example, feedback might be generated such as, "Your pronunciation was clear, but you could improve your intonation on the word 'weather'."

[0999] The generated feedback is then sent back to the terminal via the Internet, where it is displayed to the user, who can then receive specific advice on their own speech and improve their English ability.

[1000] This system combines technologies such as speech recognition software, generative AI models, and speech synthesis software to provide users with natural dialogue and feedback in real time. This allows users to gain experience using English in real conversation situations, thereby improving their English ability. The flow of the identification process in Example 3 will be described with reference to Figure 15.

[1001] Step 1:

[1002] The user speaks in English.

[1003] The user asks a question or speaks to the system in English, for example, "What is the weather like today?" The input is the user's voice, and the output is the voice data captured on the device.

[1004] Step 2:

[1005] The terminal transmits the voice data to the server.

[1006] The device (such as a smartphone or PC) captures the user's speech as voice data and sends this voice data to a server via the Internet. The input is the user's voice data, and the output is the voice data sent to the server.

[1007] Step 3:

[1008] The server converts the voice data into text data.

[1009] The server converts the received voice data into text data using voice recognition software (e.g., a voice recognition API). The voice recognition software analyzes the voice data and generates corresponding text data. The input is voice data, and the output is text data.

[1010] Step 4:

[1011] The server inputs a prompt sentence into the generative AI model and generates a response.

[1012] The server inputs a prompt sentence to the generative AI model (e.g., a natural language processing model). The prompt sentence looks like this:

[1013] User: What is the weather like today?

[1014] system:

[1015] The generative AI model generates an appropriate response based on this prompt. For example, it generates a response such as, "The weather today is sunny with a high of 25 degrees Celsius." The input is the text data and the prompt, and the output is the generated response text.

[1016] Step 5:

[1017] The server converts the generated response into audio data.

[1018] The server converts the generated response into audio data using speech synthesis software (e.g., a speech synthesis API), where the input is the generated response text and the output is the audio data.

[1019] Step 6:

[1020] The server sends the audio data to the terminal.

[1021] The server transmits the generated voice data to the terminal via the Internet. The input is the voice data, and the output is the voice data transmitted to the terminal.

[1022] Step 7:

[1023] The terminal plays the response to the user.

[1024] The terminal plays the received voice data and provides a response in English to the user. The input is the voice data and the output is the voice played to the user.

[1025] Step 8:

[1026] The server generates feedback in response to the user's utterance.

[1027] The server generates feedback on the user's speech, evaluating the accuracy of pronunciation and grammar, and providing advice to help the user improve their English skills. The input is the user's speech text, and the output is the feedback text.

[1028] Step 9:

[1029] The server sends the feedback to the device.

[1030] The server transmits the generated feedback to the terminal via the Internet. The input is the feedback text, and the output is the feedback transmitted to the terminal.

[1031] Step 10:

[1032] The device displays the feedback to the user.

[1033] The device displays the received feedback to the user, for example, advice such as "Your pronunciation was clear, but you could improve your intonation on the word 'weather'." The input is the feedback text, and the output is the feedback displayed to the user.

[1034] (Application example 3)

[1035] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1036] Conventional English education systems and mental health care systems make it difficult for users to gain experience using English in real conversational situations, and do not provide real-time feedback, limiting the improvement of English proficiency. Furthermore, virtual stores also lack real-time responses when users ask questions about products or make purchases in English, resulting in an inadequate user experience. To address these issues, a system that provides real-time English responses and feedback is needed.

[1037] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1038] In this invention, the server includes a means for providing conversational information, a means for interactively providing advice to elementary, junior high, and high school students and the elderly about their worries as part of mental care, a means for conducting conversations in real time using voice recognition and output, a means for providing real-time English responses when a user asks a question about a product in English or makes a purchase in a virtual store, and a means for providing feedback on the user's speech to help improve their English ability. This allows users to gain experience using English in actual conversational situations and improve their English ability through real-time feedback. This also improves the user experience in the virtual store.

[1039] "Conversation information" refers to information about what the user says and the system's response.

[1040] "Mental care" refers to supporting mental health by alleviating users' psychological worries and stress through dialogue.

[1041] "Speech recognition" is a technology that collects a user's speech through a microphone and converts it into text data.

[1042] "Voice output" is a technology that converts text data into voice and provides it to the user through a speaker.

[1043] "Real-time" refers to providing instant responses and feedback to user utterances.

[1044] A "virtual store" is a virtual store that exists on the Internet, where users can browse and purchase products online.

[1045] "Feedback" is information that provides evaluation and advice on a user's speech or behavior, encouraging improvement.

[1046] "English proficiency" refers collectively to a user's ability to understand, speak, and write English.

[1047] "User experience" refers to the overall satisfaction and convenience that users feel when using a system or service.

[1048] A system for carrying out this invention has the following configuration: The server includes means for providing conversation information, means for interactively offering advice on worries and the like as part of mental care for elementary, junior high, and high school students and the elderly, means for conducting conversations in real time using voice recognition and output, means for providing real-time English responses when a user asks a question about a product in English or goes through the purchasing process in a virtual store, and means for providing feedback on the user's speech and supporting the improvement of English ability.

[1049] Hardware and software used

[1050] Hardware: smartphone, head-mounted display, microphone, speaker

[1051] Software: OpenAI API (generative AI model), SpeechRecognition (voice recognition), pyttsx3 (voice synthesis)

[1052] Data processing and calculation

[1053] The server first collects the user's speech through a microphone and converts the speech data into text using the SpeechRecognition library. It then uses the OpenAI API to generate an appropriate English response to the user's speech. This response is generated based on the prompt sentence using a generative AI model. The generated response is then converted into audio using the pyttsx3 library and provided to the user through the speaker.

[1054] Additionally, feedback is generated based on the user's speech and the generated response, providing feedback in the form of audio or text to help improve the user's English proficiency.

[1055] Specific examples

[1056] For example, if a user asks in English, "Tell me about this product," in a virtual store, the server responds, "Sure, this product is a high-quality, eco-friendly item made from sustainable materials. Would you like to know more details?" It also provides feedback, "Your pronunciation was good, but try to use more complex sentences."

[1057] Prompt Sentence Examples

[1058] User utterance: "Can you tell me about this product?"

[1059] Prompt to generative AI model: "The user asked about a product. Provide a detailed description in English."

[1060] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1061] Step 1:

[1062] The user speaks into the microphone in English.

[1063] Input: User's voice data

[1064] Output: Audio data collected by microphone

[1065] Step 2:

[1066] The device uses speech recognition software (SpeechRecognition) to convert the collected voice data into text data.

[1067] Input: Audio data

[1068] Output: Text data

[1069] Specific operation: The device launches voice recognition software, analyzes the voice data, and generates corresponding text data.

[1070] Step 3:

[1071] The server uses a generative AI model (OpenAI API) to generate appropriate English responses based on the text data.

[1072] Input: User's text data

[1073] Output: Generated English response

[1074] Specific operation: The server sends the text data as a prompt to the generative AI model, which then generates a response.

[1075] Step 4:

[1076] The server converts the generated English response into voice data using speech synthesis software (pyttsx3).

[1077] Input: Generated English responses (text data)

[1078] Output: Audio data

[1079] Specific operation: The server starts the speech synthesis software and converts the text data into speech data.

[1080] Step 5:

[1081] The terminal plays the generated audio data to the user through a speaker.

[1082] Input: Audio data

[1083] Output: The audio the user hears

[1084] Specific operation: The device turns on the speaker and plays audio data.

[1085] Step 6:

[1086] The server generates feedback for the user's speech and provides it in the form of voice or text.

[1087] Input: User utterance text data, generated English responses

[1088] Output: Feedback (audio or text)

[1089] What it does: The server analyzes the user's speech and the generated responses, and generates an evaluation and advice. The generated feedback is converted into audio data using speech synthesis software and played through the speaker or displayed as text.

[1090] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1091] "Example 1"

[1092] One embodiment of the present invention is a conversation service system incorporating an emotion engine. This system receives a user's voice input and converts it into text information using speech recognition technology. The emotion engine then analyzes this text information to understand the user's emotional state. For example, if a user says, "I got a bad score on the test today. I'm feeling depressed," the emotion engine recognizes the emotional state as "depressed." Based on this emotional state, the system then generates appropriate advice or solutions and outputs them in voice.

[1093] "Example 2"

[1094] Another embodiment is a tutoring service system incorporating an emotion engine. In this system, when a test-taker inputs problem information or points of uncertainty during study, the emotion engine analyzes the user's emotional state. For example, if the user says, "I can't solve this math problem and I'm frustrated," the emotion engine recognizes the user's emotional state as "frustrated." Based on this emotional state, the system generates appropriate answers and explanations, and even provides calming advice.

[1095] "Example 3"

[1096] Yet another embodiment is an English conversation education service system incorporating an emotion engine. In this system, when a user speaks in English, the emotion engine analyzes the user's emotional state. For example, if the user says, "I'm having trouble pronouncing this English phrase," the emotion engine recognizes the user's emotional state as "trouble." Based on this emotional state, the system then generates appropriate pronunciation guidance and advice to support the user's English learning.

[1097] The processing flow of each embodiment will be described below.

[1098] "Example 1"

[1099] Step 1: Receive voice input from the user.

[1100] Step 2: Use speech recognition technology to convert the voice input into text information.

[1101] Step 3: The emotion engine analyzes the text information and understands the user's emotional state.

[1102] Step 4: Based on the emotional state, the system generates appropriate advice and solutions.

[1103] Step 5: Output the generated advice or solution aloud.

[1104] "Example 2"

[1105] Step 1: The test-taker enters the problem information and any questions they have about the test into the system.

[1106] Step 2: The emotion engine analyzes the user's emotional state.

[1107] Step 3: Based on the emotional state, the system generates an appropriate answer or explanation.

[1108] Step 4: Provide calming advice if needed.

[1109] "Example 3"

[1110] Step 1: The user inputs an English utterance into the system.

[1111] Step 2: The emotion engine analyzes the user's emotional state.

[1112] Step 3: Based on the emotional state, the system generates appropriate pronunciation guidance and advice. Step 4: The system provides the generated pronunciation guidance and advice to the user to support English learning.

[1113] Example 1

[1114] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1115] In modern society, there is a demand for providing appropriate advice and solutions to the worries and problems faced by elementary, junior high, and high school students, as well as the elderly. However, there are only a limited number of systems that allow these users to consult in real time, and in particular, there are few systems that provide responses that take emotional state into consideration. In addition, systems that combine speech recognition and generative AI models are complex, making them difficult to put into practical use.

[1116] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes means for receiving a user's voice input, means for converting the voice data into text data, means for analyzing the text data to grasp the user's emotional state, means for generating an appropriate response using a generative AI model, means for converting the generated response into voice, and means for outputting the voice response to the user. This enables real-time interactive consultation that takes the user's emotional state into consideration.

[1117] "Means for receiving user voice input" refers to devices or software that capture the voice uttered by the user and input it into the system.

[1118] A "means for converting voice data to text data" is a process or device that uses voice recognition technology to convert a voice signal into corresponding text information.

[1119] "Means for analyzing text data to understand a user's emotional state" refers to algorithms or software for analyzing text information and identifying a user's emotional or psychological state.

[1120] "Means for generating appropriate responses using generative AI models" refers to models or systems that use artificial intelligence technology to generate appropriate responses or advice in response to user input.

[1121] The "means for converting the generated response into speech" refers to speech synthesis technology or a device that converts the text response into a speech signal and plays it aloud to the user.

[1122] The "means for outputting a voice response to a user" refers to a device or system for transmitting the generated voice to a user through an output device such as a speaker.

[1123] This invention is a system that provides real-time interactive advice to elementary, junior high, and high school students and the elderly regarding their worries and problems. This system can receive user voice input, generate appropriate responses, and output them as voice.

[1124] Hardware and software used

[1125] Hardware: microphones, speakers, servers

[1126] Software: speech recognition software (e.g., Google Speech-to-Text), sentiment analysis engines (e.g., IBM Watson Tone Analyzer), generative AI models (e.g., OpenAI GPT-3), and speech synthesis software

[1127] System configuration

[1128] 1. A means of receiving user voice input:

[1129] The user talks about their worries and problems through the microphone. For example, the user might say, "I got a bad score on the test today. I'm feeling down."

[1130] 2. How to convert audio data to text data:

[1131] The device uses speech recognition software to convert the user's speech into text, specifically using Google Speech-to-Text to generate the text "I got a bad score on the test today. I'm feeling down."

[1132] 3. Analyzing text data to understand the user's emotional state:

[1133] The server uses an emotion analysis engine to analyze the text information and understand the user's emotional state. Specifically, it uses IBM Watson Tone Analyzer to recognize the emotion "depressed."

[1134] 4. Using generative AI models to generate appropriate responses:

[1135] The server uses a generative AI model to generate appropriate advice and solutions based on the user's emotional state. Specifically, it uses OpenAI GPT-3 to generate a response such as, "Even if you got a bad score on the test, you'll be fine if you try harder next time. Let me know if there's anything I can help you with."

[1136] 5. Means for converting the generated response into speech:

[1137] The server then uses speech synthesis software to convert the generated text response into speech, specifically using speech synthesis technology to generate a voice message that says, "Even if you got a bad score on the test, you can do better next time. Let me know if there's anything I can help you with."

[1138] 6. A means of outputting a voice response to the user:

[1139] The device receives the voice data from the server and outputs it to the user through the speaker. Specifically, the speaker plays a message saying, "Even if you got a bad score on the test, you'll be fine if you try harder next time. Let me know if there's anything I can help you with."

[1140] Specific examples

[1141] User Type: "I had a fight with a friend today. How can I make up?"

[1142] System response: "I see you had a fight with your friend. The first thing you need to do is stay calm and try to understand the other person's feelings. Choose the right time to apologize and honestly express your feelings."

[1143] Prompt Sentence Examples

[1144] "I had a fight with a friend. How can I make up?"

[1145] "I got a bad score on the test today. I'm feeling down."

[1146] "I feel like I'm being bullied at school lately. What should I do?"

[1147] In this way, the system provides appropriate advice to the user's concerns and problems.

[1148] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1149] Step 1:

[1150] The user inputs voice. The user talks about their worries and problems through the microphone. A specific example of this is when the user says, "I got a bad score on the test today. I'm feeling down." The input is the user's voice data, and the output is the audio signal captured by the microphone.

[1151] Step 2:

[1152] The device sends voice data to the server. The device captures the user's voice data and sends it to the server. In concrete terms, the device uploads the voice data to the server in real time. The input is the captured voice signal, and the output is the voice data sent to the server.

[1153] Step 3:

[1154] The server performs speech recognition. Using speech recognition software, the server converts the speech data into text data. Specifically, the server uses Google Speech-to-Text to generate the text "I got a bad score on the test today. I'm feeling depressed." The input is speech data, and the output is text data.

[1155] Step 4:

[1156] The server performs emotion analysis. Using an emotion analysis engine, the server analyzes text data and understands the user's emotional state. Specifically, the server uses IBM Watson Tone Analyzer to recognize the emotion "depressed." The input is text data, and the output is emotional state data.

[1157] Step 5:

[1158] The server generates a response. Using a generative AI model, the server generates appropriate advice and solutions based on the user's emotional state. Specifically, the server uses OpenAI GPT-3 to generate a response such as, "Even if you got a bad score on the test, you'll be fine if you try harder next time. Let me know if there's anything I can help you with." The input is emotional state data and text data, and the output is the generated response text data.

[1159] Step 6:

[1160] The server converts the response into speech. The server then uses speech synthesis software to convert the generated text response into speech. Specifically, the server uses speech synthesis technology to generate speech data that says, "Even if you got a bad score on the test, you'll be fine if you try harder next time. Let me know if there's anything I can help you with." The input is the text data of the generated response, and the output is speech data.

[1161] Step 7:

[1162] The device outputs a voice response to the user. The device outputs the voice data received from the server to the user through the speaker. Specifically, the device plays a voice message from the speaker saying, "Even if you got a bad score on the test, you'll be fine if you try harder next time. Let me know if there's anything I can help you with." The input is voice data, and the output is the voice that the user can hear.

[1163] (Application example 1)

[1164] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1165] In modern society, mental health issues faced by elementary, junior high, and high school students, as well as the elderly, are serious, and it is difficult to receive appropriate support, especially when they feel anxious or scared. Security services are also required to grasp the user's psychological state in real time and respond appropriately. Furthermore, there is a lack of technology for smartphone applications that can analyze the user's emotional state and provide appropriate advice.

[1166] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes a means for providing conversation information, a means for interactively offering advice on worries and the like as mental care to elementary, junior high, and high school students and the elderly, a means for conducting conversations in real time using voice recognition and output, a means for analyzing the user's emotional state using an emotion engine, and a means for generating advice on the user's worries using a generative AI model. This makes it possible to offer real-time advice and appropriate advice and solutions when the user feels anxious or scared.

[1167] "Means for providing conversational information" refers to technologies and devices for providing information through dialogue with users.

[1168] "Means of providing interactive consultation to elementary, junior high, and high school students and the elderly as part of mental care" refers to technologies and devices that provide dialogue-based consultation to elementary, junior high, and high school students and the elderly about the worries and problems they have, and provide mental care.

[1169] "Means for real-time conversations using speech recognition and output" refers to technologies and devices that recognize speech, convert it into text, and output the generated text as speech, allowing for real-time conversations.

[1170] "Means for analyzing a user's emotional state using an emotion engine" refers to technology or devices that analyze emotions from a user's statements and actions and grasp that emotional state.

[1171] "Means for generating advice for a user's concerns using a generative AI model" refers to technology or devices for using a generative AI model to generate appropriate advice or solutions for a user's concerns or problems.

[1172] "In the context of security services, a means of providing real-time consultation and appropriate advice and solutions when users feel anxious or scared" refers to technology or devices that, as part of a security service, provide real-time consultation and appropriate advice and solutions when users feel anxious or scared.

[1173] "A means for providing advice to users on their concerns using voice recognition, emotion analysis, and generative AI models as an application installed on a smartphone" refers to technology or devices for providing advice to users on their concerns using voice recognition, emotion analysis, and generative AI models as an application installed on a smartphone.

[1174] The following system configuration is used as an embodiment of the present invention: The system is composed of three main elements: a server, a terminal, and a user.

[1175] server

[1176] The server includes the following means:

[1177] 1. Means of providing conversational information: Technologies and devices for providing information through dialogue with users.

[1178] 2. Mental care tools: These are techniques and devices that provide mental care by providing advice through dialogue on the worries and problems faced by elementary, junior high, and high school students and the elderly.

[1179] 3. Speech recognition and output means: Technology and devices that recognize speech, convert it into text, and output the generated text as speech, allowing for real-time conversations.

[1180] 4. Emotion engine: A technology or device that analyzes emotions from a user's words and actions and grasps their emotional state.

[1181] 5. Generative AI model: A technology or device that generates appropriate advice or solutions to users' concerns or problems.

[1182] Terminal

[1183] The device used is primarily a smartphone, with the following applications installed:

[1184] 1. Speech recognition application: Uses a speech recognition library (e.g. speech_recognition) to convert the user's speech into text.

[1185] 2. Sentiment analysis application: Analyze the sentiment from the user's text using a sentiment analysis model (e.g., the transformers library).

[1186] 3. Generative AI model application: Using a generative AI model (e.g., GPT-3), advice is generated to address the user's concerns.

[1187] User

[1188] The user interacts with the system through their smartphone. Specifically, the system is used in the following steps:

[1189] 1. The user talks about their worries or problems into their smartphone.

[1190] 2. The smartphone's voice recognition application converts the speech into text.

[1191] 3. The text is sent to the server and the emotion engine performs emotion analysis.

[1192] 4. Based on the sentiment analysis results, the generative AI model generates appropriate advice.

[1193] 5. The generated advice is provided to the user through a voice output means.

[1194] Specific examples

[1195] For example, if a user says, "I've been afraid to walk on the street at night lately," the emotion analysis model will recognize "fear," and the generative AI model will generate a prompt that "offers advice to the fearful user about safety measures and ways to stay calm when walking on the street at night."

[1196] Prompt Sentence Examples

[1197] The user is feeling scared. Please provide advice for the following problem: Lately, I've been afraid to walk on the street at night.

[1198] In this way, it is possible to provide appropriate advice in real time according to the user's emotional state.

[1199] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1200] Step 1:

[1201] The user speaks about their worries and problems into the smartphone. The input is the user's voice, and the output is the voice data captured through the smartphone's microphone.

[1202] Step 2:

[1203] The device's speech recognition application converts the voice data into text. Specifically, it uses a speech recognition library (e.g., speech_recognition) to analyze the voice data and generate text data. The input is voice data, and the output is text data.

[1204] Step 3:

[1205] The terminal sends the generated text data to the server. The input is the text data, and the output is the text data sent to the server.

[1206] Step 4:

[1207] The server's emotion engine analyzes the text data to understand the user's emotional state. Specifically, it analyzes the text data using an emotion analysis model (e.g., the transformers library) to identify the emotional state. The input is text data, and the output is the emotional state (e.g., fear, sadness, joy, etc.).

[1208] Step 5:

[1209] The server's generative AI model generates appropriate advice based on the emotional state and text data. Specifically, a generative AI model (e.g., GPT-3) is used to generate prompt sentences according to the emotional state, and advice is generated based on those prompt sentences. The input is the emotional state and text data, and the output is the generated advice.

[1210] Step 6:

[1211] The server sends the generated advice to the terminal. The input is the generated advice, and the output is the advice sent to the terminal.

[1212] Step 7:

[1213] The audio output means of the terminal provides the advice to the user as audio. Specifically, the text data is converted into audio data and played back through a speaker. The input is the advice as text data, and the output is the advice as audio data.

[1214] In this way, it is possible to provide appropriate advice in real time according to the user's emotional state.

[1215] Example 2

[1216] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1217] Conventional learning support systems struggle to provide appropriate answers and advice to specific problems and unclear points that learners face. Furthermore, they lack support that takes into account the learner's emotional state, which often leads to learners feeling stressed. Furthermore, limited means of providing real-time dialogue and mental care make it difficult for learners to progress effectively.

[1218] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1219] In this invention, the server includes means for providing conversation information, means for interactively offering advice to learners about their worries and other concerns as part of mental care, means for conducting conversations in real time using voice recognition and output, means for receiving problem information and unclear points from learners and generating answers and advice for them, means for analyzing the learner's emotional state using an emotion analysis engine and providing appropriate answers and advice based on the results, and means for generating answers and explanations using a generative AI model. This allows learners to receive appropriate answers and advice for specific problems and unclear points, and by receiving support tailored to their emotional state, they can effectively progress with their studies while reducing stress.

[1220] "Conversational information" refers to the content of text or voice communication exchanged between a user and a system.

[1221] "Mental care" refers to interactive support to support learners' mental health and reduce stress and distress.

[1222] "Speech recognition and output" refers to the technology that converts the user's speech into text and the technology that converts text from the system into speech and outputs it.

[1223] "Learner" refers to those receiving education, especially those who use this system to further their studies.

[1224] "Problem information" refers to the content of specific problems or tasks that learners face during their studies.

[1225] "Unclear points" are parts that learners do not understand or have questions about.

[1226] An "answer" is a correct answer or solution to a problem entered by a learner.

[1227] "Advice" is advice or guidance to help learners solve problems.

[1228] An "emotion analysis engine" is a technology that analyzes the emotional state of a user based on their input and outputs the results.

[1229] A "generative AI model" is an artificial intelligence model that performs natural language processing based on user input and generates appropriate answers and explanations.

[1230] This invention is a system that allows learners to input problem information and points of uncertainty, much like a private tutor, and provides answers and advice. Furthermore, by incorporating an emotion analysis engine, it is possible to analyze the learner's emotional state and provide appropriate support based on the results.

[1231] Hardware and software used

[1232] Hardware: Servers, devices (PCs, tablets, smartphones)

[1233] Software: Generative AI models (e.g., GPT-4), sentiment analysis engines (e.g., IBM Watson Tone Analyzer)

[1234] Program processing flow

[1235] User Input

[1236] The user uses the terminal to input any problems or uncertainties they encounter while studying into the system. For example, the user might input, "I can't solve this math problem. What should I do?"

[1237] Receiving server input

[1238] The server receives input from the user, parses the input data, and stores it.

[1239] Sending prompts to the server's generated AI model

[1240] The server sends the received input data to the generative AI model, and then sends a prompt to the generative AI model saying, "Please generate answers and explanations for the questions and unclear points entered by the user."

[1241] Generative AI model answer generation

[1242] The generative AI model generates an answer or explanation based on the prompt, for example, "This problem uses quadratic equation solving techniques. First, transform the equation into standard form."

[1243] Sending input to the server's sentiment analysis engine (in another embodiment)

[1244] If the user's input contains an emotion, the server sends the input to the emotion analysis engine, for example, "I'm frustrated because I can't solve this math problem."

[1245] Receiving the analysis results of the emotion analysis engine

[1246] The emotion analysis engine analyzes the user's emotional state and returns the results to the server. For example, it recognizes the emotional state of "irritated."

[1247] Server final answer generation

[1248] The server generates a final answer that provides appropriate support to the user based on the answers and explanations from the generative AI model and the results from the sentiment analysis engine. For example, it generates a final answer such as, "This problem uses the quadratic equation solving method. First, transform the equation into standard form. Please stay calm and try it slowly."

[1249] Server response

[1250] The server sends the generated final answer to the user's terminal.

[1251] Receiving user answers

[1252] The user receives the answer sent from the server on their device and displays it. For example, the user's device receives a response from the server and displays the answer on the screen: "This problem uses the quadratic equation solution method. First, transform the equation into standard form. Please stay calm and try this slowly."

[1253] Examples and prompts

[1254] Example 1: Answers to math problems

[1255] User input: "I can't solve this math problem. What should I do?"

[1256] The generative AI model's answer: "This problem uses quadratic equation solving techniques. First, transform the equation into standard form."

[1257] Example 2: Support including sentiment analysis

[1258] User input: "I'm frustrated because I can't solve this math problem."

[1259] Sentiment analysis engine result: "I'm frustrated"

[1260] The generative AI model's answer: "This problem uses quadratic equation solving techniques. First, transform the equation into standard form. Please stay calm and take your time."

[1261] Prompt Sentence Examples

[1262] "Generate answers and explanations for problems and uncertainties entered by the user. Also, analyze the user's emotional state and provide appropriate advice."

[1263] This system allows users to receive appropriate answers and advice for specific problems and points of uncertainty, and by receiving support tailored to their emotional state, they can study effectively while reducing stress.

[1264] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1265] Step 1:

[1266] The user uses the device to input any problems or uncertainties they encounter while studying into the system. Specifically, the user uses the device's keyboard or touchscreen to type, "I can't solve this math problem. What should I do?" The input data is sent to the server in text format.

[1267] Step 2:

[1268] The server receives input from the user. Specifically, the server receives an HTTP request, analyzes the input data, and saves it. The input data is text data such as "I can't solve this math problem. What should I do?"

[1269] Step 3:

[1270] The server sends the received input data to the generative AI model. Specifically, the server sends the generative AI model a prompt message saying, "Please generate answers and explanations for the questions and unclear points entered by the user." The input data is the user's problem information, and the output data is the answer provided by the generative AI model.

[1271] Step 4:

[1272] The generative AI model generates an answer and explanation based on the prompt. Specifically, the generative AI model uses an internal natural language processing algorithm to generate the answer, "This problem uses a quadratic equation solution. First, transform the equation into standard form." The input data is the prompt, and the output data is the answer text.

[1273] Step 5:

[1274] If the user's input contains emotion, the server sends the input to the emotion analysis engine. Specifically, the server sends the input "I'm frustrated because I can't solve this math problem." to the emotion analysis engine. The input data is text containing the user's emotion, and the output data is the emotion analysis result.

[1275] Step 6:

[1276] The emotion analysis engine analyzes the user's emotional state and returns the results to the server. Specifically, the emotion analysis engine recognizes the emotional state of "irritated" and returns the results to the server. The input data is text containing emotions, and the output data is the emotion analysis results.

[1277] Step 7:

[1278] The server generates a final answer that provides appropriate support to the user based on the answer and explanation from the generative AI model and the results from the sentiment analysis engine. Specifically, the server generates the final answer: "This problem uses the quadratic equation solution method. First, transform the equation into standard form. Please stay calm and try this slowly." The input data are the answer from the generative AI model and the sentiment analysis results, and the output data is the final answer.

[1279] Step 8:

[1280] The server sends the generated final answer to the user's device. Specifically, the server generates an HTTP response and sends it to the user's device. The input data is the final answer, and the output data is the HTTP response.

[1281] Step 9:

[1282] The user receives the answer sent from the server on their device and displays it. Specifically, the user's device receives the response from the server and displays the answer on the screen: "This problem uses the quadratic equation solution method. First, transform the equation into standard form. Please stay calm and try this slowly." The input data is the HTTP response, and the output data is the displayed answer text.

[1283] (Application example 2)

[1284] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1285] Conventional systems have had difficulty providing appropriate answers and advice in real time to problems and unclear points that workers encounter while working. Furthermore, they lacked support that took into account the emotional state of workers, and were unable to provide appropriate responses to reduce stress and anxiety. This led to concerns about reduced work efficiency and the occurrence of mistakes.

[1286] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1287] In this invention, the server includes means for providing conversation information, means for interactively offering advice on worries and the like as part of mental care to elementary, junior high, and high school students and the elderly, means for conducting conversations in real time using voice recognition and output, means for providing answers and advice to problems and unclear points that workers encounter, and means for incorporating an emotion engine to analyze the user's emotional state and provide appropriate support based on that. This makes it possible to provide appropriate answers and advice in real time to problems and unclear points that workers encounter while working, and to provide support that takes into account their emotional state.

[1288] "Conversational information" is information obtained through dialogue with the user.

[1289] "Mental care" is care to support the user's mental health.

[1290] "Speech recognition" is a technology that converts speech into text data.

[1291] "Real-time" refers to immediate processing or response without delay.

[1292] "Workers" refers to personnel who perform work in factories or on-site.

[1293] "Problem information" is information about the challenges or questions users are facing.

[1294] "Unclear points" are points that users don't understand or have questions about.

[1295] An "answer" is a response to a problem or question.

[1296] "Advice" is advice or suggestions for solving problems or making improvements.

[1297] An "emotion engine" is a technology for analyzing a user's emotional state.

[1298] "Emotional state" refers to the user's current psychological state.

[1299] "Support" refers to solving users' problems and providing emotional support.

[1300] The system for carrying out this invention consists of three main elements: a server, a terminal, and a user. The server includes a means for providing conversational information, a means for providing interactive advice on worries and other issues as part of mental care for elementary, junior high, and high school students and the elderly, a means for conducting real-time conversations using voice recognition and output, a means for providing answers and advice to problems and unclear points that workers encounter, and a means for incorporating an emotion engine to analyze the user's emotional state and providing appropriate support based on that analysis.

[1301] The server uses OpenAI's API and emotion recognition engine (EmotionRecognizer) to generate answers and advice based on the problem information and emotional state entered by the user. Specifically, when the user enters problem information, the emotion recognition engine analyzes the user's emotional state from that information. Then, based on the problem information and emotional state, the server uses the OpenAI API to generate appropriate answers and advice.

[1302] The terminal provides an interface for users to input problem information. Input devices include a keyboard and a voice input device. The information entered by the user is sent in real time to the server, where it is analyzed and a response is made.

[1303] Users input problems or questions they encounter while working into the device. For example, if they input "I'm frustrated because I'm having trouble setting up this machine," the emotion recognition engine recognizes the emotional state of "frustrated." Next, it uses the OpenAI API to generate appropriate answers and advice based on the problem information ("I'm having trouble setting up this machine") and the emotional state ("frustrated").

[1304] For example, if a user types "I'm having trouble setting up this machine and I'm frustrated," the following prompt will be sent to the generative AI model:

[1305] Problem: I'm having trouble setting up this machine and I'm frustrated.

[1306] Emotional state: Irritated

[1307] Please provide appropriate answers and advice.

[1308] Based on this prompt, the OpenAI API generates answers and advice such as, "Please recheck the machine setup procedures and follow them carefully. We also recommend taking a short break to refresh yourself."

[1309] In this way, users can receive appropriate answers and advice in real time, and can also receive support that takes their emotional state into account.

[1310] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1311] Step 1:

[1312] The user inputs the problem information and emotional state into the terminal. For example, the user might input, "I'm frustrated because the settings on this machine aren't working properly." A keyboard or voice input device is used as the input device. The input information is sent from the terminal to the server.

[1313] Step 2:

[1314] The server passes the received problem information to the emotion recognition engine. The emotion recognition engine (EmotionRecognizer) analyzes the user's emotional state from the input text. For example, it recognizes the emotional state "irritated." The input is the problem information, and the output is the emotional state.

[1315] Step 3:

[1316] The server generates a prompt based on the problem information and the user's emotional state. The generated prompt has the following format:

[1317] Problem: I'm having trouble setting up this machine and I'm frustrated.

[1318] Emotional state: Irritated

[1319] Please provide appropriate answers and advice.

[1320] The input is problem information and emotional state, and the output is a prompt sentence.

[1321] Step 4:

[1322] The server sends the generated prompt to the OpenAI API, which then generates an appropriate answer or advice based on the prompt. For example, the OpenAI API might generate an answer or advice such as, "Please recheck the machine's setup procedures and follow them carefully. We also recommend taking a short break to refresh yourself." The input is the prompt, and the output is the answer or advice.

[1323] Step 5:

[1324] The server sends the answers and advice received from the OpenAI API to the device. The device displays the received answers and advice to the user. The user can check the displayed answers and advice and use them to solve the problem. The input is the answers and advice, and the output is what is displayed to the user.

[1325] Example 3

[1326] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1327] Conventional English language education systems and mental health care systems have had difficulty fully analyzing the user's speech content and emotional state, and providing appropriate responses and feedback. Furthermore, there was a lack of technology for real-time conversation and emotional analysis, limiting the effectiveness of learning and mental health care for users. Furthermore, there was a need for flexible responses tailored to the individual needs of users.

[1328] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1329] In this invention, the server includes means for providing conversation information, means for interactively consulting with users about their worries as part of mental care, means for conducting conversations in real time using voice recognition and output, means for analyzing the user's speech and using a generative AI model to generate an appropriate response, and means for analyzing the user's emotional state and using an emotion analysis engine to provide feedback based on the analysis. This makes it possible to analyze the content of the user's speech and emotional state in real time and provide appropriate responses and feedback.

[1330] "Conversation information" refers to information about what the user says and the responses to that speech.

[1331] "Mental care" means providing interactive consultation and advice to support the mental health of users.

[1332] "Speech recognition" is a technology that captures a user's speech as voice data and converts it into text data.

[1333] A "generative AI model" is an artificial intelligence model that analyzes the content of a user's speech and generates an appropriate response.

[1334] An "emotion analysis engine" is a technology for analyzing a user's emotional state from their speech content and tone of voice.

[1335] "Feedback" is advice or guidance provided based on the user's speech or emotional state.

[1336] "Real-time" means that user utterances and responses are processed immediately without delay.

[1337] "Expandability" refers to the flexibility to add new functions and options to a system.

[1338] This invention is a system for supporting users' English education and mental care. This system operates among three parties: a server, a terminal, and a user.

[1339] The server is a computer equipped with a high-performance CPU and GPU, and operates using a generative AI model (e.g., GPT-4) and a sentiment analysis engine (e.g., IBM Watson). The terminal is a device used by the user, equipped with a microphone and an internet connection.

[1340] When a user speaks in English into the device, the device captures the voice data and sends it to the server. The server uses a generative AI model to convert the voice data into text and analyze the content of the speech. The generative AI model then generates an appropriate English response based on the analysis results. It also uses an emotion analysis engine to analyze the user's emotional state and generates feedback based on that. The server sends the generated response and feedback to the device, which then provides it to the user.

[1341] As a concrete example, consider the case where a user utters, "I find it difficult to pronounce this phrase." In this case, the device sends the voice data to the server. The server uses a generative AI model to generate and analyze the text "I find it difficult to pronounce this phrase." The generative AI model generates a response saying, "Try breaking the phrase into smaller parts and practice each part separately." The sentiment analysis engine analyzes the emotional state of "I'm having trouble" and generates feedback saying, "It seems you are having trouble. Don't worry, with practice, it will get easier." The server sends this response and feedback to the device, which then provides it to the user.

[1342] An example of a prompt is as follows:

[1343] User utterance: "I find it difficult to pronounce this phrase"

[1344] Prompt for generative AI models: "Provide a helpful response to the user's difficulty in pronouncing the phrase."

[1345] Prompt to sentiment analysis engine: "Analyze the user's emotional state based on their speech and provide appropriate feedback."

[1346] This system allows users to gain experience using English in real conversation situations, and by receiving emotion-based feedback, they can improve their English ability. Furthermore, mental health consultations and advice are provided in real time, which can support the user's mental health. The flow of the identification process in Example 3 will be described with reference to FIG. 21.

[1347] Step 1:

[1348] The user speaks in English to the terminal. For example, they say, "I find it difficult to pronounce this phrase." The terminal captures this speech with a microphone. The input is the user's speech data, and the output is the captured speech data.

[1349] Step 2:

[1350] The device sends the captured audio data to the server. Specifically, the device application uploads the audio data to the server via the Internet. The input is the captured audio data, and the output is the audio data sent to the server.

[1351] Step 3:

[1352] The server converts the received voice data into text using a generative AI model. For example, it uses speech recognition technology to generate the text "I find it difficult to pronounce this phrase." The input is voice data, and the output is the converted text data.

[1353] Step 4:

[1354] The server uses a generative AI model to analyze text data and understand the user's intent and content. For example, it analyzes the content "The pronunciation is difficult." The input is text data, and the output is the analysis result.

[1355] Step 5:

[1356] Based on the analysis results, the server uses a generative AI model to generate an appropriate English response, for example, "Try breaking the phrase into smaller parts and practice each part separately." The input is the analysis results, and the output is the generated response.

[1357] Step 6:

[1358] The server uses an emotion analysis engine to analyze the user's emotional state from their speech and tone of voice. For example, it can recognize the emotion "I'm in trouble." The input is text data and tone of voice data, and the output is the emotion analysis result.

[1359] Step 7:

[1360] The server generates feedback for the user based on the sentiment analysis results. For example, it generates feedback such as "It seems you are having trouble. Don't worry, with practice, it will get easier." The input is the sentiment analysis results, and the output is the generated feedback.

[1361] Step 8:

[1362] The server sends the generated response and feedback to the terminal. Specifically, the server's application sends the response and feedback to the terminal over the Internet. The input is the generated response and feedback, and the output is the response and feedback sent to the terminal.

[1363] Step 9:

[1364] The terminal provides the received responses and feedback to the user. Specifically, an application on the terminal displays the feedback as a text message or plays it as an audio message. The input is the response and feedback sent from the server, and the output is the response and feedback provided to the user.

[1365] (Application example 3)

[1366] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1367] Conventional English conversation training systems do not provide feedback that takes into account the user's emotional state, limiting the effectiveness of learning. Furthermore, in training English customer service in brick-and-mortar stores, real-time feedback and emotion analysis are not provided, making it difficult to improve staff's English skills.

[1368] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes means for providing conversation information, means for interactively offering advice to users about their worries and other issues as part of mental care, means for conducting real-time conversations using voice recognition and output, means for performing emotion analysis and providing feedback based on the user's emotional state, means for generating responses to the user's utterances using a generative AI model, and means including an application to be installed on a smart device. This enables real-time feedback that takes into account the user's emotional state, making it possible to effectively improve English ability and provide English customer service training in physical stores.

[1369] "Conversation information" refers to information about what the user says and the system responds to.

[1370] "Mental care" means providing interactive consultation and advice to support the mental health of users.

[1371] "Speech recognition" is a technology that analyzes a user's speech as digital data and converts it into text data.

[1372] "Real-time" refers to providing instant responses and feedback to user utterances.

[1373] "Emotion analysis" is a technology that analyzes a user's emotional state from the content of their speech and the tone of their voice.

[1374] "Feedback" refers to evaluations and advice provided in response to a user's speech or actions.

[1375] A "generative AI model" is an algorithm or system that uses artificial intelligence to generate responses to user utterances.

[1376] A "smart device" is a portable electronic device with internet connectivity and advanced computing capabilities.

[1377] An "application" is software that is installed on a smart device to provide specific functions or services.

[1378] A system for carrying out this invention has the following configuration: a server includes means for providing conversation information, means for interactively offering advice to users about their worries as part of mental care, means for conducting real-time conversations using voice recognition and output, means for performing emotion analysis and providing feedback based on the user's emotional state, means for generating responses to user utterances using a generative AI model, and means including an application to be installed on a smart device.

[1379] Hardware and Software Configuration

[1380] Hardware: Smart devices (e.g., smart glasses, smartphones)

[1381] software:

[1382] speech_recognition library: software for converting speech to text

[1383] The transformers library: software for generating responses using generative AI models (e.g., DialoGPT).

[1384] emotion_recognition library: Software for analyzing a user's emotional state

[1385] Data processing and calculation

[1386] The server receives voice input from the smart device and converts the speech to text using a speech recognition library. It then analyzes the user's emotional state using an emotion analysis library. It uses a generative AI model to generate an appropriate response to the user's utterance. It also generates and provides feedback to the user based on the emotional state.

[1387] Specific examples

[1388] Consider a scenario in which a store staff member wears smart glasses and serves customers in English. When a customer asks, "Can you help me find a specific product?", the staff member responds, "Sure, what are you looking for?" The system then analyzes the staff member's speech and provides appropriate feedback.

[1389] Example prompt sentence:

[1390] User: "Can you help me find a specific product?"

[1391] System: "Sure, what are you looking for?"

[1392] Feedback: "If you have difficulty pronouncing something, try speaking more slowly."

[1393] In this way, real-time feedback that takes into account the user's emotional state becomes possible, enabling effective improvement of English ability and English customer service training in physical stores.

[1394] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1395] Step 1:

[1396] A smart device (terminal) receives voice input. When a user speaks into the smart device, the device's microphone captures the voice. The input is the user's voice, and the output is digital voice data.

[1397] Step 2:

[1398] The device uses a speech recognition library to convert digital audio data into text data. The speech recognition library (e.g., speech_recognition) analyzes the audio data and generates corresponding text. The input is digital audio data, and the output is text data.

[1399] Step 3:

[1400] The server uses an emotion analysis library to analyze the user's emotional state from the text data. The emotion analysis library (e.g., emotion_recognition) analyzes the text data and identifies the user's emotional state (e.g., confusion, joy, etc.). The input is text data, and the output is emotional state data.

[1401] Step 4:

[1402] The server uses a generative AI model to generate a response to the user's utterance. The generative AI model (e.g., DialoGPT from the transformers library) takes text data as input and generates an appropriate response text. The input is the user's utterance text, and the output is the response text.

[1403] Step 5:

[1404] The server generates feedback for the user based on the emotional state data. It analyzes the emotional state data and generates feedback such as "If the pronunciation is difficult, please try speaking more slowly" if the user is confused. The input is the emotional state data and the output is the feedback text.

[1405] Step 6:

[1406] The terminal provides the generated response text and feedback text to the user. The terminal's display or audio output function is used to display or audio-output the response and feedback to the user. The input is the response text and feedback text, and the output is the display or audio output to the user.

[1407] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1408] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1409] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1410] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1411] [Third embodiment]

[1412] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1413] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1414] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1415] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1416] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1417] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1418] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1419] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1420] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1421] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1422] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1423] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1424] "Example 1"

[1425] One embodiment of the present invention is a system that provides interactive advice to elementary, junior high, and high school students and the elderly about their concerns, etc., as a means of providing conversational information. This system receives input information from the user and generates a response in response. Specifically, it provides advice and solutions to the concerns and problems the user is facing. It is also possible to have real-time conversations using voice recognition and output. This allows the user to talk about their concerns in their own words, and the system then provides appropriate advice.

[1426] "Example 2"

[1427] Furthermore, another embodiment of the present invention is a system that provides answers and advice to test-takers, much like a private tutor, by inputting problem information and unclear points. This system receives problem information and unclear points from the user and generates answers and advice accordingly. Specifically, it provides answers and explanations for problems and unclear points that the user encounters while studying. This allows the user to study at their own pace and receive the necessary support when needed.

[1428] "Example 3"

[1429] A further embodiment of the present invention is a system for conducting English conversation and providing English language education. The system receives English utterances from a user and generates English responses in response to the utterances. Specifically, the user conducts a conversation in English, and the system provides responses in English in response. This allows the user to gain experience using English in a real conversational situation. Furthermore, the system provides feedback on the user's utterances to help improve the user's English ability.

[1430] The processing flow of each embodiment will be described below.

[1431] "Example 1"

[1432] Step 1: The user speaks their concerns or problems into the system.

[1433] Step 2: The system uses voice recognition technology to convert the user's input into text information.

[1434] Step 3: The system analyzes the text information and generates appropriate advice or solutions. Step 4: The system outputs the generated advice or solutions to the user aloud.

[1435] "Example 2"

[1436] Step 1: The test-taker enters the problem information and any questions they have about the problem into the system.

[1437] Step 2: The system analyzes the entered problem information and unclear points and generates appropriate answers and explanations.

[1438] Step 3: Output the generated answers and explanations to the test-taker.

[1439] "Example 3"

[1440] Step 1: The user speaks to the system in English.

[1441] Step 2: The system analyzes the user's utterance and generates an appropriate English response.

[1442] Step 3: Output the generated English response to the user.

[1443] Step 4: The system provides feedback on the user's speech to help improve their English skills.

[1444] Example 1

[1445] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1446] In modern society, elementary, junior high, and high school students, as well as the elderly, have a wide range of worries and problems, and there is a need for appropriate mental care for them. However, with conventional systems, it is difficult to provide consultation using real-time dialogue or voice recognition and output, making it difficult for users to talk about their worries in their own words and receive appropriate advice. In addition, there is a lack of support, such as private tutoring for students taking exams, or English conversation education. A system that can solve these issues is needed.

[1447] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1448] In this invention, the server includes means for acquiring user input information, means for converting voice data into text data using voice recognition technology, means for generating a response based on the text data using a generative AI model, means for converting the generated response into voice data using voice output technology and providing it, means for providing interactive advice on worries and other issues as mental care to elementary, junior high, and high school students and the elderly, and means for conducting real-time conversations. This allows users to talk about their worries in their own words and receive appropriate advice in real time. It can also provide support like a private tutor for students taking exams and English conversation lessons.

[1449] The "means for acquiring user input information" refers to a device or software that provides an interface for users to input information to the system in the form of voice, text, or the like.

[1450] "Speech recognition technology" is a technology that converts a user's voice into a digital signal, analyzes that signal, and converts it into text data.

[1451] A "generative AI model" is an artificial intelligence algorithm that generates appropriate responses based on input text data.

[1452] "Voice output technology" is a technology that converts text data into voice data and provides it to the user as voice.

[1453] "A means of providing interactive consultation on worries and problems as part of mental care" is a system that receives consultations in an interactive format about the worries and problems that users are facing, and provides appropriate advice and solutions.

[1454] "Means for real-time conversation" refers to technologies and devices that allow users and systems to exchange information instantly and engage in dialogue without delay.

[1455] "Tutor-like support for test takers" is a system that provides appropriate answers and advice to test takers' learning problems and questions.

[1456] "Providing English conversation education" is a system that provides educational support to help users practice English conversation and improve their English skills.

[1457] "Extensibility to add options" refers to a design that is flexible enough to allow new functions and services to be added to the system.

[1458] MODE FOR CARRYING OUT THE INVENTION

[1459] This invention is a system that provides interactive advice to elementary, junior high, and high school students and the elderly about their concerns, etc. It acquires user input information and generates responses to that information. Specifically, it can conduct real-time conversations using voice recognition technology, a generative AI model, and voice output technology.

[1460] Hardware and software used

[1461] The system uses the following hardware and software:

[1462] Device: The device used by the user, such as a smartphone, tablet, or computer

[1463] Speech recognition technology: Speech recognition software such as Google Cloud Speech-to-Text

[1464] Generative AI models: Generative AI models such as OpenAI's GPT-4

[1465] Speech output technology: Text-to-speech software such as Amazon Polly

[1466] Server: A server for processing data and running generative AI models

[1467] System Operation

[1468] The user speaks to the system using the device. For example, an elementary school student might ask, "I had a fight with a friend. How can I make up?" This voice input is captured through the device's microphone.

[1469] The device uses Google Cloud Speech-to-Text to convert the voice data into text data, which contains the user's concerns and problems.

[1470] The server receives the text data sent from the device and generates a prompt sentence to be input to the generative AI model (GPT-4). The prompt sentence has the following format:

[1471] "A primary school student is asking for advice about a fight he had with a friend. Can you offer some advice on how to make up?"

[1472] The generative AI model generates appropriate advice based on this prompt, such as, "First, it's important to apologize to your friend. Then, it's important to talk about the cause of the argument and come to a mutual understanding."

[1473] The server sends the generated response to the device, which then uses Amazon Polly to convert the response into audio data and provides it to the user, who can receive the system's response through the audio played from the device.

[1474] This system allows users to talk about their concerns in their own words and receive appropriate advice in real time. It can also provide support like a private tutor for students taking exams, or English conversation lessons.

[1475] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1476] Step 1:

[1477] The user speaks into the device's microphone to describe their worries or problems. For example, they might say, "I had a fight with a friend. How can we make up?" The input is captured in the device as voice data.

[1478] Step 2:

[1479] The device uses speech recognition technology to convert the voice data into text data. Specifically, it uses Google Cloud Speech-to-Text to analyze the voice data and generate text data such as "I had a fight with my friend. How can I make up?" The input is voice data, and the output is text data.

[1480] Step 3:

[1481] The device sends the generated text data to the server. The server receives this text data and generates a prompt to be input into the generative AI model. For example, the prompt might read, "An elementary school student has had a fight with a friend and is asking for advice. Please provide some advice on how to make up." The input is text data, and the output is a prompt.

[1482] Step 4:

[1483] The server inputs a prompt into a generative AI model (GPT-4) and generates a response to the user's concerns or problems. The generative AI model generates text data containing appropriate advice based on the prompt. For example, it generates a response such as, "First, it's important to apologize to your friend. After that, it's important to discuss the cause of the fight and come to a mutual understanding." The input is the prompt, and the output is the response text.

[1484] Step 5:

[1485] The server sends the generated response text to the device. The device then uses voice output technology to convert the response text into voice data. Specifically, Amazon Polly is used to convert the text data into voice data, generating voice data such as, "First, it's important to apologize to your friend. Then, it's important to talk about the cause of the argument and come to a mutual understanding." The input is the response text, and the output is voice data.

[1486] Step 6:

[1487] The terminal plays the generated voice data and provides it to the user. The user receives the system's response through the voice played from the terminal. This allows the user to receive appropriate advice in real time. The input is voice data, and the output is a voice response to the user.

[1488] (Application example 1)

[1489] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1490] In modern society, mental health issues faced by elementary, junior high, and high school students, as well as the elderly, are serious, and appropriate support is needed. However, conventional mental care systems have difficulty providing appropriate advice to users in real time regarding their concerns, and few systems have advanced dialogue functions that utilize voice recognition or generative AI models. For this reason, there is a need for a system that allows users to talk about their concerns in their own words and receive appropriate advice in real time.

[1491] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1492] In this invention, the server includes means for providing conversation information, means for interactively offering advice on worries and the like as part of mental care to elementary, junior high, and high school students and the elderly, means for conducting conversations in real time using voice recognition and output, means for providing advice and solutions to users' worries using a generative AI model, means for generating prompt sentences and generating appropriate responses, and means for providing feedback in real time using a voice output function. This enables users to talk about their worries in their own words and receive appropriate advice in response to them in real time.

[1493] "Means for providing conversational information" is a function for collecting information through dialogue with the user and generating an appropriate response.

[1494] "A means of providing interactive consultation to elementary, junior high, and high school students and the elderly regarding their worries and problems as part of mental care" is a function that allows elementary, junior high, and high school students and the elderly to discuss their worries and problems in an interactive format and provide appropriate advice and support.

[1495] "Means for conducting real-time conversations using voice recognition and output" refers to a function that enables real-time conversations by recognizing the user's voice, converting it into text, and outputting the generated response as voice.

[1496] "Means of providing advice and solutions to users' concerns using generative AI models" refers to a function that uses generative AI models to generate appropriate advice and solutions to users' concerns.

[1497] The "means for generating a prompt sentence and generating an appropriate response" is a function for generating a prompt sentence based on a user's input and generating an appropriate response based on the prompt sentence.

[1498] The "means for providing feedback in real time using a voice output function" is a function for outputting the generated response by voice and providing feedback to the user in real time.

[1499] A system for carrying out this invention is configured as follows: The server includes means for providing conversation information, means for interactively offering advice on worries and the like as part of mental care to elementary, junior high, and high school students and the elderly, means for conducting conversations in real time using voice recognition and output, means for providing advice and solutions to users' worries using a generative AI model, means for generating prompt sentences to generate appropriate responses, and means for providing feedback in real time using a voice output function.

[1500] Program processing explanation

[1501] The server uses speech recognition software (e.g., the speech_recognition library) to convert the user's voice into text. What the user says through the microphone is sent as voice data to the server, which then converts this voice data into text data.

[1502] Next, a generative AI model (e.g., OpenAI's API) is used to generate appropriate advice or solutions based on the text data. The generative AI model generates prompts based on the user's input and then generates responses based on those prompts.

[1503] The generated response is converted into voice data using voice output software (e.g., gTTS library) and fed back to the user in real time. The voice output is provided to the user through a speaker.

[1504] Specific examples

[1505] For example, if a user says, "I'm having trouble making friends at school," the server converts this speech to text and generates the following prompt:

[1506] Example prompt sentence:

[1507] User's problem: I'm having trouble making friends at school

[1508] Please provide advice.

[1509] The generative AI model generates appropriate advice based on the prompt, and then converts the advice into voice data using voice output software, allowing the user to receive the advice in real time through a speaker.

[1510] In this way, users can talk about their concerns in their own words and receive appropriate advice in real time.

[1511] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1512] Step 1:

[1513] The user speaks about their concerns through a microphone. The device collects this voice data and sends it to the server. The input is the user's voice data, and the output is the voice data sent to the server.

[1514] Step 2:

[1515] The server uses speech recognition software (e.g., the speech_recognition library) to convert the received voice data into text data. The input is voice data, and the output is text data. Specifically, the speech recognition software analyzes the voice data and generates the corresponding text.

[1516] Step 3:

[1517] The server uses a generative AI model (e.g., OpenAI's API) to generate a prompt based on the text data. The input is the user's text data, and the output is the prompt. Specifically, the generative AI model analyzes the text data and generates an appropriate prompt.

[1518] Step 4:

[1519] The server uses the generated prompt sentence to have the generative AI model generate appropriate advice or solutions. The input is the prompt sentence, and the output is the generated advice or solution. In concrete terms, the generative AI model generates a response based on the prompt sentence.

[1520] Step 5:

[1521] The server converts the generated advice or solution into voice data using voice output software (e.g., gTTS library). The input is the generated text data, and the output is voice data. Specifically, the voice output software converts the text data into voice data.

[1522] Step 6:

[1523] The server transmits the voice data to the terminal, and the terminal provides real-time feedback to the user through the speaker. The input is the voice data, and the output is the voice feedback provided to the user. In concrete terms, the terminal plays the voice data and provides feedback to the user.

[1524] Example 2

[1525] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1526] Conventional learning support systems have had the problem of being unable to provide immediate, appropriate answers or advice to specific problems or unclear points that learners encounter during their studies. Another problem is that learners cannot receive the support they need when they need it, as they progress through their studies at their own pace. Furthermore, there is a lack of means for providing mental care or interactive consultations, and comprehensive support for learners is insufficient.

[1527] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1528] In this invention, the server includes means for providing conversation information, means for interactively offering advice to learners about their worries and other concerns as part of mental care, means for conducting conversations in real time using voice recognition and output, means for receiving problem information and unclear points entered by the learner and generating answers and advice for them, means for generating answers and advice using a generative AI model, and means for providing the generated answers and advice to the learner. This allows learners to progress through their studies at their own pace and receive the support they need when they need it.

[1529] "Conversational information" refers to the content of text and voice communication exchanged between a user and a system.

[1530] "Mental care" refers to interactive support to reduce learners' psychological worries and stress and support their mental health.

[1531] "Speech recognition" is a technology that converts speech into text data.

[1532] "Audio output" is a technology that reproduces text data as audio.

[1533] "Real-time" refers to immediate processing or response without delay.

[1534] "Learner" refers to an individual engaged in an education or learning activity.

[1535] "Problem information" refers to the content of specific problems or issues that a learner encounters during learning.

[1536] "Unclear points" refer to parts that learners do not understand or have questions about.

[1537] An "answer" is a correct answer or solution to a problem submitted by a learner.

[1538] "Advice" is advice or guidance to help learners solve problems.

[1539] A "generative AI model" is an algorithm or system that uses artificial intelligence to generate text or speech.

[1540] A "prompt" is text that contains instructions or questions that are input to a generative AI model.

[1541] The present invention is a system that instantly provides appropriate answers and advice to specific problems and unclear points that a learner encounters during learning. A specific embodiment of this system will be described below.

[1542] Hardware and software used

[1543] 1. Terminal

[1544] These are devices such as computers and smartphones used by learners. These devices are connected to the Internet and are used to input problem information and any questions through a user interface.

[1545] 2. Server

[1546] A computer system for processing data and running a generative AI model. The server receives data sent by users and inputs it into the generative AI model.

[1547] 3. Generative AI Models

[1548] For example, advanced natural language processing models such as OpenAI's GPT-4 generate appropriate answers and advice based on the problem information and questions users provide.

[1549] Data processing and calculation

[1550] 1. User Input

[1551] The user uses the terminal to input information about the problems and points of confusion they encounter while studying, for example, "I don't know how to solve differential equations in mathematics."

[1552] 2. Data transmission

[1553] The device sends the problem information and uncertainties entered by the user to the server, which converts the received data into an appropriate format and inputs it into the generative AI model.

[1554] 3. Use of generative AI models

[1555] The server inputs data to the generative AI model as a prompt sentence. For example, it sends the prompt sentence "Please tell me how to solve a differential equation" to the generative AI model.

[1556] 4. Generating Answers and Advice

[1557] The generative AI model generates appropriate answers and advice based on the input prompt. For example, it might generate an answer like, "To solve a differential equation, first transform the equation into standard form, then solve the characteristic equation."

[1558] 5. User Feedback

[1559] The server sends the generated answers and advice to the terminal. The terminal displays the received answers and advice to the user. For example, the user's screen may display a message such as, "Here's how to solve a differential equation: First, transform the equation into standard form, and then solve the characteristic equation."

[1560] Specific examples

[1561] Example user input

[1562] "I don't know how to graph a linear function."

[1563] Example prompts for generative AI models

[1564] "How do I draw the graph of a linear function?"

[1565] Example of a generated answer

[1566] "To graph a linear function, first determine the shape of the function. For example, in y = 2x + 3, the y-intercept is 3 and the slope is 2. Next, plot the y-intercept and draw a line according to the slope."

[1567] This way, users can learn at their own pace and get the support they need when they need it.

[1568] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1569] Step 1:

[1570] Users use the terminal to input information about problems and questions they encounter while studying.

[1571] Specifically, the user opens a dedicated application or web browser on their device and enters the problem information and any points of uncertainty into the input form. For example, they might enter, "I don't know how to solve a differential equation in mathematics."

[1572] Input: User-entered problem information and questions.

[1573] Output: Problem information and unknown data entered into the terminal.

[1574] Step 2:

[1575] The terminal sends the problem information and unclear points entered by the user to the server.

[1576] Specifically, the terminal transmits the user's input data to the server via the Internet.

[1577] Input: Problem information and question data entered into the terminal.

[1578] Output: Problem information and unknown data sent to the server.

[1579] Step 3:

[1580] The server converts the received problem information and uncertainties into a suitable format for input into the generative AI model.

[1581] Specifically, the server converts the data into a prompt, for example, "Please tell me how to solve a differential equation."

[1582] Input: Problem information and unknown data sent to the server.

[1583] Output: A prompt to be fed into the generative AI model.

[1584] Step 4:

[1585] The server inputs a prompt sentence into the generative AI model.

[1586] Specifically, the server sends a prompt to the generative AI model and waits for the model to generate an answer or advice.

[1587] Input: A prompt to be used as input to the generative AI model.

[1588] Output: The answer or advice returned by the generative AI model.

[1589] Step 5:

[1590] The generative AI model generates answers and advice based on the input prompt.

[1591] Specifically, the generative AI model uses natural language processing technology to generate appropriate answers and advice. For example, it might generate an answer like, "Here's how to solve a differential equation: First, transform the equation into standard form, and then solve the characteristic equation."

[1592] Input: The prompt text fed into the generative AI model.

[1593] Output: The generated answer or advice.

[1594] Step 6:

[1595] The server sends the generated answers and advice to the terminal.

[1596] Specifically, the server sends the answers and advice received from the generative AI model to the user's device.

[1597] Input: Generated answers and advice.

[1598] Output: Answers and advice sent to your device.

[1599] Step 7:

[1600] The device displays the answers and advice received to the user.

[1601] Specifically, the device displays the answer or advice on the user's screen. For example, the device displays a message on the user's screen saying, "Here's how to solve a differential equation: First, transform the equation into standard form, then solve the characteristic equation."

[1602] Input: Answers and advice sent to the device.

[1603] Output: The answer or advice displayed to the user.

[1604] (Application example 2)

[1605] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1606] Conventional learning support systems have difficulty providing immediate answers and advice to problems and unclear points that students encounter while studying. Furthermore, they lack the scalability to accommodate diverse needs, such as mental health care and English education. This makes it difficult for users to progress through their studies at their own pace, and they are unable to receive the support they need when they need it.

[1607] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for providing conversation information, means for interactively offering advice on worries and the like as part of mental care to elementary, junior high, and high school students and the elderly, means for conducting real-time conversations using voice recognition and output, means for inputting problem information and unclear points from the user and generating answers and advice, means for providing answers and advice using a generative AI model, and means including an application to be installed on a smartphone. This allows users to study at their own pace and receive the necessary support when they need it.

[1608] "Conversational information" is information provided through dialogue with the user.

[1609] "Mental care" is care to support the user's mental health.

[1610] "Speech recognition" is a technology that converts speech into text data.

[1611] "Audio output" is a technology that reproduces text data as audio.

[1612] "Real-time" refers to processing occurring immediately without delay.

[1613] "Problem information" refers to the specific problems or questions that users encounter while studying.

[1614] "Unclear points" are parts that users do not understand or have questions about.

[1615] An "answer" is the correct answer to a question.

[1616] "Advice" is advice on how to solve problems or proceed with learning.

[1617] A "generative AI model" is a model that uses artificial intelligence to generate answers and advice from data.

[1618] A "smartphone" is a type of mobile phone, a device that can connect to the Internet and use applications.

[1619] An "application" is software that provides specific functions or services.

[1620] To implement the present invention, the following system configuration and program are required.

[1621] System Configuration

[1622] 1. Hardware:

[1623] Smartphone: A device where users can enter problem information or questions.

[1624] Server: A computer system that runs generative AI models and generates answers and advice.

[1625] 2. Software:

[1626] Python: A programming language used to implement the program.

[1627] OpenAI API: An API for using generative AI models.

[1628] Program processing

[1629] 1. User input:

[1630] Users enter any problems or questions they encounter while studying in text format through a smartphone application.

[1631] 2. Data transmission:

[1632] The smartphone sends the entered problem information and any unclear points to the server.

[1633] 3. Run the generative AI model:

[1634] The server uses the OpenAI API to run the generative AI model and generate answers and advice for the input problem information and unknown points.

[1635] 4. Return of answers and advice:

[1636] The server sends the generated answers and advice back to the smartphone and displays them to the user.

[1637] Specific examples

[1638] For example, if a user types, "Please tell me the formula for solving a quadratic equation," the following prompt sentence is sent to the generative AI model:

[1639] Example prompt sentence:

[1640] A candidate is asking the following question: "What is the formula for solving a quadratic equation?"

[1641] Please provide answers and advice.

[1642] The generative AI model generates answers and advice based on this prompt, returning the following results:

[1643] Answers and advice:

[1644] The formula for solving the quadratic equation ax^2 + bx + c = 0 is:

[1645] x = (-b ± √(b^2 - 4ac)) / 2a

[1646] When using this formula, it is important to substitute the values ​​of a, b, and c correctly. Also, note that the number of solutions changes depending on the value of the discriminant (b^2 - 4ac): if the discriminant is positive, you get two real solutions, if it is zero, you get one real solution, and if it is negative, you get two imaginary solutions.

[1647] This way, users can learn at their own pace and get the support they need when they need it.

[1648] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1649] Step 1:

[1650] The user starts the smartphone application and inputs any problems or questions they encounter while studying in text format. The input text is temporarily saved in the application.

[1651] Input: Text data of the problem information or questions entered by the user

[1652] Output: Temporarily saved text data

[1653] Step 2:

[1654] The device (smartphone) sends the entered text data to the server via an internet connection.

[1655] Input: Temporarily saved text data

[1656] Output: Text data sent to the server

[1657] Step 3:

[1658] The server analyzes the received text data and generates a prompt suitable for the generative AI model, which is then formatted to clarify the user's question.

[1659] Input: Text data sent to the server

[1660] Output: Generated prompt statement

[1661] Step 4:

[1662] The server sends the generated prompt to the OpenAI API and executes the generative AI model, which generates answers and advice based on the prompt.

[1663] Input: Generated prompt text

[1664] Output: The answer or advice generated by the generative AI model

[1665] Step 5:

[1666] The server receives the answers and advice returned by the generative AI model, formats them in a user-friendly format, and then sends the formatted answers and advice back to the smartphone.

[1667] Input: Answers and advice generated by a generative AI model

[1668] Output: Formatted answers and advice

[1669] Step 6:

[1670] The terminal (smartphone) displays the answers and advice received from the server to the user, allowing the user to obtain answers and advice for problems and unclear points during study.

[1671] Input: Formatted answers and advice

[1672] Output: The answer or advice displayed to the user

[1673] Example 3

[1674] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1675] Conventional English language education systems and mental health care systems have difficulty providing real-time dialogue and feedback, and have been unable to quickly provide appropriate responses or advice to users' utterances. It has also been difficult to accurately understand the content of users' utterances and generate natural responses that fit the context. This has led to issues such as insufficient improvement in users' English ability and in the effectiveness of mental health care.

[1676] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1677] In this invention, the server includes means for providing conversation information, means for interactively consulting with a user about worries and the like as part of mental care, means for conducting conversations in real time using voice recognition and output, means for receiving the user's speech as voice data and converting it into text data using voice recognition software, means for inputting prompt sentences into a generative AI model and generating an appropriate response, means for converting the generated response into voice data and providing it to the user, and means for generating feedback on the user's speech and providing it to the user. This enables users to improve their English ability and receive the benefits of mental care while engaging in natural conversations in real time.

[1678] "Conversation information" is information about the dialogue that users have through the system.

[1679] "Mental care" means providing interactive consultation and advice to support the mental health of users.

[1680] "Speech recognition" is a technology that converts voice data into text data.

[1681] "Voice output" is a technology that converts text data into voice data and provides it to the user as voice.

[1682] "Real-time" refers to providing immediate responses and feedback to user utterances.

[1683] "Voice data" refers to data that is a digital recording of a user's speech.

[1684] "Text data" is character information converted from voice data using voice recognition technology.

[1685] A "generative AI model" is an artificial intelligence model that understands the content of a user's speech and generates natural responses according to the context.

[1686] A "prompt sentence" is text containing contextual information that is input into a generative AI model.

[1687] "Feedback" refers to providing evaluations and advice on the user's speech.

[1688] The present invention is a system that allows a user to make an utterance in English, generates an appropriate response to the utterance, and provides feedback. A specific embodiment of this system will be described below.

[1689] First, a user accesses the system using a device (e.g., a smartphone or PC). When the user speaks in English, the device captures the speech as voice data. This voice data is then sent to a server via the Internet.

[1690] The server converts the received voice data into text data using speech recognition software (e.g., a speech recognition API). The speech recognition software analyzes the voice data and generates corresponding text data. This text data is used as part of a prompt sentence that is input into the generative AI model.

[1691] Next, the server inputs a prompt sentence to the generative AI model (e.g., a natural language processing model), which looks like this:

[1692] User: What is the weather like today?

[1693] system:

[1694] The generative AI model generates an appropriate response based on this prompt sentence. For example, it might generate a response such as, "The weather today is sunny with a high of 25 degrees Celsius." This response is then converted into voice data by the server using speech synthesis software (e.g., a speech synthesis API).

[1695] The generated voice data is then sent back to the terminal via the Internet, where it is played back and a response in English is provided to the user, allowing the user to have a natural conversation in real time.

[1696] Additionally, the server generates feedback on the user's speech, assessing things like pronunciation accuracy and grammar, and including suggestions to help the user improve their English proficiency. For example, feedback might be generated such as, "Your pronunciation was clear, but you could improve your intonation on the word 'weather'."

[1697] The generated feedback is then sent back to the terminal via the Internet, where it is displayed to the user, who can then receive specific advice on their own speech and improve their English ability.

[1698] This system combines technologies such as speech recognition software, generative AI models, and speech synthesis software to provide users with natural dialogue and feedback in real time. This allows users to gain experience using English in real conversation situations, thereby improving their English ability. The flow of the identification process in Example 3 will be described with reference to Figure 15.

[1699] Step 1:

[1700] The user speaks in English.

[1701] The user asks a question or speaks to the system in English, for example, "What is the weather like today?" The input is the user's voice, and the output is the voice data captured on the device.

[1702] Step 2:

[1703] The terminal transmits the voice data to the server.

[1704] The device (such as a smartphone or PC) captures the user's speech as voice data and sends this voice data to a server via the Internet. The input is the user's voice data, and the output is the voice data sent to the server.

[1705] Step 3:

[1706] The server converts the voice data into text data.

[1707] The server converts the received voice data into text data using voice recognition software (e.g., a voice recognition API). The voice recognition software analyzes the voice data and generates corresponding text data. The input is voice data, and the output is text data.

[1708] Step 4:

[1709] The server inputs a prompt sentence into the generative AI model and generates a response.

[1710] The server inputs a prompt sentence to the generative AI model (e.g., a natural language processing model). The prompt sentence looks like this:

[1711] User: What is the weather like today?

[1712] system:

[1713] The generative AI model generates an appropriate response based on this prompt. For example, it generates a response such as, "The weather today is sunny with a high of 25 degrees Celsius." The input is the text data and the prompt, and the output is the generated response text.

[1714] Step 5:

[1715] The server converts the generated response into audio data.

[1716] The server converts the generated response into audio data using speech synthesis software (e.g., a speech synthesis API), where the input is the generated response text and the output is the audio data.

[1717] Step 6:

[1718] The server sends the audio data to the terminal.

[1719] The server transmits the generated voice data to the terminal via the Internet. The input is the voice data, and the output is the voice data transmitted to the terminal.

[1720] Step 7:

[1721] The terminal plays the response to the user.

[1722] The terminal plays the received voice data and provides a response in English to the user. The input is the voice data and the output is the voice played to the user.

[1723] Step 8:

[1724] The server generates feedback in response to the user's utterance.

[1725] The server generates feedback on the user's speech, evaluating the accuracy of pronunciation and grammar, and providing advice to help improve the user's English skills. The input is the user's speech text, and the output is the feedback text.

[1726] Step 9:

[1727] The server sends the feedback to the device.

[1728] The server transmits the generated feedback to the terminal via the Internet. The input is the feedback text, and the output is the feedback transmitted to the terminal.

[1729] Step 10:

[1730] The device displays the feedback to the user.

[1731] The device displays the received feedback to the user, for example, advice such as "Your pronunciation was clear, but you could improve your intonation on the word 'weather'." The input is the feedback text, and the output is the feedback displayed to the user.

[1732] (Application example 3)

[1733] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1734] Conventional English education systems and mental health care systems make it difficult for users to gain experience using English in real conversational situations, and do not provide real-time feedback, limiting the improvement of English proficiency. Furthermore, virtual stores also lack real-time responses when users ask questions about products or make purchases in English, resulting in an inadequate user experience. To address these issues, a system that provides real-time English responses and feedback is needed.

[1735] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1736] In this invention, the server includes a means for providing conversational information, a means for interactively providing advice to elementary, junior high, and high school students and the elderly about their worries as part of mental care, a means for conducting conversations in real time using voice recognition and output, a means for providing real-time English responses when a user asks a question about a product in English or makes a purchase in a virtual store, and a means for providing feedback on the user's speech to help improve their English ability. This allows users to gain experience using English in actual conversational situations and improve their English ability through real-time feedback. This also improves the user experience in the virtual store.

[1737] "Conversation information" refers to information about what the user says and the system's response.

[1738] "Mental care" refers to supporting mental health by alleviating users' psychological worries and stress through dialogue.

[1739] "Speech recognition" is a technology that collects a user's speech through a microphone and converts it into text data.

[1740] "Voice output" is a technology that converts text data into voice and provides it to the user through a speaker.

[1741] "Real-time" refers to providing instant responses and feedback to user utterances.

[1742] A "virtual store" is a virtual store that exists on the Internet, where users can browse and purchase products online.

[1743] "Feedback" is information that provides evaluation and advice on a user's speech or behavior, encouraging improvement.

[1744] "English proficiency" refers collectively to a user's ability to understand, speak, and write English.

[1745] "User experience" refers to the overall satisfaction and convenience that users feel when using a system or service.

[1746] A system for carrying out this invention has the following configuration: The server includes means for providing conversation information, means for interactively offering advice on worries and the like as part of mental care for elementary, junior high, and high school students and the elderly, means for conducting conversations in real time using voice recognition and output, means for providing real-time English responses when a user asks a question about a product in English or goes through the purchasing process in a virtual store, and means for providing feedback on the user's speech and supporting the improvement of English ability.

[1747] Hardware and software used

[1748] Hardware: smartphone, head-mounted display, microphone, speaker

[1749] Software: OpenAI API (generative AI model), SpeechRecognition (voice recognition), pyttsx3 (voice synthesis)

[1750] Data processing and calculation

[1751] The server first collects the user's speech through a microphone and converts the speech data into text using the SpeechRecognition library. It then uses the OpenAI API to generate an appropriate English response to the user's speech. This response is generated based on the prompt sentence using a generative AI model. The generated response is then converted into audio using the pyttsx3 library and provided to the user through the speaker.

[1752] Additionally, feedback is generated based on the user's speech and the generated response, providing feedback in the form of audio or text to help improve the user's English proficiency.

[1753] Specific examples

[1754] For example, if a user asks in English, "Tell me about this product," in a virtual store, the server responds, "Sure, this product is a high-quality, eco-friendly item made from sustainable materials. Would you like to know more details?" It also provides feedback, "Your pronunciation was good, but try to use more complex sentences."

[1755] Prompt Sentence Examples

[1756] User utterance: "Can you tell me about this product?"

[1757] Prompt to generative AI model: "The user asked about a product. Provide a detailed description in English."

[1758] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1759] Step 1:

[1760] The user speaks into the microphone in English.

[1761] Input: User's voice data

[1762] Output: Audio data collected by microphone

[1763] Step 2:

[1764] The device uses speech recognition software (SpeechRecognition) to convert the collected voice data into text data.

[1765] Input: Audio data

[1766] Output: Text data

[1767] Specific operation: The device launches voice recognition software, analyzes the voice data, and generates corresponding text data.

[1768] Step 3:

[1769] The server uses a generative AI model (OpenAI API) to generate appropriate English responses based on the text data.

[1770] Input: User's text data

[1771] Output: Generated English response

[1772] Specific operation: The server sends the text data as a prompt to the generative AI model, which then generates a response.

[1773] Step 4:

[1774] The server converts the generated English response into voice data using speech synthesis software (pyttsx3).

[1775] Input: Generated English responses (text data)

[1776] Output: Audio data

[1777] Specific operation: The server starts the speech synthesis software and converts the text data into speech data.

[1778] Step 5:

[1779] The terminal plays the generated audio data to the user through a speaker.

[1780] Input: Audio data

[1781] Output: The audio the user hears

[1782] Specific operation: The device turns on the speaker and plays audio data.

[1783] Step 6:

[1784] The server generates feedback for the user's speech and provides it in the form of voice or text.

[1785] Input: User utterance text data, generated English responses

[1786] Output: Feedback (audio or text)

[1787] What it does: The server analyzes the user's speech and the generated responses, and generates an evaluation and advice. The generated feedback is converted into audio data using speech synthesis software and played through the speaker or displayed as text.

[1788] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1789] "Example 1"

[1790] One embodiment of the present invention is a conversation service system incorporating an emotion engine. This system receives a user's voice input and converts it into text information using speech recognition technology. The emotion engine then analyzes this text information to understand the user's emotional state. For example, if a user says, "I got a bad score on the test today. I'm feeling depressed," the emotion engine recognizes the emotional state as "depressed." Based on this emotional state, the system then generates appropriate advice or solutions and outputs them in voice.

[1791] "Example 2"

[1792] Another embodiment is a tutoring service system incorporating an emotion engine. In this system, when a test-taker inputs problem information or points of uncertainty during study, the emotion engine analyzes the user's emotional state. For example, if the user says, "I can't solve this math problem and I'm frustrated," the emotion engine recognizes the user's emotional state as "frustrated." Based on this emotional state, the system generates appropriate answers and explanations, and even provides calming advice.

[1793] "Example 3"

[1794] Yet another embodiment is an English conversation education service system incorporating an emotion engine. In this system, when a user speaks in English, the emotion engine analyzes the user's emotional state. For example, if the user says, "I'm having trouble pronouncing this English phrase," the emotion engine recognizes the user's emotional state as "trouble." Based on this emotional state, the system then generates appropriate pronunciation guidance and advice to support the user's English learning.

[1795] The processing flow of each embodiment will be described below.

[1796] "Example 1"

[1797] Step 1: Receive voice input from the user.

[1798] Step 2: Use speech recognition technology to convert the voice input into text information.

[1799] Step 3: The emotion engine analyzes the text information and understands the user's emotional state.

[1800] Step 4: Based on the emotional state, the system generates appropriate advice and solutions.

[1801] Step 5: Output the generated advice or solution aloud.

[1802] "Example 2"

[1803] Step 1: The test-taker enters the problem information and any questions they have about the test into the system.

[1804] Step 2: The emotion engine analyzes the user's emotional state.

[1805] Step 3: Based on the emotional state, the system generates an appropriate answer or explanation.

[1806] Step 4: Provide calming advice if needed.

[1807] "Example 3"

[1808] Step 1: The user inputs an English utterance into the system.

[1809] Step 2: The emotion engine analyzes the user's emotional state.

[1810] Step 3: Based on the emotional state, the system generates appropriate pronunciation guidance and advice. Step 4: The system provides the generated pronunciation guidance and advice to the user to support English learning.

[1811] Example 1

[1812] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1813] In modern society, there is a demand for providing appropriate advice and solutions to the worries and problems faced by elementary, junior high, and high school students, as well as the elderly. However, there are only a limited number of systems that allow these users to consult in real time, and in particular, there are few systems that provide responses that take emotional state into consideration. In addition, systems that combine speech recognition and generative AI models are complex, making them difficult to put into practical use.

[1814] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes means for receiving a user's voice input, means for converting the voice data into text data, means for analyzing the text data to grasp the user's emotional state, means for generating an appropriate response using a generative AI model, means for converting the generated response into voice, and means for outputting the voice response to the user. This enables real-time interactive consultation that takes the user's emotional state into consideration.

[1815] "Means for receiving user voice input" refers to devices or software that capture the voice uttered by the user and input it into the system.

[1816] A "means for converting voice data to text data" is a process or device that uses voice recognition technology to convert a voice signal into corresponding text information.

[1817] "Means for analyzing text data to understand a user's emotional state" refers to algorithms or software for analyzing text information and identifying a user's emotional or psychological state.

[1818] "Means for generating appropriate responses using generative AI models" refers to models or systems that use artificial intelligence technology to generate appropriate responses or advice in response to user input.

[1819] The "means for converting the generated response into speech" refers to speech synthesis technology or a device that converts the text response into a speech signal and plays it aloud to the user.

[1820] The "means for outputting a voice response to a user" refers to a device or system for transmitting the generated voice to a user through an output device such as a speaker.

[1821] This invention is a system that provides real-time interactive advice to elementary, junior high, and high school students and the elderly regarding their worries and problems. This system can receive user voice input, generate appropriate responses, and output them as voice.

[1822] Hardware and software used

[1823] Hardware: microphones, speakers, servers

[1824] Software: speech recognition software (e.g., Google Speech-to-Text), sentiment analysis engines (e.g., IBM Watson Tone Analyzer), generative AI models (e.g., OpenAI GPT-3), and speech synthesis software

[1825] System configuration

[1826] 1. A means of receiving user voice input:

[1827] The user talks about their worries and problems through the microphone. For example, the user might say, "I got a bad score on the test today. I'm feeling down."

[1828] 2. How to convert audio data to text data:

[1829] The device uses speech recognition software to convert the user's speech into text, specifically using Google Speech-to-Text to generate the text "I got a bad score on the test today. I'm feeling down."

[1830] 3. Analyzing text data to understand the user's emotional state:

[1831] The server uses an emotion analysis engine to analyze the text information and understand the user's emotional state. Specifically, it uses IBM Watson Tone Analyzer to recognize the emotion "depressed."

[1832] 4. Using generative AI models to generate appropriate responses:

[1833] The server uses a generative AI model to generate appropriate advice and solutions based on the user's emotional state. Specifically, it uses OpenAI GPT-3 to generate a response such as, "Even if you got a bad score on the test, you'll be fine if you try harder next time. Let me know if there's anything I can help you with."

[1834] 5. Means for converting the generated response into speech:

[1835] The server then uses speech synthesis software to convert the generated text response into speech, specifically using speech synthesis technology to generate a voice message that says, "Even if you got a bad score on the test, you can do better next time. Let me know if there's anything I can help you with."

[1836] 6. A means of outputting a voice response to the user:

[1837] The device receives the voice data from the server and outputs it to the user through the speaker. Specifically, the speaker plays a message saying, "Even if you got a bad score on the test, you'll be fine if you try harder next time. Let me know if there's anything I can help you with."

[1838] Specific examples

[1839] User Type: "I had a fight with a friend today. How can I make up?"

[1840] System response: "I see you had a fight with your friend. The first thing you need to do is stay calm and try to understand the other person's feelings. Choose the right time to apologize and honestly express your feelings."

[1841] Prompt Sentence Examples

[1842] "I had a fight with a friend. How can I make up?"

[1843] "I got a bad score on the test today. I'm feeling down."

[1844] "I feel like I'm being bullied at school lately. What should I do?"

[1845] In this way, the system provides appropriate advice to the user's concerns and problems.

[1846] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1847] Step 1:

[1848] The user inputs voice. The user talks about their worries and problems through the microphone. A specific example of this is when the user says, "I got a bad score on the test today. I'm feeling down." The input is the user's voice data, and the output is the audio signal captured by the microphone.

[1849] Step 2:

[1850] The device sends voice data to the server. The device captures the user's voice data and sends it to the server. In concrete terms, the device uploads the voice data to the server in real time. The input is the captured voice signal, and the output is the voice data sent to the server.

[1851] Step 3:

[1852] The server performs speech recognition. Using speech recognition software, the server converts the speech data into text data. Specifically, the server uses Google Speech-to-Text to generate the text "I got a bad score on the test today. I'm feeling depressed." The input is speech data, and the output is text data.

[1853] Step 4:

[1854] The server performs emotion analysis. Using an emotion analysis engine, the server analyzes text data and understands the user's emotional state. Specifically, the server uses IBM Watson Tone Analyzer to recognize the emotion "depressed." The input is text data, and the output is emotional state data.

[1855] Step 5:

[1856] The server generates a response. Using a generative AI model, the server generates appropriate advice and solutions based on the user's emotional state. Specifically, the server uses OpenAI GPT-3 to generate a response such as, "Even if you got a bad score on the test, you'll be fine if you try harder next time. Let me know if there's anything I can help you with." The input is emotional state data and text data, and the output is the generated response text data.

[1857] Step 6:

[1858] The server converts the response into speech. The server then uses speech synthesis software to convert the generated text response into speech. Specifically, the server uses speech synthesis technology to generate speech data that says, "Even if you got a bad score on the test, you'll be fine if you try harder next time. Let me know if there's anything I can help you with." The input is the text data of the generated response, and the output is speech data.

[1859] Step 7:

[1860] The device outputs a voice response to the user. The device outputs the voice data received from the server to the user through the speaker. Specifically, the device plays a voice message from the speaker saying, "Even if you got a bad score on the test, you'll be fine if you try harder next time. Let me know if there's anything I can help you with." The input is voice data, and the output is the voice that the user can hear.

[1861] (Application example 1)

[1862] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal." ...

Claims

1. means for receiving problem information and question input from a learner; means for generating a prompt requesting an answer and an explanation for said input; means for inputting the prompt to a generative AI to generate an answer and an explanation for the input; means for analyzing the emotional state of the learner using an emotion analysis engine; A means for generating an answer that provides appropriate support based on the generated answer and explanation for the input and an analysis result of the learner's emotional state; A system including:

2. 2. The system according to claim 1, further comprising means for inputting problem information and unclear points to the learner and providing answers and advice to the learner in the manner of a private tutor.

3. 2. The system according to claim 1, further comprising: means for conducting English conversation and providing language education; and means for providing expandability to allow for adding options.

Images