System

Abstract

A system is provided.SOLUTION: The system includes a means for inputting contents of a homework question, a means for analyzing the inputted contents of the question and generating an appropriate answer, and a means for providing the generated answer to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Traditionally, when children from dual-income families are left home alone during summer vacation, they can feel isolated because they have no one to help them with their homework or to communicate with. This can be even more of a burden for children who find after-school care boring or when after-school care facilities are full. Under these circumstances, there is a need for ways for children to feel safe at home, learn, and have fun. [Means for solving the problem]

[0005] To solve the above problems, the present invention provides a system that includes a means for inputting homework questions, a means for analyzing the input questions and generating appropriate answers, and a means for providing the generated answers to the user. The system also includes a means for inputting chat messages while the user is at home alone, a means for analyzing the input messages and generating appropriate replies, and a means for providing the generated replies to the user, thereby supporting children's learning and communication even when they are alone. Furthermore, the system includes a means for converting questions and messages into JSON format and sending them to a server, and a means for analyzing the JSON-formatted answers and replies from the server and displaying or playing them back to the user, thereby enabling these processes to be carried out smoothly.

[0006] "A means for inputting homework questions" means software or hardware that provides an interface for a user to input homework questions using text or voice.

[0007] "Means for analyzing the content of the question entered and generating an appropriate answer" refers to software or algorithms that use natural language processing or database search to analyze the content of the question entered by the user and generate an appropriate answer to that question.

[0008] "Means for providing the generated answer to the user" refers to software or hardware that provides an interface or functionality for displaying and playing the analyzed and generated answer visually or audibly on the user's device.

[0009] "Means for inputting chat messages while the user is away" refers to software or hardware that provides an interface for the user to input messages in chat format.

[0010] "Means for analyzing input messages and generating appropriate replies" refers to software or algorithms that analyze chat messages input by users using natural language processing technology or the like and generate appropriate replies to those messages.

[0011] "Means for providing the generated reply to the user" refers to software or hardware that provides an interface or functionality for visually or audibly displaying or playing the generated reply on the user's device.

[0012] "Means for converting questions and messages into JSON format and sending them to a server" refers to software or modules that convert questions and messages entered by users into JSON, a structured data format, and send them to a server via internet communication.

[0013] "Means for parsing JSON format answers and replies from the server and displaying or playing them to the user" refers to software or hardware that parses JSON format answers and replies sent from the server and displays or plays their contents visually or audibly to the user. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6]FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] The present invention provides a system that uses a smartphone application to provide homework help and a communication partner for children in dual-income households who are left home alone. Specific embodiments of the system are described below.

[0036] Homework Help

[0037] overview

[0038] Users (children) can input questions about their homework through a smartphone app and receive appropriate answers. The system analyzes the questions entered by the user and generates and provides appropriate answers.

[0039] Specific operation example

[0040] 1. The user launches the smartphone app and selects the "Homework Help" menu.

[0041] 2. The user types the text "I don't know how to solve the equation" and taps the "Submit" button.

[0042] 3. The device receives the user's question and converts it into JSON format.

[0043] 4. The device sends the converted question to the server as an HTTP POST request.

[0044] 5. The server receives the request and analyzes the question, for example, using natural language processing technology to understand the question and search for related articles or calculation methods.

[0045] 6. Based on the analysis results, the server generates JSON format data including the URL of an appropriate explanatory article and specific calculation methods.

[0046] 7. The server sends the generated JSON data to the terminal as an HTTP response.

[0047] 8. The device receives the response and displays it visually to the user.

[0048] 9. The user taps on the link to the explanatory article displayed to view a detailed explanation.

[0049] Communication partner

[0050] overview

[0051] While a user is at home alone, they can chat with a virtual communication partner through a smartphone app. The system analyzes the user's input messages and generates and provides appropriate replies.

[0052] Specific operation example

[0053] 1. The user launches the smartphone app and selects the "Communication Partner" menu.

[0054] 2. The user types the text "Tell me what you'd like to do today" and taps the "Send" button.

[0055] 3. The device converts the message into JSON format and sends it to the server as an HTTP POST request.

[0056] 4. The server receives the request and analyzes the message content, for example using natural language processing techniques to understand the intent of the message and suggest appropriate activities.

[0057] 5. Based on the analysis results, the server generates JSON format data containing the suggestion, "How about playing Animal Crossing?"

[0058] 6. The server sends the generated JSON data to the terminal as an HTTP response.

[0059] 7. The device receives the response and displays or plays it back to the user visually or audibly.

[0060] 8. The user sees the suggestions and decides to play Animal Crossing.

[0061] This system allows children who are alone at home to receive support with their schoolwork and avoid feeling lonely by interacting with virtual communication partners.

[0062] The processing flow will be explained below.

[0063] Homework Help

[0064] Program processing

[0065] Step 1:

[0066] The user launches the smartphone app and selects the "Homework Help" menu.

[0067] Step 2:

[0068] A user texts in a homework question, for example, "I don't know how to solve the equation."

[0069] Step 3:

[0070] The user taps the "Send" button.

[0071] Step 4:

[0072] The terminal receives the user's question.

[0073] Step 5:

[0074] The question received by the device is converted into JSON format.

[0075] Step 6:

[0076] The device sends the converted JSON-formatted question to the server as an HTTP POST request.

[0077] Step 7:

[0078] The server receives the HTTP POST request and analyzes the question, specifically using natural language processing technology to understand the intent of the question.

[0079] Step 8:

[0080] The server searches a database or calls an external API to generate an answer that is appropriate for the question.

[0081] Step 9:

[0082] The server generates JSON data containing the URL of the appropriate explanatory article and the specific answer.

[0083] Step 10:

[0084] The server sends the generated JSON data to the terminal as an HTTP response.

[0085] Step 11:

[0086] The device receives the HTTP response and parses the JSON data.

[0087] Step 12:

[0088] The terminal displays the analyzed data in a format that the user can visually confirm.

[0089] Step 13:

[0090] For example, the user taps the link of the displayed explanatory article to check the detailed content.

[0091] Communication partner

[0092] Program processing

[0093] Step 1:

[0094] The user launches the smartphone app and selects the "Communication Partner" menu.

[0095] Step 2:

[0096] The user types the text "Tell me what you'd like to do today" into the chat window.

[0097] Step 3:

[0098] The user taps the "Send" button.

[0099] Step 4:

[0100] The terminal receives the user's message.

[0101] Step 5:

[0102] The terminal converts the received message into JSON format.

[0103] Step 6:

[0104] The device sends the converted JSON formatted message to the server as an HTTP POST request.

[0105] Step 7:

[0106] The server receives the HTTP POST request and analyzes the message content. Specifically, it uses natural language processing technology to understand the intent of the message.

[0107] Step 8:

[0108] The server searches a relevant database to generate an appropriate reply, or generates suitable reply candidates.

[0109] Step 9:

[0110] The server generates a JSON response containing the suggestion, "How about playing Animal Crossing?"

[0111] Step 10:

[0112] The server sends the generated JSON data to the terminal as an HTTP response.

[0113] Step 11:

[0114] The device receives the HTTP response and parses the JSON data.

[0115] Step 12:

[0116] The terminal presents the analyzed data to the user visually or audibly.

[0117] Step 13:

[0118] The user sees the suggestions and chooses to play Animal Crossing.

[0119] Example 1

[0120] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0121] In situations where children from dual-income households are left home alone, it is difficult to provide appropriate communication methods to support their schoolwork and alleviate loneliness. Furthermore, in order for children to efficiently complete their homework at home, they need immediate, easy-to-understand answers, but there is a lack of technology to adequately resolve this issue. Furthermore, there is a need for a system that facilitates smooth dialogue with virtual communication partners to alleviate children's feelings of loneliness.

[0122] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0123] In this invention, the server includes a means for inputting a question, a means for analyzing the input question, a means for generating an appropriate answer based on the analysis results, and a means for providing the generated answer to the user. This allows a child to efficiently complete their homework at home alone. The server also includes a means for inputting a message while the user is at home alone, a means for analyzing the input message, a means for generating an appropriate reply based on the analysis results, and a means for providing the generated reply to the user. This allows a child to feel less alone by interacting with a virtual communication partner. The server also includes a means for converting the input question and message into a data format and transmitting it to the server, and a means for analyzing the data-formatted answers and replies from the server and displaying or playing them back to the user. This allows for efficient and accurate responses to the user's questions and messages and for immediate information provision.

[0124] The "means for inputting questions" is an interface that allows the user to provide questions about homework to the system in the form of input such as text.

[0125] "Means for analyzing the content of the entered question" refers to technology that uses natural language processing technology to understand the question entered by the user and extract appropriate keywords and context.

[0126] "Means for generating appropriate answers" are algorithms and databases that provide relevant information and explanations based on the analyzed question content.

[0127] "Means for providing the generated answer to the user" refers to an interface for visually or audibly displaying and playing the answer generated by the system on the user's terminal.

[0128] The "means for inputting a message while the user is away" is an interface through which the user inputs a message to converse with the communication partner.

[0129] The "means for analyzing an input message" is a technology that analyzes the contents of the message using natural language processing technology and understands the intent and important elements of the message.

[0130] The "means for generating an appropriate reply" refers to an algorithm or database for generating an appropriate reply to the user based on the analysis results.

[0131] "Means for providing the generated reply to the user" means an interface for visually or audibly displaying and playing the reply generated by the system on the user's terminal.

[0132] "Means for converting the input question and message into a data format and sending it to the server" refers to a technology that converts the user's input into a data format such as JSON and sends it to the server via a network.

[0133] "Means for analyzing answers and replies in data format from the server and displaying or playing them back to the user" refers to technology that analyzes answer and reply data returned from the server on the user's terminal and displays or plays them back visually or audibly.

[0134] This invention is a system that provides learning support and communication partners for children in dual-income households who are left home alone. In particular, it is a smartphone application that combines an algorithm for analyzing questions and providing appropriate answers with technology for generating appropriate replies to user messages.

[0135] Homework Help

[0136] overview

[0137] Users (children) can input questions about their homework through a smartphone app and receive appropriate answers. The system analyzes the questions entered by the user and generates and provides appropriate answers.

[0138] Specific Embodiments

[0139] 1. The user launches the smartphone app and selects the "Homework Help" menu.

[0140] 2. The user enters the question in text format and taps the "Submit" button. For example, the user might enter "I don't know how to solve the equation."

[0141] 3. The device receives the question and converts it into JSON format.

[0142] 4. The device sends the converted question to the server as an HTTP POST request.

[0143] 5. The server uses the Google® Cloud NLP API to analyze the question, extracting keywords such as "equation" and "solution method."

[0144] 6. Based on the analysis results, the server generates a response in the form of data, including appropriate explanatory articles and specific calculation methods.

[0145] 7. The server sends the generated response in the data format to the terminal as an HTTP response.

[0146] 8. The device receives the response and displays it visually to the user, for example, a link to an explanatory article or a calculation method.

[0147] 9. The user taps the displayed link to view a detailed explanation.

[0148] Specific examples

[0149] Example prompt: "I don't know how to solve a linear equation like x + 3 = 7. Please help."

[0150] Communication partner

[0151] overview

[0152] While a user is at home alone, they can chat with a virtual communication partner through a smartphone app. The system analyzes the user's input messages and generates and provides appropriate replies.

[0153] Specific Embodiments

[0154] 1. The user launches the smartphone app and selects the "Communication Partner" menu.

[0155] 2. The user types a message and taps the "Send" button. For example, "Tell me what you'd like to do today."

[0156] 3. The device converts the message into JSON format and sends it to the server as an HTTP POST request.

[0157] 4. The server analyzes the message content using OpenAI's GPT-3 model and suggests appropriate activities.

[0158] 5. Based on the analysis results, the server generates a response in the form of data containing the suggestion, "How about playing Animal Crossing?"

[0159] 6. The server sends the reply in the generated data format to the terminal as an HTTP response.

[0160] 7. The device receives the response and displays or plays it back to the user visually or audibly.

[0161] 8. The user sees the suggestions and decides to play Animal Crossing.

[0162] Specific examples

[0163] Example prompt: "I'm alone and bored today. Can you suggest some fun activities?"

[0164] Hardware and software used

[0165] This system is realized using a smartphone, a server, an internet connection, the Google Cloud NLP API, and OpenAI's GPT-3 model. Specifically, the smartphone receives input from the user, converts it into a data format, and sends it to the server via the internet connection. The server analyzes the received data and generates and returns an appropriate answer or reply. The smartphone displays or plays the returned data to the user.

[0166] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0167] Homework Help

[0168] Processing steps and specific explanations

[0169] Step 1:

[0170] A user launches a smartphone app and selects the "Homework Help" menu, which displays the in-app "Homework Help" interface.

[0171] Input: Launching the smartphone app

[0172] Output: Homework help interface displayed

[0173] Step 2:

[0174] The user enters the question in text format, such as "I don't know how to solve the equation," and taps the "Send" button. This allows the input content to be managed within the app.

[0175] Input: User text input

[0176] Output: Keep input

[0177] Step 3:

[0178] The device receives the user's question and converts it into JSON format, which structures the data and makes it ready for transmission.

[0179] Input: User text input

[0180] Output: JSON format data

[0181] Specific operation:Conversion process to { "question": "I don't know how to solve the equation"}

[0182] Step 4:

[0183] The terminal sends the converted JSON data to the server as an HTTP POST request. The data is sent using the communication protocol.

[0184] Input: JSON format data

[0185] Output: An HTTP POST request is made to the server.

[0186] Step 5:

[0187] The server receives the HTTP request and retrieves the JSON data. The server then uses the Google Cloud NLP API to analyze the question, tokenizing the content and extracting keywords.

[0188] Input: JSON data from an HTTP POST request

[0189] Output: Analysis results (keywords and context)

[0190] Specific operation: Using natural language processing technology, important keywords such as "equation" and "solution method" are extracted from the question.

[0191] Step 6:

[0192] The server searches for appropriate explanatory articles and specific calculation methods based on the analysis results and generates an appropriate answer, which is then converted back into JSON format.

[0193] Input: Keywords of analysis results

[0194] Output: JSON formatted response data

[0195] Specific operation: Generate data including the URL of the related explanatory article and the calculation method { "url": "https: / / example.com / article", "method": "x = 7 - 3"}

[0196] Step 7:

[0197] The server sends the generated JSON data to the terminal as an HTTP response.

[0198] Input: JSON formatted response data

[0199] Output: Sent as an HTTP response

[0200] Step 8:

[0201] The device receives the response, parses the JSON data, and displays it visually to the user, such as links to explanatory articles and calculation instructions.

[0202] Input: JSON data of the HTTP response from the server

[0203] Output: A visual display to the user

[0204] Specific operation: Display an explanatory article link and calculation method { "url": "https: / / example.com / article", "method": "x = 7 - 3"}

[0205] Step 9:

[0206] Users can tap the link to the displayed explanatory article to view a detailed explanation, and by accessing specific learning content, they can solve the problem.

[0207] Input: Tap the displayed link

[0208] Output: View detailed explanation

[0209] Communication partner

[0210] Processing steps and specific explanations

[0211] Step 1:

[0212] The user starts the smartphone app and selects the "Communication Partner" menu, which displays a chat interface with the communication partner.

[0213] Input: Launching the smartphone app

[0214] Output: Display of communication partner interface

[0215] Step 2:

[0216] The user enters a message such as "Tell me what you'd like to do today" and taps the "Send" button. The input content is then managed within the app.

[0217] Input: User text input

[0218] Output: Keep input

[0219] Step 3:

[0220] The device receives the user's message content and converts it to JSON format, which structures the data and makes it ready for transmission.

[0221] Input: User text input

[0222] Output: JSON format data

[0223] Specific operation: Conversion process to { "message": "Tell me what you'd like to do today"}

[0224] Step 4:

[0225] The terminal sends the converted JSON data to the server as an HTTP POST request. The data is sent using the communication protocol.

[0226] Input: JSON format data

[0227] Output: An HTTP POST request is made to the server.

[0228] Step 5:

[0229] The server receives the HTTP request and retrieves the JSON data. The server then analyzes the message content using OpenAI's GPT-3 model. This analysis allows the server to understand the user's intent and derive an appropriate response.

[0230] Input: JSON data from an HTTP POST request

[0231] Output: Analysis results (user intent and appropriate response)

[0232] Specific operation: Using natural language processing technology, the system analyzes the message content and suggests appropriate activities in response to the question, "What would you like to do today?"

[0233] Step 6:

[0234] The server generates a response based on the parsing results, including the suggestion, "How about playing Animal Crossing?" The response is then converted back to JSON format.

[0235] Input: Intention of analysis results and proposals

[0236] Output: Reply data in JSON format

[0237] Specific behavior: Generate reply data such as "How about playing Animal Crossing?" { "reply": "How about playing Animal Crossing?"}

[0238] Step 7:

[0239] The server sends the generated JSON data to the terminal as an HTTP response.

[0240] Input: Reply data in JSON format

[0241] Output: Sent as an HTTP response

[0242] Step 8:

[0243] The device receives the response, parses the JSON data, and displays or plays the suggestion to the user visually or audibly.

[0244] Input: JSON data of the HTTP response from the server

[0245] Output: Visual or audio display / playback to the user

[0246] Specific behavior: Display or play a suggestion saying "How about playing Animal Crossing?"

[0247] Step 9:

[0248] The user sees the suggestions and decides to play Animal Crossing. By actually doing the suggested activities, they spend their time meaningfully while at home.

[0249] Input: Confirm and select the proposed content

[0250] Output: Activity execution

[0251] In this way, the system of the present invention allows users to efficiently complete their homework alone at home and also provides support for learning and alleviates feelings of loneliness by interacting with appropriate communication partners.

[0252] (Application example 1)

[0253] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0254] When children in dual-income households are left at home alone, they need help with their homework and appropriate communication partners to help them spend their alone time meaningfully. However, current systems lack the means to quickly and accurately obtain information when customers ask questions about products, especially when shopping in physical stores. This leaves many customers, including children, with difficulty in selecting and making decisions about products.

[0255] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0256] In this invention, the server includes a means for inputting homework questions and product information, a means for analyzing the input questions and product information and generating appropriate answers and information, and a means for providing the generated answers and information to a user. This not only enables children in dual-income households to receive help with their homework, but also enables customers in physical stores to quickly and accurately obtain appropriate information when asking questions about products.

[0257] The "means for inputting homework questions" refers to an interface that allows a user to input homework questions in text format using a device such as a smartphone or tablet.

[0258] The "means for analyzing the content of the entered question and generating an appropriate answer" refers to an algorithm and process that uses natural language processing technology to analyze the content of the question entered by the user and generate an answer that is highly relevant to the question.

[0259] The "means for providing the generated answer to the user" is an interface that displays or plays the analyzed and generated answer visually or audibly on the user's device.

[0260] The "means for inputting product information" is an interface through which customers in a physical store can input questions about product details and characteristics.

[0261] "Means for analyzing input product information and generating appropriate information" refers to algorithms and processes that use natural language processing technology to analyze questions and requests about products input by customers and generate relevant product information.

[0262] The "means for providing the generated product information to the user" is an interface that displays or plays back the analyzed and generated product information on the customer's device.

[0263] The "means for inputting chat messages while the user is at home alone" refers to an interface that allows the user to input messages in chat format using a smartphone or tablet while at home alone.

[0264] The "means for analyzing input messages and generating appropriate replies" refers to algorithms and processes that use natural language processing technology to analyze chat messages input by users and generate highly relevant replies to those messages.

[0265] The "means for providing the generated reply to the user" is an interface that displays or plays the parsed and generated reply on the user's device.

[0266] "Means for converting questions and product information into JSON format and sending it to a server" refers to the process and means for converting questions and product information entered by a user into a standardized JSON format and sending it to a server via a network as an HTTP request.

[0267] "Means for analyzing JSON format responses, product information, and replies from the server and displaying or playing them back to the user" refers to an interface that analyzes JSON format data received from the server and displays or plays it back on the device in a form that is easily understandable to the user.

[0268] "Means for analyzing question content and product information through natural language processing using a generative AI model" refers to algorithms and processes that use a generative AI model (e.g., GPT-3) to analyze question content and product information using advanced natural language processing technology.

[0269] This invention is a system that provides homework help for children in dual-income households who are left home alone, and also provides fast and accurate information when customers in physical stores have questions about products. The system consists of an application installed on a smartphone or tablet and a server that processes requests from these applications.

[0270] Hardware and software used

[0271] Hardware:

[0272] A smartphone or tablet as the user device

[0273] Data center or cloud computing infrastructure as a server

[0274] software:

[0275] Client side: Mobile application developed using React Native

[0276] Server side: Server application developed using Python and Flask

[0277] Natural language processing engine: OpenAI's GPT-3 model

[0278] Data processing and calculation flow

[0279] When a user types a question about homework or product information, the application installed on the smartphone or tablet receives the question. The application then converts the input question into JSON format and sends it as an HTTP POST request to a server. The server receives the request and uses a generative AI model (e.g., GPT-3) to analyze the question.

[0280] The server searches a database or knowledge base for appropriate answers and product information for the question, and returns the generated answers and information in JSON format to the application. The application then analyzes the received JSON data and displays or plays it back to the user visually or audibly. This process allows the user to obtain information quickly and accurately.

[0281] Specific examples

[0282] Example questions for user homework:

[0283] When a user submits a question by entering text such as "I don't know how to solve an equation," the server analyzes the question, searches for appropriate explanatory articles and calculation methods, and provides them to the user.

[0284] Example prompt sentence:

[0285] User asked: Are these sneakers waterproof?

[0286] AI Response:

[0287] Based on this, the server provides information such as, "These sneakers are made of Gore-Tex material and are waterproof."

[0288] Example prompts for the generative AI model used

[0289] User asked: Are these sneakers waterproof?

[0290] AI Response:

[0291] Specific server processing

[0292] The server utilizes natural language processing technology and generative AI models to generate appropriate answers to user questions and product information. Specifically, it analyzes the user's text input and understands their intent. It then retrieves the information most relevant to that intent from the database and generative AI model and provides it to the user in an appropriate format.

[0293] This system not only allows users to receive help with their homework, but also provides quick and accurate information for selecting products in physical stores, allowing children in dual-income households and customers in physical stores to spend their time more meaningfully.

[0294] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0295] Step 1:

[0296] The user launches the application on their smartphone or tablet and enters a question about homework or a product. The entered question is recorded in text format on the device.

[0297] input:

[0298] Questions typed in by users (e.g., "I don't know how to solve an equation" or "Are these sneakers waterproof?")

[0299] output:

[0300] Text questions recorded on the device

[0301] Specific behavior:

[0302] Using the application's interface, users enter their homework or product question and tap the "Submit" button.

[0303] Step 2:

[0304] The terminal converts the questions entered by the user into JSON format.

[0305] input:

[0306] Text-based questions

[0307] output:

[0308] Question content converted to JSON format

[0309] Specific behavior:

[0310] A script in the application converts the text question into JSON format (e.g., {"question": "I don't know how to solve the equation"}).

[0311] Step 3:

[0312] The device sends the converted JSON-formatted question to the server as an HTTP POST request.

[0313] input:

[0314] JSON formatted question

[0315] output:

[0316] HTTP POST request sent to the server

[0317] Specific behavior:

[0318] The device issues an HTTP request to the server and sends the question (e.g., a POST request with a JSON payload).

[0319] Step 4:

[0320] The server receives the HTTP POST request and parses the JSON-formatted question.

[0321] input:

[0322] Received question in JSON format

[0323] output:

[0324] Analyzed questions

[0325] Specific behavior:

[0326] The server parses the request and gets the question (e.g., parses the JSON payload using a JSON parsing library).

[0327] Step 5:

[0328] The server inputs the analyzed question into a generative AI model (e.g., GPT-3) to generate an appropriate answer.

[0329] input:

[0330] Analyzed questions

[0331] output:

[0332] Answers generated by generative AI models

[0333] Specific behavior:

[0334] The server issues an API request to the generated AI model and generates an answer based on the question (e.g., an API call to GPT-3).

[0335] Step 6:

[0336] The server generates a JSON response containing the generated answer and sends it to the device as an HTTP response.

[0337] input:

[0338] Generated Answer

[0339] output:

[0340] JSON-formatted response

[0341] Specific behavior:

[0342] The server converts the answer into JSON format and sends it to the terminal as an HTTP response (e.g., {"answer": "To solve the equation..."}).

[0343] Step 7:

[0344] The device receives and parses the JSON response from the server.

[0345] input:

[0346] JSON format response received from the server

[0347] output:

[0348] Analyzed answers

[0349] Specific behavior:

[0350] The device parses the received JSON data and formats it into the format required for display to the user.

[0351] Step 8:

[0352] The terminal displays the analyzed answer content to the user visually or audibly.

[0353] input:

[0354] Analyzed answers

[0355] output:

[0356] Answers presented to the user

[0357] Specific behavior:

[0358] The device displays the answer in a user interface so that the user can review it (e.g., displaying the answer in a text view).

[0359] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0360] The present invention combines a smartphone application system that provides homework help and communication partners for children in dual-income households who are left home alone with an emotion engine that recognizes the user's emotions. Specific embodiments of the present invention are described below.

[0361] Homework Help

[0362] overview

[0363] Users (children) can input questions about their homework through a smartphone app and receive appropriate answers. The system analyzes the questions entered by the user and generates and provides appropriate answers. It also utilizes an emotion engine that recognizes the user's emotions and responds based on the user's emotional state.

[0364] Specific operation example

[0365] 1. The user launches the smartphone app and selects the "Homework Help" menu.

[0366] 2. The user types the text "I don't know how to solve the equation" and taps the "Submit" button.

[0367] 3. The device receives the user's question and converts it into JSON format.

[0368] 4. The emotion engine recognizes emotions from the user's facial expressions and voice and generates emotion data.

[0369] 5. The device sends the question and emotion data to the server as an HTTP POST request.

[0370] 6. The server receives the request and analyzes the question and sentiment data, for example using natural language processing to understand the question and tailor the answer taking into account the sentiment data.

[0371] 7. The server generates JSON data including the URL of the appropriate explanatory article and specific calculation methods.

[0372] 8. The server sends the generated JSON data to the terminal as an HTTP response.

[0373] 9. The device receives the response and displays it visually to the user.

[0374] 10. The user taps on the link to the explanatory article displayed to view a detailed explanation.

[0375] Communication partner

[0376] overview

[0377] While the user is home alone, they can chat with a virtual communication partner through a smartphone app. The system analyzes the user's input messages, generates and provides appropriate replies, and uses an emotion engine to respond based on the user's emotional state.

[0378] Specific operation example

[0379] 1. The user launches the smartphone app and selects the "Communication Partner" menu.

[0380] 2. The user types the text "Tell me what you'd like to do today" and taps the "Send" button.

[0381] 3. The device converts the message into JSON format and sends it to the server as an HTTP POST request.

[0382] 4. The emotion engine recognizes emotions from the user's facial expressions and voice and generates emotion data.

[0383] 5. The device sends the message content and emotion data to the server.

[0384] 6. The server receives the request and analyzes the message content and emotional data, for example using natural language processing techniques to generate an appropriate reply and adjust the reply content taking the emotional data into account.

[0385] 7. The server generates a JSON response containing the suggestion, "How about playing Animal Crossing?"

[0386] 8. The server sends the generated JSON data to the terminal as an HTTP response.

[0387] 9. The device receives the response and presents it to the user visually or audibly.

[0388] 10. The user sees the suggestions and chooses to play Animal Crossing.

[0389] This system allows children who are alone at home to receive support with their schoolwork and avoid feeling lonely through conversations with virtual communication partners. Furthermore, the introduction of an emotion engine enables flexible responses based on the user's emotional state, providing more personalized support.

[0390] The processing flow will be explained below.

[0391] Homework Help

[0392] Program processing

[0393] Step 1:

[0394] The user launches the smartphone app and selects the "Homework Help" menu.

[0395] Step 2:

[0396] A user texts in a homework question, for example, "I don't know how to solve the equation."

[0397] Step 3:

[0398] The user taps the "Send" button.

[0399] Step 4:

[0400] The terminal receives the user's question.

[0401] Step 5:

[0402] In order for the device to analyze the user's facial expressions and voice and recognize emotions, it activates an emotion engine and acquires emotion data.

[0403] Step 6:

[0404] The device converts the question content and emotion data into JSON format.

[0405] Step 7:

[0406] The device sends the converted JSON-formatted question and emotion data to the server as an HTTP POST request.

[0407] Step 8:

[0408] The server receives the HTTP POST request, analyzes the question and emotional data, and uses natural language processing technology to understand the intent of the question and generate an answer that takes the emotional data into account.

[0409] Step 9:

[0410] The server searches a database or calls external APIs to generate an answer that fits the question, for example adding detailed explanations or encouraging messages if the user is confused.

[0411] Step 10:

[0412] The server generates JSON data containing the URL of the appropriate explanatory article and the specific answer.

[0413] Step 11:

[0414] The server sends the generated JSON data to the terminal as an HTTP response.

[0415] Step 12:

[0416] The device receives the HTTP response and parses the JSON data.

[0417] Step 13:

[0418] The device displays the analyzed data in a format that the user can visually confirm, such as a link to an explanatory article or a message of support.

[0419] Step 14:

[0420] The user taps the link to the explanatory article displayed to check the detailed content.

[0421] Communication partner

[0422] Program processing

[0423] Step 1:

[0424] The user launches the smartphone app and selects the "Communication Partner" menu.

[0425] Step 2:

[0426] The user types the text "Tell me what you'd like to do today" into the chat window.

[0427] Step 3:

[0428] The user taps the "Send" button.

[0429] Step 4:

[0430] The terminal receives the user's message.

[0431] Step 5:

[0432] In order for the device to analyze the user's facial expressions and voice and recognize emotions, it activates an emotion engine and acquires emotion data.

[0433] Step 6:

[0434] The device converts the message content and emotion data into JSON format.

[0435] Step 7:

[0436] The device sends the converted JSON-formatted message content and emotion data to the server as an HTTP POST request.

[0437] Step 8:

[0438] The server receives the HTTP POST request, analyzes the message content and emotional data, and uses natural language processing technology to understand the intent of the message and generate a reply that takes the emotional data into account.

[0439] Step 9:

[0440] The server searches a relevant database to generate an appropriate reply, and generates suitable reply suggestions, such as activity suggestions if the user is bored.

[0441] Step 10:

[0442] The server generates a JSON response containing the suggestion, "How about playing Animal Crossing?"

[0443] Step 11:

[0444] The server sends the generated JSON data to the terminal as an HTTP response.

[0445] Step 12:

[0446] The device receives the HTTP response and parses the JSON data.

[0447] Step 13:

[0448] The device presents the analyzed data to the user visually or audibly, such as suggestion messages or specific steps for an activity.

[0449] Step 14:

[0450] The user sees the suggestions and chooses to play Animal Crossing.

[0451] Example 2

[0452] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0453] In situations where children in dual-income households are left home alone, there is a need to provide appropriate support for homework questions and communication methods to reduce feelings of loneliness. It is also important that this support and communication take into account the child's emotional state, and personalized responses are required.

[0454] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for inputting homework questions; means for analyzing the input questions and generating appropriate answers; means for providing the generated answers to the user; means having an engine for recognizing the user's emotions and generating emotion data; means for transmitting the questions and emotion data and receiving answers from the server; means for inputting chat messages while the user is at home alone; means for analyzing the input messages and generating appropriate replies; means for providing the generated replies to the user; means for transmitting the message contents and emotion data and receiving the replies from the server; means for converting the questions and messages into a data format and transmitting them; and means for analyzing the answers and replies in data format from the server and displaying or playing them back to the user. This allows children to receive support with their homework while at home alone and helps them feel less lonely through virtual communication.

[0455] The "means for inputting homework questions" is a system component that provides an interface for users to input homework questions as text.

[0456] The "means for analyzing the entered question and generating an appropriate answer" is a component of the system that uses natural language processing technology to analyze the homework question entered by the user and generate the optimal answer to that question.

[0457] A "means for providing a generated answer to a user" is a system component that provides an interface for presenting the system-generated answer to a user visually or audibly.

[0458] "Means having an engine for recognizing a user's emotions and generating emotional data" refers to a system component that includes an engine that uses a camera or microphone to analyze a user's facial expressions and tone of voice, identify their emotional state, and generate emotional data.

[0459] The "means for sending question content and emotion data to a server and receiving a response from the server" refers to a system component that sends a question input by a user and generated emotion data to a server via an HTTP POST request and receives the response data received from the server.

[0460] The "means for inputting chat messages while the user is away" is a system component that provides an interface for the user to input chat messages in text format.

[0461] The "means for analyzing input messages and generating appropriate replies" is a component of a system that uses natural language processing technology to analyze chat messages input by users and generate optimal replies to those messages.

[0462] A "means for providing a generated reply to a user" is a system component that provides an interface for presenting a system-generated reply to a user visually or audibly.

[0463] "Means for transmitting message content and emotion data to a server and receiving a reply from the server" refers to a system component for transmitting message content entered by a user and generated emotion data to a server and receiving reply data received from the server.

[0464] "Means for converting questions and messages into a data format and sending them to the server" refers to a system component that converts questions and messages entered by users into a data format such as JSON and sends them to the server.

[0465] "Means for analyzing answers and replies in data format from the server and displaying or playing them back to the user" refers to a system component that analyzes answers and replies provided in data format such as JSON received from the server and displays or plays them back to the user visually or audibly.

[0466] The present invention combines a smartphone application system that provides homework help and communication partners for children in dual-income households who are left home alone with an emotion engine that recognizes the user's emotions. Specific embodiments of the present invention are described below.

[0467] Homework help feature

[0468] Basic configuration

[0469] This system uses a smartphone app to input questions related to homework and provides a series of functions to obtain appropriate answers. The main elements are a means for inputting the user's question, a means for analyzing the question and generating an appropriate answer, a means for providing the generated answer to the user, and a means for generating emotion data using an emotion engine. This system uses HTTPS communication to send the user's question and emotion data to the server and receive the answer from the server. It is also configured to convert the question and emotion data into JSON format and send it, and to analyze the answer received from the server and display it to the user.

[0470] Operation overview

[0471] 1. The user launches the smartphone app and selects the "Homework Help" menu.

[0472] 2. The user types the text "I don't know how to solve the equation" and taps the "Submit" button.

[0473] 3. The device converts the input question into JSON format and uses an emotion engine to recognize emotions from the user's facial expressions and voice, generating emotion data. The emotion engine uses, for example, the Microsoft® Azure® emotion analysis API.

[0474] 4. The device sends the question and emotion data to the server as an HTTP POST request, using a communication library such as OkHttp.

[0475] 5. The server analyzes the received request and generates an answer appropriate to the question. For natural language processing, it uses an NLP model, such as Hugging Face.

[0476] 6. The server generates a response and sends it to the device in JSON format.

[0477] 7. The terminal analyzes the received response and displays it visually to the user.

[0478] Examples of concrete examples and prompts

[0479] Examples:

[0480] User types, "Help me with my math homework today!"

[0481] The device uses an emotion engine to recognize when the user is feeling stressed.

[0482] The server provides detailed explanatory articles in a gentle tone.

[0483] Example prompt sentence:

[0484] If your child types "Help me with my math homework today" into your app, generate an appropriate response that takes into account the stress your child may be under.

[0485] Communication partner function

[0486] Basic configuration

[0487] This system provides a function that enables users to chat with virtual communication partners through a smartphone app while they are home alone. The main elements are a means for inputting messages, a means for analyzing the message content and generating an appropriate reply, a means for providing the generated reply to the user, and a means for generating emotion data using an emotion engine. The system is configured to send the message content and emotion data to a server, analyze the replies received from the server, and display or play them back to the user.

[0488] Operation overview

[0489] 1. The user launches the smartphone app and selects the "Communication Partner" menu.

[0490] 2. The user types the text "Tell me what you'd like to do today" and taps the "Send" button.

[0491] 3. The device converts the message content into JSON format and sends it to the server as an HTTP POST request.

[0492] 4. The emotion engine generates emotion data from the user's facial expressions and voice. For emotion analysis, it uses the API of IBM Watson (registered trademark), for example.

[0493] 5. The device sends the message content and emotion data to the server.

[0494] 6. The server analyzes the received request and generates an appropriate reply, for example using OpenAI GPT-3.

[0495] 7. The server sends the appropriate reply in JSON format to the device.

[0496] 8. The device analyzes the reply received and presents it to the user visually or audibly.

[0497] Examples of concrete examples and prompts

[0498] Examples:

[0499] A user types, "What would be fun to do today?"

[0500] The device uses its emotion engine to recognize that the user is feeling a little lonely.

[0501] "Animal Crossing is a fun game," the server suggested.

[0502] Example prompt sentence:

[0503] When a child types into your app, "What would be fun to do today?", consider that they might be feeling a little lonely and generate fun suggestions. Include popular games and activities in your suggestions.

[0504] This system allows children who are alone at home to receive support with their schoolwork and avoid feeling lonely by interacting with virtual communication partners. The introduction of an emotion engine also enables flexible responses based on the user's emotional state, providing more personalized support.

[0505] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0506] Homework help feature

[0507] Step 1:

[0508] The user launches the smartphone app and selects the "Homework Help" menu.

[0509] Specific operation: The user taps the app from the smartphone home screen to launch it, and selects the "Homework Help" menu on the initial screen.

[0510] Input: User taps.

[0511] Output: Homework help menu will be displayed.

[0512] Step 2:

[0513] The user types the text "I don't know how to solve the equation" and taps the "Submit" button.

[0514] Specific actions: The user uses the smartphone keyboard to enter a question into the text box and taps the "Send" button.

[0515] Input: The question entered by the user is "I don't know how to solve an equation."

[0516] Output: The question is sent to the app.

[0517] Step 3:

[0518] The device converts the entered question into JSON format.

[0519] What it does: The app on the device reads the user's text input and converts it to JSON format.

[0520] Input: The question entered by the user is "I don't know how to solve an equation."

[0521] Data processing: Converting to JSON format (e.g., {"question": "I don't know how to solve the equation"}).

[0522] Output: Data converted to JSON format.

[0523] Step 4:

[0524] The emotion engine recognizes emotions from the user's facial expressions and voice and generates emotion data.

[0525] Specific operation: Using the device's camera and microphone, the system analyzes the user's facial expressions and voice to generate emotion data. For example, it uses Microsoft Azure's emotion analysis API as an emotion engine.

[0526] Input: User facial and voice data obtained from the camera and microphone.

[0527] Data operations: Identify emotional states and generate emotion data (e.g., {"emotion": "frustrated"}).

[0528] Output: Emotion data.

[0529] Step 5:

[0530] The device sends the question and emotion data to the server as an HTTP POST request.

[0531] Specific operation: The device sends the generated question and emotion data together to the server. A communication library (e.g., OkHttp) is used for communication.

[0532] Input: Question and emotion data in JSON format (e.g., {"question": "I don't know how to solve the equation", "emotion": "frustrated"}).

[0533] Output: An HTTP POST request is sent to the server.

[0534] Step 6:

[0535] The server receives the request and analyzes the question and emotion data.

[0536] What happens: The server receives the request and uses natural language processing techniques (e.g., the Hugging Face NLP model) to analyze the question and sentiment data.

[0537] Input: Submitted question and sentiment data.

[0538] Data Computing: Uses natural language processing to understand questions and tailor answers based on sentiment data.

[0539] Output: The appropriate answer data (e.g., an explanation of how to solve the equation).

[0540] Step 7:

[0541] The server generates JSON-formatted data including the URL of the appropriate explanatory article and specific calculation methods.

[0542] Specific operation: The server generates an appropriate answer and creates JSON-formatted data including the URL of the explanatory article and the specific calculation method.

[0543] Input: Analysis result data.

[0544] Data processing: Generate answer data in JSON format (e.g., {"answer": "To solve the equation...", "url": "http: / / example.com / solution"}).

[0545] Output: Response data in JSON format.

[0546] Step 8:

[0547] The server sends the generated JSON data to the terminal as an HTTP response.

[0548] Specific operation: The server generates data and sends it to the terminal as an HTTP response.

[0549] Input: Response data in JSON format.

[0550] Output: The data sent as the HTTP response.

[0551] Step 9:

[0552] The terminal receives the response and visually displays it to the user.

[0553] Specific operation: Analyzes the data received by the device and displays it in an easy-to-read format for the user. Uses a UI library (e.g., RecyclerView in ANDROID (registered trademark)).

[0554] Input: The JSON data sent as an HTTP response.

[0555] Data processing: Parsing and visualizing JSON data.

[0556] Output: The explanation or calculation displayed to the user.

[0557] Step 10:

[0558] The user taps the link of the displayed explanatory article to view a detailed explanation.

[0559] What happens: The user taps on a link to open a browser.

[0560] Input: User taps.

[0561] Output: A detailed explanatory article displayed in your browser.

[0562] Communication partner function

[0563] Step 1:

[0564] The user launches the smartphone app and selects the "Communication Partner" menu.

[0565] Specific behavior: The user opens the app and selects the "People to communicate with" menu.

[0566] Input: User taps.

[0567] Output: The communication partner menu is displayed.

[0568] Step 2:

[0569] The user types the text "Tell me what you'd like to do today" and taps the "Send" button.

[0570] Specific behavior: The user types "What would you like to do for fun today?" into the text box and taps the "Send" button.

[0571] Input: User-entered message: "Tell me what you'd like to do today."

[0572] Output: The message content is sent to the app.

[0573] Step 3:

[0574] The device converts the message content into JSON format and sends it to the server as an HTTP POST request.

[0575] Specific operation: The device converts the input message into JSON format and sends it to the server.

[0576] Input: User-entered message: "Tell me what you'd like to do today."

[0577] Data processing: Converting to JSON format (e.g., {"message": "Tell me what you'd like to do today"}).

[0578] Output: The data is converted to JSON format and sent to the server.

[0579] Step 4:

[0580] The emotion engine generates emotion data from the user's facial expressions and voice.

[0581] Specific operation: The emotion engine recognizes emotions through the camera and microphone and generates data on them.

[0582] Input: User facial and voice data obtained from the camera and microphone.

[0583] Data operations: Identify emotional states and generate emotion data (e.g., {"emotion": "lonely"}).

[0584] Output: Emotion data.

[0585] Step 5:

[0586] The device sends the message content and emotion data to the server.

[0587] Specific operation: The device sends the acquired message content and emotion data to the server.

[0588] Input: Message content and emotion data in JSON format (e.g., {"message": "Tell me what you'd like to do today", "emotion": "lonely"}).

[0589] Output: An HTTP POST request is sent to the server.

[0590] Step 6:

[0591] The server receives the request and analyzes the message content and emotion data.

[0592] Specific operation: The server analyzes the received data and generates appropriate suggestions. For natural language processing, OpenAI GPT-3 is used as an example.

[0593] Input: Message content and sentiment data.

[0594] Data Computation: Analysis and response generation using natural language processing.

[0595] Output: Appropriate reply data (e.g., fun activity suggestions).

[0596] Step 7:

[0597] The server generates a JSON representation of the appropriate response.

[0598] Specific operation: The server converts the proposal based on the analysis results into JSON format data.

[0599] Input: Analysis results.

[0600] Data processing: Generate reply data in JSON format (e.g., {"reply": "How about playing Animal Crossing?"}).

[0601] Output: Reply data in JSON format.

[0602] Step 8:

[0603] The server sends the generated JSON data to the terminal as an HTTP response.

[0604] Specific operation: The data generated by the server is sent to the terminal as an HTTP response.

[0605] Input: Reply data in JSON format.

[0606] Output: The data sent as the HTTP response.

[0607] Step 9:

[0608] The terminal receives the response and presents it to the user visually or audibly.

[0609] Specific operation: The device analyzes the reply data received and displays it on the screen or reads it aloud. It uses a UI library or text-to-speech library.

[0610] Input: The JSON data sent as an HTTP response.

[0611] Data processing: Parse JSON data and display or output it as audio.

[0612] Output: Screen display and audio notification.

[0613] Step 10:

[0614] The user sees the suggestions and chooses to play Animal Crossing.

[0615] Specific behavior: The user accepts the suggestion and selects an activity.

[0616] Input: User's choice.

[0617] Output: The execution of the selected activity.

[0618] Through the above processing steps, children can receive help with their homework while at home alone, and can avoid feeling lonely through virtual communication.

[0619] (Application example 2)

[0620] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0621] In recent years, the number of dual-income households has increased, resulting in more opportunities for children to be left home alone. However, in these situations, children may have questions or feel anxious about their homework. In particular, there is a need for communication methods to receive appropriate answers to questions about homework and to reduce feelings of loneliness. There is also a need for systems that can be used safely for small payments. It is hoped that these issues can be resolved and an environment in which children can spend their time with peace of mind will be provided.

[0622] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting homework questions, means for analyzing the input questions and generating appropriate answers, means for providing the generated answers to the user, means for recognizing the user's emotions and generating emotion data, and means for adjusting a response based on the generated emotion data. This makes it possible to provide a sense of security to children who are home alone through support with homework and communication, and to enable safe small-amount payments.

[0623] "Means for inputting homework questions" refers to a function that allows users to input questions about homework via devices such as smartphones or personal computers.

[0624] "Means for analyzing the content of the question entered and generating an appropriate answer" refers to a function that analyzes the content of the question entered by the user using natural language processing technology, etc., and generates an appropriate answer.

[0625] The "means for providing the generated answer to the user" is a function for presenting the answer generated through analysis to the user visually or audibly.

[0626] The "means for recognizing the user's emotions and generating emotion data" is a function for recognizing emotions from the user's facial expressions, voice, etc., and generating emotion data based on the emotions.

[0627] The "means for adjusting a response based on the generated emotion data" is a function for taking the optimal response to the user's emotional state based on the generated emotion data.

[0628] "Means for inputting chat messages while the user is at home alone" is a function for inputting messages to chat with a virtual communication partner while the user is at home alone.

[0629] The "means for analyzing input messages and generating appropriate replies" refers to a function that analyzes chat messages input by users using natural language processing technology, etc., and generates appropriate replies.

[0630] The "means for providing the generated reply to the user" is a function for presenting the generated reply to the user and continuing the dialogue with the virtual communication partner.

[0631] The "means for generating emotion data and transmitting the emotion data to a server" is a function for transmitting emotion data generated by an emotion engine to a server.

[0632] "Means for converting questions and messages into JSON format and sending them to the server" is a function for converting questions and messages entered by the user into JSON format and sending them to the server.

[0633] "Means for analyzing JSON format answers and replies from the server and displaying or playing them back to the user" refers to a function that analyzes JSON format answers and replies received from the server and provides them to the user visually or audibly.

[0634] This invention combines a smartphone application system that provides homework help and communication partners for children in dual-income households while they are home alone with an emotion engine that recognizes the user's emotions. Specific embodiments will be described in detail below.

[0635] 1. Homework Help

[0636] Program processing

[0637] The terminal accepts questions about homework entered by the child. The questions are entered in text format through the "means for entering homework question content." The terminal analyzes the questions using a natural language processing engine, generates an appropriate answer, and provides it to the user through the "means for analyzing entered question content and generating an appropriate answer." In doing so, the terminal uses an emotion engine to analyze the user's facial expressions and voice and generate emotion data. The emotion data generated by the "means for recognizing user emotions and generating emotion data" is sent to the server, and the answer is adjusted based on the analysis results. The resulting answer is displayed to the user through the "means for providing the generated answer to the user."

[0638] Hardware and software used

[0639] Devices: Smartphones, tablets

[0640] Natural language processing engine: TENSORFLOW®, e.g., GPT-based models

[0641] Emotion Engine: EmotionRecognition Library

[0642] Specific examples

[0643] The user types, "I don't know how to solve the equation."

[0644] The emotion engine analyzes the user's facial expressions and detects stress.

[0645] The server generates an answer presented in a gentler tone based on the question content and stress emotional data.

[0646] 2. Communication Partner

[0647] Program processing

[0648] The terminal accepts chat messages entered by the user, and the messages entered through the "means for entering chat messages while the user is away" are analyzed using a natural language processing engine. The server generates an appropriate reply and provides it to the user through the "means for analyzing the entered message and generating an appropriate reply." At this time, an emotion engine recognizes emotions from the user's facial expressions and voice and generates emotion data. The reply content is adjusted based on the generated emotion data. The reply is presented to the user through the "means for providing the generated reply to the user."

[0649] Hardware and software used

[0650] Devices: Smartphones, tablets

[0651] Natural language processing engines: TensorFlow, e.g., GPT-based models

[0652] Emotion Engine: EmotionRecognition Library

[0653] Specific examples

[0654] The user types, "Tell me what you'd like to do today."

[0655] The emotion engine detects boredom from the user's tone of voice.

[0656] The server uses the suggestion to generate a reply: "How about playing Animal Crossing?"

[0657] 3. Examples of Electronic Payment Services

[0658] Program processing

[0659] The terminal accepts a payment request entered by the user. The payment details are converted to JSON format and sent to the server using the "means for converting questions and messages into JSON format and sending them to the server." The server then analyzes the user's emotions using an emotion engine and takes appropriate action. For example, if the user is feeling anxious, a notification is sent to the parent. Finally, the payment result is presented to the user using the "means for analyzing JSON-formatted answers and replies from the server and displaying or playing them back to the user."

[0660] Hardware and software used

[0661] Devices: Smartphones, tablets

[0662] Emotion Engine: EmotionRecognition Library

[0663] Notification System: Email Service API

[0664] Specific examples

[0665] The user types, "Please tell me which math textbook you recommend that costs less than 500 yen."

[0666] The emotion engine recognizes the user's tension.

[0667] The server uses emotional data from stressful situations to suggest recommended textbooks using reassuring language.

[0668] The above is an embodiment of the present invention. This system allows children in dual-income households to safely make small payments when they are left at home alone. Furthermore, by utilizing an emotion engine, personalized support is provided based on the emotional state of the user (child).

[0669] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0670] Step 1:

[0671] The user starts up the smartphone and asks for help with their homework or enters a chat message. Specifically, the user types, "I don't know how to solve this equation," and this question is sent to the device.

[0672] Step 2:

[0673] The terminal converts the input question content into JSON format for data analysis, structuring the user's input data and facilitating processing on the server side.

[0674] Step 3:

[0675] The emotion engine generates emotion data in real time from the user's facial expressions and voice. For example, it receives the user's facial expression images and voice data as input, performs emotion analysis based on them, and outputs emotion labels such as "stress" or "boredom."

[0676] Step 4:

[0677] The device sends the question content converted into JSON format and the generated emotion data to the server as an HTTP POST request. Specifically, the data format is a JSON object containing the question content and emotion data, which is sent in the body of the HTTP request.

[0678] Step 5:

[0679] The server analyzes the received request, extracts the question content and emotion data, and uses this data to analyze the question content using a natural language processing engine and generate an appropriate answer.

[0680] Step 6:

[0681] The server adjusts the generated response taking into account the emotional data. For example, if the user is feeling "stressed," the server adjusts the response to be gentler and more reassuring.

[0682] Step 7:

[0683] The server constructs the tailored answer in JSON format and sends it to the device as an HTTP response, which includes the answer to the question and a sentiment-sensitive message.

[0684] Step 8:

[0685] The terminal analyzes the received response and presents the answer data to the user visually or audibly, and the user views the presented answer and solves the homework problem.

[0686] Step 9:

[0687] If the user enters another question or chat message, the process repeats, for example, "Tell me what you'd like to do today."

[0688] Step 10:

[0689] The server generates an appropriate reply from the chatbot based on the new message entered and adjusts it again taking into account the emotional data. The device then displays the adjusted reply to the user.

[0690] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0691] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0692] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0693] [Second embodiment]

[0694] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0695] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0696] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0697] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0698] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0699] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0700] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0701] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0702] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0703] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0704] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0705] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0706] The present invention provides a system that uses a smartphone application to provide homework help and a communication partner for children in dual-income households who are left home alone. Specific embodiments of the system are described below.

[0707] Homework Help

[0708] overview

[0709] Users (children) can input questions about their homework through a smartphone app and receive appropriate answers. The system analyzes the questions entered by the user and generates and provides appropriate answers.

[0710] Specific operation example

[0711] 1. The user launches the smartphone app and selects the "Homework Help" menu.

[0712] 2. The user types the text "I don't know how to solve the equation" and taps the "Submit" button.

[0713] 3. The device receives the user's question and converts it into JSON format.

[0714] 4. The device sends the converted question to the server as an HTTP POST request.

[0715] 5. The server receives the request and analyzes the question, for example, using natural language processing technology to understand the question and search for related articles or calculation methods.

[0716] 6. Based on the analysis results, the server generates JSON format data including the URL of an appropriate explanatory article and specific calculation methods.

[0717] 7. The server sends the generated JSON data to the terminal as an HTTP response.

[0718] 8. The device receives the response and displays it visually to the user.

[0719] 9. The user taps on the link to the explanatory article displayed to view a detailed explanation.

[0720] Communication partner

[0721] overview

[0722] While a user is at home alone, they can chat with a virtual communication partner through a smartphone app. The system analyzes the user's input messages and generates and provides appropriate replies.

[0723] Specific operation example

[0724] 1. The user launches the smartphone app and selects the "Communication Partner" menu.

[0725] 2. The user types the text "Tell me what you'd like to do today" and taps the "Send" button.

[0726] 3. The device converts the message into JSON format and sends it to the server as an HTTP POST request.

[0727] 4. The server receives the request and analyzes the message content, for example using natural language processing techniques to understand the intent of the message and suggest appropriate activities.

[0728] 5. Based on the analysis results, the server generates JSON format data containing the suggestion, "How about playing Animal Crossing?"

[0729] 6. The server sends the generated JSON data to the terminal as an HTTP response.

[0730] 7. The device receives the response and displays or plays it back to the user visually or audibly.

[0731] 8. The user sees the suggestions and decides to play Animal Crossing.

[0732] This system allows children who are alone at home to receive support with their schoolwork and avoid feeling lonely by interacting with virtual communication partners.

[0733] The processing flow will be explained below.

[0734] Homework Help

[0735] Program processing

[0736] Step 1:

[0737] The user launches the smartphone app and selects the "Homework Help" menu.

[0738] Step 2:

[0739] A user texts in a homework question, for example, "I don't know how to solve the equation."

[0740] Step 3:

[0741] The user taps the "Send" button.

[0742] Step 4:

[0743] The terminal receives the user's question.

[0744] Step 5:

[0745] The question received by the device is converted into JSON format.

[0746] Step 6:

[0747] The device sends the converted JSON-formatted question to the server as an HTTP POST request.

[0748] Step 7:

[0749] The server receives the HTTP POST request and analyzes the question, specifically using natural language processing technology to understand the intent of the question.

[0750] Step 8:

[0751] The server searches a database or calls an external API to generate an answer that is appropriate for the question.

[0752] Step 9:

[0753] The server generates JSON data containing the URL of the appropriate explanatory article and the specific answer.

[0754] Step 10:

[0755] The server sends the generated JSON data to the terminal as an HTTP response.

[0756] Step 11:

[0757] The device receives the HTTP response and parses the JSON data.

[0758] Step 12:

[0759] The terminal displays the analyzed data in a format that the user can visually confirm.

[0760] Step 13:

[0761] For example, the user taps the link of the displayed explanatory article to check the detailed content.

[0762] Communication partner

[0763] Program processing

[0764] Step 1:

[0765] The user launches the smartphone app and selects the "Communication Partner" menu.

[0766] Step 2:

[0767] The user types the text "Tell me what you'd like to do today" into the chat window.

[0768] Step 3:

[0769] The user taps the "Send" button.

[0770] Step 4:

[0771] The terminal receives the user's message.

[0772] Step 5:

[0773] The terminal converts the received message into JSON format.

[0774] Step 6:

[0775] The device sends the converted JSON formatted message to the server as an HTTP POST request.

[0776] Step 7:

[0777] The server receives the HTTP POST request and analyzes the message content. Specifically, it uses natural language processing technology to understand the intent of the message.

[0778] Step 8:

[0779] The server searches a relevant database to generate an appropriate reply, or generates suitable reply candidates.

[0780] Step 9:

[0781] The server generates a JSON response containing the suggestion, "How about playing Animal Crossing?"

[0782] Step 10:

[0783] The server sends the generated JSON data to the terminal as an HTTP response.

[0784] Step 11:

[0785] The device receives the HTTP response and parses the JSON data.

[0786] Step 12:

[0787] The terminal presents the analyzed data to the user visually or audibly.

[0788] Step 13:

[0789] The user sees the suggestions and chooses to play Animal Crossing.

[0790] Example 1

[0791] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0792] In situations where children from dual-income households are left home alone, it is difficult to provide appropriate communication methods to support their schoolwork and alleviate loneliness. Furthermore, in order for children to efficiently complete their homework at home, they need immediate, easy-to-understand answers, but there is a lack of technology to adequately resolve this issue. Furthermore, there is a need for a system that facilitates smooth dialogue with virtual communication partners to alleviate children's feelings of loneliness.

[0793] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0794] In this invention, the server includes a means for inputting a question, a means for analyzing the input question, a means for generating an appropriate answer based on the analysis results, and a means for providing the generated answer to the user. This allows a child to efficiently complete their homework at home alone. The server also includes a means for inputting a message while the user is at home alone, a means for analyzing the input message, a means for generating an appropriate reply based on the analysis results, and a means for providing the generated reply to the user. This allows a child to feel less alone by interacting with a virtual communication partner. The server also includes a means for converting the input question and message into a data format and transmitting it to the server, and a means for analyzing the data-formatted answers and replies from the server and displaying or playing them back to the user. This allows for efficient and accurate responses to the user's questions and messages and for immediate information provision.

[0795] The "means for inputting questions" is an interface that allows the user to provide questions about homework to the system in the form of input such as text.

[0796] "Means for analyzing the content of the entered question" refers to technology that uses natural language processing technology to understand the question entered by the user and extract appropriate keywords and context.

[0797] "Means for generating appropriate answers" are algorithms and databases that provide relevant information and explanations based on the analyzed question content.

[0798] "Means for providing the generated answer to the user" refers to an interface for visually or audibly displaying and playing the answer generated by the system on the user's terminal.

[0799] The "means for inputting a message while the user is away" is an interface through which the user inputs a message to converse with the communication partner.

[0800] The "means for analyzing an input message" is a technology that analyzes the contents of the message using natural language processing technology and understands the intent and important elements of the message.

[0801] The "means for generating an appropriate reply" refers to an algorithm or database for generating an appropriate reply to the user based on the analysis results.

[0802] "Means for providing the generated reply to the user" means an interface for visually or audibly displaying and playing the reply generated by the system on the user's terminal.

[0803] "Means for converting the input question and message into a data format and sending it to the server" refers to a technology that converts the user's input into a data format such as JSON and sends it to the server via a network.

[0804] "Means for analyzing answers and replies in data format from the server and displaying or playing them back to the user" refers to technology that analyzes answer and reply data returned from the server on the user's terminal and displays or plays them back visually or audibly.

[0805] This invention is a system that provides learning support and communication partners for children in dual-income households who are left home alone. In particular, it is a smartphone application that combines an algorithm for analyzing questions and providing appropriate answers with technology for generating appropriate replies to user messages.

[0806] Homework Help

[0807] overview

[0808] Users (children) can input questions about their homework through a smartphone app and receive appropriate answers. The system analyzes the questions entered by the user and generates and provides appropriate answers.

[0809] Specific Embodiments

[0810] 1. The user launches the smartphone app and selects the "Homework Help" menu.

[0811] 2. The user enters the question in text format and taps the "Submit" button. For example, the user might enter "I don't know how to solve the equation."

[0812] 3. The device receives the question and converts it into JSON format.

[0813] 4. The device sends the converted question to the server as an HTTP POST request.

[0814] 5. The server uses the Google Cloud NLP API to analyze the question, extracting keywords such as "equation" and "solution method."

[0815] 6. Based on the analysis results, the server generates a response in the form of data, including appropriate explanatory articles and specific calculation methods.

[0816] 7. The server sends the generated response in the data format to the terminal as an HTTP response.

[0817] 8. The device receives the response and displays it visually to the user, for example, a link to an explanatory article or a calculation method.

[0818] 9. The user taps the displayed link to view a detailed explanation.

[0819] Specific examples

[0820] Example prompt: "I don't know how to solve a linear equation like x + 3 = 7. Please help."

[0821] Communication partner

[0822] overview

[0823] While a user is at home alone, they can chat with a virtual communication partner through a smartphone app. The system analyzes the user's input messages and generates and provides appropriate replies.

[0824] Specific Embodiments

[0825] 1. The user launches the smartphone app and selects the "Communication Partner" menu.

[0826] 2. The user types a message and taps the "Send" button. For example, "Tell me what you'd like to do today."

[0827] 3. The device converts the message into JSON format and sends it to the server as an HTTP POST request.

[0828] 4. The server uses OpenAI's GPT-3 model to analyze the message content and suggest appropriate activities.

[0829] 5. Based on the analysis results, the server generates a response in the form of data containing the suggestion, "How about playing Animal Crossing?"

[0830] 6. The server sends the reply in the generated data format to the terminal as an HTTP response.

[0831] 7. The device receives the response and displays or plays it back to the user visually or audibly.

[0832] 8. The user sees the suggestions and decides to play Animal Crossing.

[0833] Specific examples

[0834] Example prompt: "I'm alone and bored today. Can you suggest some fun activities?"

[0835] Hardware and software used

[0836] This system is realized using a smartphone, a server, an internet connection, the Google Cloud NLP API, and OpenAI's GPT-3 model. Specifically, the smartphone receives input from the user, converts it into a data format, and sends it to the server via the internet connection. The server analyzes the received data and generates and returns an appropriate answer or reply. The smartphone displays or plays the returned data to the user.

[0837] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0838] Homework Help

[0839] Processing steps and specific explanations

[0840] Step 1:

[0841] A user launches a smartphone app and selects the "Homework Help" menu, which displays the in-app "Homework Help" interface.

[0842] Input: Launching the smartphone app

[0843] Output: Homework help interface displayed

[0844] Step 2:

[0845] The user enters the question in text format, such as "I don't know how to solve the equation," and taps the "Send" button. This allows the input content to be managed within the app.

[0846] Input: User text input

[0847] Output: Keep input

[0848] Step 3:

[0849] The device receives the user's question and converts it into JSON format, which structures the data and makes it ready for transmission.

[0850] Input: User text input

[0851] Output: JSON format data

[0852] Specific operation:Conversion process to { "question": "I don't know how to solve the equation"}

[0853] Step 4:

[0854] The terminal sends the converted JSON data to the server as an HTTP POST request. The data is sent using the communication protocol.

[0855] Input: JSON format data

[0856] Output: An HTTP POST request is made to the server.

[0857] Step 5:

[0858] The server receives the HTTP request and retrieves the JSON data. The server then uses the Google Cloud NLP API to analyze the question, tokenizing the content and extracting keywords.

[0859] Input: JSON data from an HTTP POST request

[0860] Output: Analysis results (keywords and context)

[0861] Specific operation: Using natural language processing technology, important keywords such as "equation" and "solution method" are extracted from the question.

[0862] Step 6:

[0863] The server searches for appropriate explanatory articles and specific calculation methods based on the analysis results and generates an appropriate answer, which is then converted back into JSON format.

[0864] Input: Keywords of analysis results

[0865] Output: JSON formatted response data

[0866] Specific operation: Generate data including the URL of the related explanatory article and the calculation method { "url": "https: / / example.com / article", "method": "x = 7 - 3"}

[0867] Step 7:

[0868] The server sends the generated JSON data to the terminal as an HTTP response.

[0869] Input: JSON formatted response data

[0870] Output: Sent as an HTTP response

[0871] Step 8:

[0872] The device receives the response, parses the JSON data, and displays it visually to the user, such as links to explanatory articles and calculation instructions.

[0873] Input: JSON data of the HTTP response from the server

[0874] Output: A visual display to the user

[0875] Specific operation: Display an explanatory article link and calculation method { "url": "https: / / example.com / article", "method": "x = 7 - 3"}

[0876] Step 9:

[0877] Users can tap the link to the displayed explanatory article to view a detailed explanation, and by accessing specific learning content, they can solve the problem.

[0878] Input: Tap the displayed link

[0879] Output: View detailed explanation

[0880] Communication partner

[0881] Processing steps and specific explanations

[0882] Step 1:

[0883] The user starts the smartphone app and selects the "Communication Partner" menu, which displays a chat interface with the communication partner.

[0884] Input: Launching the smartphone app

[0885] Output: Display of communication partner interface

[0886] Step 2:

[0887] The user enters a message such as "Tell me what you'd like to do today" and taps the "Send" button. The input content is then managed within the app.

[0888] Input: User text input

[0889] Output: Keep input

[0890] Step 3:

[0891] The device receives the user's message content and converts it to JSON format, which structures the data and makes it ready for transmission.

[0892] Input: User text input

[0893] Output: JSON format data

[0894] Specific operation: Conversion process to { "message": "Tell me what you'd like to do today"}

[0895] Step 4:

[0896] The terminal sends the converted JSON data to the server as an HTTP POST request. The data is sent using the communication protocol.

[0897] Input: JSON format data

[0898] Output: An HTTP POST request is made to the server.

[0899] Step 5:

[0900] The server receives the HTTP request and retrieves the JSON data. The server then analyzes the message content using OpenAI's GPT-3 model. This analysis allows the server to understand the user's intent and derive an appropriate response.

[0901] Input: JSON data from an HTTP POST request

[0902] Output: Analysis results (user intent and appropriate response)

[0903] Specific operation: Using natural language processing technology, the system analyzes the message content and suggests appropriate activities in response to the question, "What would you like to do today?"

[0904] Step 6:

[0905] The server generates a response based on the parsing results, including the suggestion, "How about playing Animal Crossing?" The response is then converted back to JSON format.

[0906] Input: Intention of analysis results and proposals

[0907] Output: Reply data in JSON format

[0908] Specific behavior: Generate reply data such as "How about playing Animal Crossing?" { "reply": "How about playing Animal Crossing?"}

[0909] Step 7:

[0910] The server sends the generated JSON data to the terminal as an HTTP response.

[0911] Input: Reply data in JSON format

[0912] Output: Sent as an HTTP response

[0913] Step 8:

[0914] The device receives the response, parses the JSON data, and displays or plays the suggestion to the user visually or audibly.

[0915] Input: JSON data of the HTTP response from the server

[0916] Output: Visual or audio display / playback to the user

[0917] Specific behavior: Display or play a suggestion saying "How about playing Animal Crossing?"

[0918] Step 9:

[0919] The user sees the suggestions and decides to play Animal Crossing. By actually doing the suggested activities, they spend their time meaningfully while at home.

[0920] Input: Confirm and select the proposed content

[0921] Output: Activity execution

[0922] In this way, the system of the present invention allows users to efficiently complete their homework alone at home and also provides support for learning and alleviates feelings of loneliness by interacting with appropriate communication partners.

[0923] (Application example 1)

[0924] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0925] When children in dual-income households are left at home alone, they need help with their homework and appropriate communication partners to help them spend their alone time meaningfully. However, current systems lack the means to quickly and accurately obtain information when customers ask questions about products, especially when shopping in physical stores. This leaves many customers, including children, with difficulty in selecting and making decisions about products.

[0926] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0927] In this invention, the server includes a means for inputting homework questions and product information, a means for analyzing the input questions and product information and generating appropriate answers and information, and a means for providing the generated answers and information to a user. This not only enables children in dual-income households to receive help with their homework, but also enables customers in physical stores to quickly and accurately obtain appropriate information when asking questions about products.

[0928] The "means for inputting homework questions" refers to an interface that allows a user to input homework questions in text format using a device such as a smartphone or tablet.

[0929] The "means for analyzing the content of the entered question and generating an appropriate answer" refers to an algorithm and process that uses natural language processing technology to analyze the content of the question entered by the user and generate an answer that is highly relevant to the question.

[0930] The "means for providing the generated answer to the user" is an interface that displays or plays the analyzed and generated answer visually or audibly on the user's device.

[0931] The "means for inputting product information" is an interface through which customers in a physical store can input questions about product details and characteristics.

[0932] "Means for analyzing input product information and generating appropriate information" refers to algorithms and processes that use natural language processing technology to analyze questions and requests about products input by customers and generate relevant product information.

[0933] The "means for providing the generated product information to the user" is an interface that displays or plays back the analyzed and generated product information on the customer's device.

[0934] The "means for inputting chat messages while the user is at home alone" refers to an interface that allows the user to input messages in chat format using a smartphone or tablet while at home alone.

[0935] The "means for analyzing input messages and generating appropriate replies" refers to algorithms and processes that use natural language processing technology to analyze chat messages input by users and generate highly relevant replies to those messages.

[0936] The "means for providing the generated reply to the user" is an interface that displays or plays the parsed and generated reply on the user's device.

[0937] "Means for converting questions and product information into JSON format and sending it to a server" refers to the process and means for converting questions and product information entered by a user into a standardized JSON format and sending it to a server via a network as an HTTP request.

[0938] "Means for analyzing JSON format responses, product information, and replies from the server and displaying or playing them back to the user" refers to an interface that analyzes JSON format data received from the server and displays or plays it back on the device in a form that is easily understandable to the user.

[0939] "Means for analyzing question content and product information through natural language processing using a generative AI model" refers to algorithms and processes that use a generative AI model (e.g., GPT-3) to analyze question content and product information using advanced natural language processing technology.

[0940] This invention is a system that provides homework help for children in dual-income households who are left home alone, and also provides fast and accurate information when customers in physical stores have questions about products. The system consists of an application installed on a smartphone or tablet and a server that processes requests from these applications.

[0941] Hardware and software used

[0942] Hardware:

[0943] A smartphone or tablet as the user device

[0944] Data center or cloud computing infrastructure as a server

[0945] software:

[0946] Client side: Mobile application developed using React Native

[0947] Server side: Server application developed using Python and Flask

[0948] Natural language processing engine: OpenAI's GPT-3 model

[0949] Data processing and calculation flow

[0950] When a user types a question about homework or product information, the application installed on the smartphone or tablet receives the question. The application then converts the input question into JSON format and sends it as an HTTP POST request to a server. The server receives the request and uses a generative AI model (e.g., GPT-3) to analyze the question.

[0951] The server searches a database or knowledge base for appropriate answers and product information for the question, and returns the generated answers and information in JSON format to the application. The application then analyzes the received JSON data and displays or plays it back to the user visually or audibly. This process allows the user to obtain information quickly and accurately.

[0952] Specific examples

[0953] Example questions for user homework:

[0954] When a user submits a question by entering text such as "I don't know how to solve an equation," the server analyzes the question, searches for appropriate explanatory articles and calculation methods, and provides them to the user.

[0955] Example prompt sentence:

[0956] User asked: Are these sneakers waterproof?

[0957] AI Response:

[0958] Based on this, the server provides information such as, "These sneakers are made of Gore-Tex material and are waterproof."

[0959] Example prompts for the generative AI model used

[0960] User asked: Are these sneakers waterproof?

[0961] AI Response:

[0962] Specific server processing

[0963] The server utilizes natural language processing technology and generative AI models to generate appropriate answers to user questions and product information. Specifically, it analyzes the user's text input and understands their intent. It then retrieves the information most relevant to that intent from the database and generative AI model and provides it to the user in an appropriate format.

[0964] This system not only allows users to receive help with their homework, but also provides quick and accurate information for selecting products in physical stores, allowing children in dual-income households and customers in physical stores to spend their time more meaningfully.

[0965] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0966] Step 1:

[0967] The user launches the application on their smartphone or tablet and enters a question about homework or a product. The entered question is recorded in text format on the device.

[0968] input:

[0969] Questions typed in by users (e.g., "I don't know how to solve an equation" or "Are these sneakers waterproof?")

[0970] output:

[0971] Text questions recorded on the device

[0972] Specific behavior:

[0973] Using the application's interface, users enter their homework or product question and tap the "Submit" button.

[0974] Step 2:

[0975] The terminal converts the questions entered by the user into JSON format.

[0976] input:

[0977] Text-based questions

[0978] output:

[0979] Question content converted to JSON format

[0980] Specific behavior:

[0981] A script in the application converts the text question into JSON format (e.g., {"question": "I don't know how to solve the equation"}).

[0982] Step 3:

[0983] The device sends the converted JSON-formatted question to the server as an HTTP POST request.

[0984] input:

[0985] JSON formatted question

[0986] output:

[0987] HTTP POST request sent to the server

[0988] Specific behavior:

[0989] The device issues an HTTP request to the server and sends the question (e.g., a POST request with a JSON payload).

[0990] Step 4:

[0991] The server receives the HTTP POST request and parses the JSON-formatted question.

[0992] input:

[0993] Received question in JSON format

[0994] output:

[0995] Analyzed questions

[0996] Specific behavior:

[0997] The server parses the request and gets the question (e.g., parses the JSON payload using a JSON parsing library).

[0998] Step 5:

[0999] The server inputs the analyzed question into a generative AI model (e.g., GPT-3) to generate an appropriate answer.

[1000] input:

[1001] Analyzed questions

[1002] output:

[1003] Answers generated by generative AI models

[1004] Specific behavior:

[1005] The server issues an API request to the generated AI model and generates an answer based on the question (e.g., an API call to GPT-3).

[1006] Step 6:

[1007] The server generates a JSON response containing the generated answer and sends it to the device as an HTTP response.

[1008] input:

[1009] Generated Answer

[1010] output:

[1011] JSON-formatted response

[1012] Specific behavior:

[1013] The server converts the answer into JSON format and sends it to the terminal as an HTTP response (e.g., {"answer": "To solve the equation..."}).

[1014] Step 7:

[1015] The device receives and parses the JSON response from the server.

[1016] input:

[1017] JSON format response received from the server

[1018] output:

[1019] Analyzed answers

[1020] Specific behavior:

[1021] The device parses the received JSON data and formats it into the format required for display to the user.

[1022] Step 8:

[1023] The terminal displays the analyzed answer content to the user visually or audibly.

[1024] input:

[1025] Analyzed answers

[1026] output:

[1027] Answers presented to the user

[1028] Specific behavior:

[1029] The device displays the answer in a user interface so that the user can review it (e.g., displaying the answer in a text view).

[1030] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1031] The present invention combines a smartphone application system that provides homework help and communication partners for children in dual-income households who are left home alone with an emotion engine that recognizes the user's emotions. Specific embodiments of the present invention are described below.

[1032] Homework Help

[1033] overview

[1034] Users (children) can input questions about their homework through a smartphone app and receive appropriate answers. The system analyzes the questions entered by the user and generates and provides appropriate answers. It also utilizes an emotion engine that recognizes the user's emotions and responds based on the user's emotional state.

[1035] Specific operation example

[1036] 1. The user launches the smartphone app and selects the "Homework Help" menu.

[1037] 2. The user types the text "I don't know how to solve the equation" and taps the "Submit" button.

[1038] 3. The device receives the user's question and converts it into JSON format.

[1039] 4. The emotion engine recognizes emotions from the user's facial expressions and voice and generates emotion data.

[1040] 5. The device sends the question and emotion data to the server as an HTTP POST request.

[1041] 6. The server receives the request and analyzes the question and sentiment data, for example using natural language processing to understand the question and tailor the answer taking into account the sentiment data.

[1042] 7. The server generates JSON data including the URL of the appropriate explanatory article and specific calculation methods.

[1043] 8. The server sends the generated JSON data to the terminal as an HTTP response.

[1044] 9. The device receives the response and displays it visually to the user.

[1045] 10. The user taps on the link to the explanatory article displayed to view a detailed explanation.

[1046] Communication partner

[1047] overview

[1048] While the user is home alone, they can chat with a virtual communication partner through a smartphone app. The system analyzes the user's input messages, generates and provides appropriate replies, and uses an emotion engine to respond based on the user's emotional state.

[1049] Specific operation example

[1050] 1. The user launches the smartphone app and selects the "Communication Partner" menu.

[1051] 2. The user types the text "Tell me what you'd like to do today" and taps the "Send" button.

[1052] 3. The device converts the message into JSON format and sends it to the server as an HTTP POST request.

[1053] 4. The emotion engine recognizes emotions from the user's facial expressions and voice and generates emotion data.

[1054] 5. The device sends the message content and emotion data to the server.

[1055] 6. The server receives the request and analyzes the message content and emotional data, for example using natural language processing techniques to generate an appropriate reply and adjust the reply content taking the emotional data into account.

[1056] 7. The server generates a JSON response containing the suggestion, "How about playing Animal Crossing?"

[1057] 8. The server sends the generated JSON data to the terminal as an HTTP response.

[1058] 9. The device receives the response and presents it to the user visually or audibly.

[1059] 10. The user sees the suggestions and chooses to play Animal Crossing.

[1060] This system allows children who are alone at home to receive support with their schoolwork and avoid feeling lonely through conversations with virtual communication partners. Furthermore, the introduction of an emotion engine enables flexible responses based on the user's emotional state, providing more personalized support.

[1061] The processing flow will be explained below.

[1062] Homework Help

[1063] Program processing

[1064] Step 1:

[1065] The user launches the smartphone app and selects the "Homework Help" menu.

[1066] Step 2:

[1067] A user texts in a homework question, for example, "I don't know how to solve the equation."

[1068] Step 3:

[1069] The user taps the "Send" button.

[1070] Step 4:

[1071] The terminal receives the user's question.

[1072] Step 5:

[1073] In order for the device to analyze the user's facial expressions and voice and recognize emotions, it activates an emotion engine and acquires emotion data.

[1074] Step 6:

[1075] The device converts the question content and emotion data into JSON format.

[1076] Step 7:

[1077] The device sends the converted JSON-formatted question and emotion data to the server as an HTTP POST request.

[1078] Step 8:

[1079] The server receives the HTTP POST request, analyzes the question and emotional data, and uses natural language processing technology to understand the intent of the question and generate an answer that takes the emotional data into account.

[1080] Step 9:

[1081] The server searches a database or calls external APIs to generate an answer that fits the question, for example adding detailed explanations or encouraging messages if the user is confused.

[1082] Step 10:

[1083] The server generates JSON data containing the URL of the appropriate explanatory article and the specific answer.

[1084] Step 11:

[1085] The server sends the generated JSON data to the terminal as an HTTP response.

[1086] Step 12:

[1087] The device receives the HTTP response and parses the JSON data.

[1088] Step 13:

[1089] The device displays the analyzed data in a format that the user can visually confirm, such as a link to an explanatory article or a message of support.

[1090] Step 14:

[1091] The user taps the link to the explanatory article displayed to check the detailed content.

[1092] Communication partner

[1093] Program processing

[1094] Step 1:

[1095] The user launches the smartphone app and selects the "Communication Partner" menu.

[1096] Step 2:

[1097] The user types the text "Tell me what you'd like to do today" into the chat window.

[1098] Step 3:

[1099] The user taps the "Send" button.

[1100] Step 4:

[1101] The terminal receives the user's message.

[1102] Step 5:

[1103] In order for the device to analyze the user's facial expressions and voice and recognize emotions, it activates an emotion engine and acquires emotion data.

[1104] Step 6:

[1105] The device converts the message content and emotion data into JSON format.

[1106] Step 7:

[1107] The device sends the converted JSON-formatted message content and emotion data to the server as an HTTP POST request.

[1108] Step 8:

[1109] The server receives the HTTP POST request, analyzes the message content and emotional data, and uses natural language processing technology to understand the intent of the message and generate a reply that takes the emotional data into account.

[1110] Step 9:

[1111] The server searches a relevant database to generate an appropriate reply, and generates suitable reply suggestions, such as activity suggestions if the user is bored.

[1112] Step 10:

[1113] The server generates a JSON response containing the suggestion, "How about playing Animal Crossing?"

[1114] Step 11:

[1115] The server sends the generated JSON data to the terminal as an HTTP response.

[1116] Step 12:

[1117] The device receives the HTTP response and parses the JSON data.

[1118] Step 13:

[1119] The device presents the analyzed data to the user visually or audibly, such as suggestion messages or specific steps for an activity.

[1120] Step 14:

[1121] The user sees the suggestions and chooses to play Animal Crossing.

[1122] Example 2

[1123] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1124] In situations where children in dual-income households are left home alone, there is a need to provide appropriate support for homework questions and communication methods to reduce feelings of loneliness. It is also important that this support and communication take into account the child's emotional state, and personalized responses are required.

[1125] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for inputting homework questions; means for analyzing the input questions and generating appropriate answers; means for providing the generated answers to the user; means having an engine for recognizing the user's emotions and generating emotion data; means for transmitting the questions and emotion data and receiving answers from the server; means for inputting chat messages while the user is at home alone; means for analyzing the input messages and generating appropriate replies; means for providing the generated replies to the user; means for transmitting the message contents and emotion data and receiving the replies from the server; means for converting the questions and messages into a data format and transmitting them; and means for analyzing the answers and replies in data format from the server and displaying or playing them back to the user. This allows children to receive support with their homework while at home alone and helps them feel less lonely through virtual communication.

[1126] The "means for inputting homework questions" is a system component that provides an interface for users to input homework questions as text.

[1127] The "means for analyzing the entered question and generating an appropriate answer" is a component of the system that uses natural language processing technology to analyze the homework question entered by the user and generate the optimal answer to that question.

[1128] A "means for providing a generated answer to a user" is a system component that provides an interface for presenting the system-generated answer to a user visually or audibly.

[1129] "Means having an engine for recognizing a user's emotions and generating emotional data" refers to a system component that includes an engine that uses a camera or microphone to analyze a user's facial expressions and tone of voice, identify their emotional state, and generate emotional data.

[1130] The "means for sending question content and emotion data to a server and receiving a response from the server" refers to a system component that sends a question input by a user and generated emotion data to a server via an HTTP POST request and receives the response data received from the server.

[1131] The "means for inputting chat messages while the user is away" is a system component that provides an interface for the user to input chat messages in text format.

[1132] The "means for analyzing input messages and generating appropriate replies" is a component of a system that uses natural language processing technology to analyze chat messages input by users and generate optimal replies to those messages.

[1133] A "means for providing a generated reply to a user" is a system component that provides an interface for presenting a system-generated reply to a user visually or audibly.

[1134] "Means for transmitting message content and emotion data to a server and receiving a reply from the server" refers to a system component for transmitting message content entered by a user and generated emotion data to a server and receiving reply data received from the server.

[1135] "Means for converting questions and messages into a data format and sending them to the server" refers to a system component that converts questions and messages entered by users into a data format such as JSON and sends them to the server.

[1136] "Means for analyzing answers and replies in data format from the server and displaying or playing them back to the user" refers to a system component that analyzes answers and replies provided in data format such as JSON received from the server and displays or plays them back to the user visually or audibly.

[1137] The present invention combines a smartphone application system that provides homework help and communication partners for children in dual-income households who are left home alone with an emotion engine that recognizes the user's emotions. Specific embodiments of the present invention are described below.

[1138] Homework help feature

[1139] Basic configuration

[1140] This system uses a smartphone app to input questions related to homework and provides a series of functions to obtain appropriate answers. The main elements are a means for inputting the user's question, a means for analyzing the question and generating an appropriate answer, a means for providing the generated answer to the user, and a means for generating emotion data using an emotion engine. This system uses HTTPS communication to send the user's question and emotion data to the server and receive the answer from the server. It is also configured to convert the question and emotion data into JSON format and send it, and to analyze the answer received from the server and display it to the user.

[1141] Operation overview

[1142] 1. The user launches the smartphone app and selects the "Homework Help" menu.

[1143] 2. The user types the text "I don't know how to solve the equation" and taps the "Submit" button.

[1144] 3. The device converts the input question into JSON format and uses an emotion engine to recognize emotions from the user's facial expressions and voice, generating emotion data. The emotion engine can use, for example, Microsoft Azure's emotion analysis API.

[1145] 4. The device sends the question and emotion data to the server as an HTTP POST request, using a communication library such as OkHttp.

[1146] 5. The server analyzes the received request and generates an answer appropriate to the question. For natural language processing, it uses an NLP model, such as Hugging Face.

[1147] 6. The server generates a response and sends it to the device in JSON format.

[1148] 7. The terminal analyzes the received response and displays it visually to the user.

[1149] Examples of concrete examples and prompts

[1150] Examples:

[1151] User types, "Help me with my math homework today!"

[1152] The device uses an emotion engine to recognize when the user is feeling stressed.

[1153] The server provides detailed explanatory articles in a gentle tone.

[1154] Example prompt sentence:

[1155] If your child types "Help me with my math homework today" into your app, generate an appropriate response that takes into account the stress your child may be under.

[1156] Communication partner function

[1157] Basic configuration

[1158] This system provides a function that enables users to chat with virtual communication partners through a smartphone app while they are home alone. The main elements are a means for inputting messages, a means for analyzing the message content and generating an appropriate reply, a means for providing the generated reply to the user, and a means for generating emotion data using an emotion engine. The system is configured to send the message content and emotion data to a server, analyze the replies received from the server, and display or play them back to the user.

[1159] Operation overview

[1160] 1. The user launches the smartphone app and selects the "Communication Partner" menu.

[1161] 2. The user types the text "Tell me what you'd like to do today" and taps the "Send" button.

[1162] 3. The device converts the message content into JSON format and sends it to the server as an HTTP POST request.

[1163] 4. The emotion engine generates emotion data from the user's facial expressions and voice. For emotion analysis, it uses the IBM Watson API, for example.

[1164] 5. The device sends the message content and emotion data to the server.

[1165] 6. The server analyzes the received request and generates an appropriate reply, for example using OpenAI GPT-3.

[1166] 7. The server sends the appropriate reply in JSON format to the device.

[1167] 8. The device analyzes the reply received and presents it to the user visually or audibly.

[1168] Examples of concrete examples and prompts

[1169] Examples:

[1170] A user types, "What would be fun to do today?"

[1171] The device uses its emotion engine to recognize that the user is feeling a little lonely.

[1172] "Animal Crossing is a fun game," the server suggested.

[1173] Example prompt sentence:

[1174] When a child types into your app, "What would be fun to do today?", consider that they might be feeling a little lonely and generate fun suggestions. Include popular games and activities in your suggestions.

[1175] This system allows children who are alone at home to receive support with their schoolwork and avoid feeling lonely by interacting with virtual communication partners. The introduction of an emotion engine also enables flexible responses based on the user's emotional state, providing more personalized support.

[1176] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1177] Homework help feature

[1178] Step 1:

[1179] The user launches the smartphone app and selects the "Homework Help" menu.

[1180] Specific operation: The user taps the app from the smartphone home screen to launch it, and selects the "Homework Help" menu on the initial screen.

[1181] Input: User taps.

[1182] Output: Homework help menu will be displayed.

[1183] Step 2:

[1184] The user types the text "I don't know how to solve the equation" and taps the "Submit" button.

[1185] Specific actions: The user uses the smartphone keyboard to enter a question into the text box and taps the "Send" button.

[1186] Input: The question entered by the user is "I don't know how to solve an equation."

[1187] Output: The question is sent to the app.

[1188] Step 3:

[1189] The device converts the entered question into JSON format.

[1190] What it does: The app on the device reads the user's text input and converts it to JSON format.

[1191] Input: The question entered by the user is "I don't know how to solve an equation."

[1192] Data processing: Converting to JSON format (e.g., {"question": "I don't know how to solve the equation"}).

[1193] Output: Data converted to JSON format.

[1194] Step 4:

[1195] The emotion engine recognizes emotions from the user's facial expressions and voice and generates emotion data.

[1196] Specific operation: Using the device's camera and microphone, the system analyzes the user's facial expressions and voice to generate emotion data. For example, it uses Microsoft Azure's emotion analysis API as an emotion engine.

[1197] Input: User facial and voice data obtained from the camera and microphone.

[1198] Data operations: Identify emotional states and generate emotion data (e.g., {"emotion": "frustrated"}).

[1199] Output: Emotion data.

[1200] Step 5:

[1201] The device sends the question and emotion data to the server as an HTTP POST request.

[1202] Specific operation: The device sends the generated question and emotion data together to the server. A communication library (e.g., OkHttp) is used for communication.

[1203] Input: Question and emotion data in JSON format (e.g., {"question": "I don't know how to solve the equation", "emotion": "frustrated"}).

[1204] Output: An HTTP POST request is sent to the server.

[1205] Step 6:

[1206] The server receives the request and analyzes the question and emotion data.

[1207] What happens: The server receives the request and uses natural language processing techniques (e.g., the Hugging Face NLP model) to analyze the question and sentiment data.

[1208] Input: Submitted question and sentiment data.

[1209] Data Computing: Uses natural language processing to understand questions and tailor answers based on sentiment data.

[1210] Output: The appropriate answer data (e.g., an explanation of how to solve the equation).

[1211] Step 7:

[1212] The server generates JSON-formatted data including the URL of the appropriate explanatory article and specific calculation methods.

[1213] Specific operation: The server generates an appropriate answer and creates JSON-formatted data including the URL of the explanatory article and the specific calculation method.

[1214] Input: Analysis result data.

[1215] Data processing: Generate answer data in JSON format (e.g., {"answer": "To solve the equation...", "url": "http: / / example.com / solution"}).

[1216] Output: Response data in JSON format.

[1217] Step 8:

[1218] The server sends the generated JSON data to the terminal as an HTTP response.

[1219] Specific operation: The server generates data and sends it to the terminal as an HTTP response.

[1220] Input: Response data in JSON format.

[1221] Output: The data sent as the HTTP response.

[1222] Step 9:

[1223] The terminal receives the response and visually displays it to the user.

[1224] Specific operation: Analyzes the data received by the device and displays it in an easy-to-read format for the user. Uses a UI library (e.g., Android's RecyclerView).

[1225] Input: The JSON data sent as an HTTP response.

[1226] Data processing: Parsing and visualizing JSON data.

[1227] Output: The explanation or calculation displayed to the user.

[1228] Step 10:

[1229] The user taps the link of the displayed explanatory article to view a detailed explanation.

[1230] What happens: The user taps on a link to open a browser.

[1231] Input: User taps.

[1232] Output: A detailed explanatory article displayed in your browser.

[1233] Communication partner function

[1234] Step 1:

[1235] The user launches the smartphone app and selects the "Communication Partner" menu.

[1236] Specific behavior: The user opens the app and selects the "People to communicate with" menu.

[1237] Input: User taps.

[1238] Output: The communication partner menu is displayed.

[1239] Step 2:

[1240] The user types the text "Tell me what you'd like to do today" and taps the "Send" button.

[1241] Specific behavior: The user types "What would you like to do for fun today?" into the text box and taps the "Send" button.

[1242] Input: User-entered message: "Tell me what you'd like to do today."

[1243] Output: The message content is sent to the app.

[1244] Step 3:

[1245] The device converts the message content into JSON format and sends it to the server as an HTTP POST request.

[1246] Specific operation: The device converts the input message into JSON format and sends it to the server.

[1247] Input: User-entered message: "Tell me what you'd like to do today."

[1248] Data processing: Converting to JSON format (e.g., {"message": "Tell me what you'd like to do today"}).

[1249] Output: The data is converted to JSON format and sent to the server.

[1250] Step 4:

[1251] The emotion engine generates emotion data from the user's facial expressions and voice.

[1252] Specific operation: The emotion engine recognizes emotions through the camera and microphone and generates data on them.

[1253] Input: User facial and voice data obtained from the camera and microphone.

[1254] Data operations: Identify emotional states and generate emotion data (e.g., {"emotion": "lonely"}).

[1255] Output: Emotion data.

[1256] Step 5:

[1257] The device sends the message content and emotion data to the server.

[1258] Specific operation: The device sends the acquired message content and emotion data to the server.

[1259] Input: Message content and emotion data in JSON format (e.g., {"message": "Tell me what you'd like to do today", "emotion": "lonely"}).

[1260] Output: An HTTP POST request is sent to the server.

[1261] Step 6:

[1262] The server receives the request and analyzes the message content and emotion data.

[1263] Specific operation: The server analyzes the received data and generates appropriate suggestions. For natural language processing, OpenAI GPT-3 is used as an example.

[1264] Input: Message content and sentiment data.

[1265] Data Computation: Analysis and response generation using natural language processing.

[1266] Output: Appropriate reply data (e.g., fun activity suggestions).

[1267] Step 7:

[1268] The server generates a JSON representation of the appropriate response.

[1269] Specific operation: The server converts the proposal based on the analysis results into JSON format data.

[1270] Input: Analysis results.

[1271] Data processing: Generate reply data in JSON format (e.g., {"reply": "How about playing Animal Crossing?"}).

[1272] Output: Reply data in JSON format.

[1273] Step 8:

[1274] The server sends the generated JSON data to the terminal as an HTTP response.

[1275] Specific operation: The data generated by the server is sent to the terminal as an HTTP response.

[1276] Input: Reply data in JSON format.

[1277] Output: The data sent as the HTTP response.

[1278] Step 9:

[1279] The terminal receives the response and presents it to the user visually or audibly.

[1280] Specific operation: The device analyzes the reply data received and displays it on the screen or reads it aloud. It uses a UI library or text-to-speech library.

[1281] Input: The JSON data sent as an HTTP response.

[1282] Data processing: Parse JSON data and display or output it as audio.

[1283] Output: Screen display and audio notification.

[1284] Step 10:

[1285] The user sees the suggestions and chooses to play Animal Crossing.

[1286] Specific behavior: The user accepts the suggestion and selects an activity.

[1287] Input: User's choice.

[1288] Output: The execution of the selected activity.

[1289] Through the above processing steps, children can receive help with their homework while at home alone, and can avoid feeling lonely through virtual communication.

[1290] (Application example 2)

[1291] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1292] In recent years, the number of dual-income households has increased, resulting in more opportunities for children to be left home alone. However, in these situations, children may have questions or feel anxious about their homework. In particular, there is a need for communication methods to receive appropriate answers to questions about homework and to reduce feelings of loneliness. There is also a need for systems that can be used safely for small payments. It is hoped that these issues can be resolved and an environment in which children can spend their time with peace of mind will be provided.

[1293] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting homework questions, means for analyzing the input questions and generating appropriate answers, means for providing the generated answers to the user, means for recognizing the user's emotions and generating emotion data, and means for adjusting a response based on the generated emotion data. This makes it possible to provide a sense of security to children who are home alone through support with homework and communication, and to enable safe small-amount payments.

[1294] "Means for inputting homework questions" refers to a function that allows users to input questions about homework via devices such as smartphones or personal computers.

[1295] "Means for analyzing the content of the question entered and generating an appropriate answer" refers to a function that analyzes the content of the question entered by the user using natural language processing technology, etc., and generates an appropriate answer.

[1296] The "means for providing the generated answer to the user" is a function for presenting the answer generated through analysis to the user visually or audibly.

[1297] The "means for recognizing the user's emotions and generating emotion data" is a function for recognizing emotions from the user's facial expressions, voice, etc., and generating emotion data based on the emotions.

[1298] The "means for adjusting a response based on the generated emotion data" is a function for taking the optimal response to the user's emotional state based on the generated emotion data.

[1299] "Means for inputting chat messages while the user is at home alone" is a function for inputting messages to chat with a virtual communication partner while the user is at home alone.

[1300] The "means for analyzing input messages and generating appropriate replies" refers to a function that analyzes chat messages input by users using natural language processing technology, etc., and generates appropriate replies.

[1301] The "means for providing the generated reply to the user" is a function for presenting the generated reply to the user and continuing the dialogue with the virtual communication partner.

[1302] The "means for generating emotion data and transmitting the emotion data to a server" is a function for transmitting emotion data generated by an emotion engine to a server.

[1303] "Means for converting questions and messages into JSON format and sending them to the server" is a function for converting questions and messages entered by the user into JSON format and sending them to the server.

[1304] "Means for analyzing JSON format answers and replies from the server and displaying or playing them back to the user" refers to a function that analyzes JSON format answers and replies received from the server and provides them to the user visually or audibly.

[1305] This invention combines a smartphone application system that provides homework help and communication partners for children in dual-income households while they are home alone with an emotion engine that recognizes the user's emotions. Specific embodiments will be described in detail below.

[1306] 1. Homework Help

[1307] Program processing

[1308] The terminal accepts questions about homework entered by the child. The questions are entered in text format through the "means for entering homework question content." The terminal analyzes the questions using a natural language processing engine, generates an appropriate answer, and provides it to the user through the "means for analyzing entered question content and generating an appropriate answer." In doing so, the terminal uses an emotion engine to analyze the user's facial expressions and voice and generate emotion data. The emotion data generated by the "means for recognizing user emotions and generating emotion data" is sent to the server, and the answer is adjusted based on the analysis results. The resulting answer is displayed to the user through the "means for providing the generated answer to the user."

[1309] Hardware and software used

[1310] Devices: Smartphones, tablets

[1311] Natural language processing engines: TensorFlow, e.g., GPT-based models

[1312] Emotion Engine: EmotionRecognition Library

[1313] Specific examples

[1314] The user types, "I don't know how to solve the equation."

[1315] The emotion engine analyzes the user's facial expressions and detects stress.

[1316] The server generates an answer presented in a gentler tone based on the question content and stress emotional data.

[1317] 2. Communication Partner

[1318] Program processing

[1319] The terminal accepts chat messages entered by the user, and the messages entered through the "means for entering chat messages while the user is away" are analyzed using a natural language processing engine. The server generates an appropriate reply and provides it to the user through the "means for analyzing the entered message and generating an appropriate reply." At this time, an emotion engine recognizes emotions from the user's facial expressions and voice and generates emotion data. The reply content is adjusted based on the generated emotion data. The reply is presented to the user through the "means for providing the generated reply to the user."

[1320] Hardware and software used

[1321] Devices: Smartphones, tablets

[1322] Natural language processing engines: TensorFlow, e.g., GPT-based models

[1323] Emotion Engine: EmotionRecognition Library

[1324] Specific examples

[1325] The user types, "Tell me what you'd like to do today."

[1326] The emotion engine detects boredom from the user's tone of voice.

[1327] The server uses the suggestion to generate a reply: "How about playing Animal Crossing?"

[1328] 3. Examples of Electronic Payment Services

[1329] Program processing

[1330] The terminal accepts a payment request entered by the user. The payment details are converted to JSON format and sent to the server using the "means for converting questions and messages into JSON format and sending them to the server." The server then analyzes the user's emotions using an emotion engine and takes appropriate action. For example, if the user is feeling anxious, a notification is sent to the parent. Finally, the payment result is presented to the user using the "means for analyzing JSON-formatted answers and replies from the server and displaying or playing them back to the user."

[1331] Hardware and software used

[1332] Devices: Smartphones, tablets

[1333] Emotion Engine: EmotionRecognition Library

[1334] Notification System: Email Service API

[1335] Specific examples

[1336] The user types, "Please tell me which math textbook you recommend that costs less than 500 yen."

[1337] The emotion engine recognizes the user's tension.

[1338] The server uses emotional data from stressful situations to suggest recommended textbooks using reassuring language.

[1339] The above is an embodiment of the present invention. This system allows children in dual-income households to safely make small payments when they are left at home alone. Furthermore, by utilizing an emotion engine, personalized support is provided based on the emotional state of the user (child).

[1340] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1341] Step 1:

[1342] The user starts up the smartphone and asks for help with their homework or enters a chat message. Specifically, the user types, "I don't know how to solve this equation," and this question is sent to the device.

[1343] Step 2:

[1344] The terminal converts the input question content into JSON format for data analysis, structuring the user's input data and facilitating processing on the server side.

[1345] Step 3:

[1346] The emotion engine generates emotion data in real time from the user's facial expressions and voice. For example, it receives the user's facial expression images and voice data as input, performs emotion analysis based on them, and outputs emotion labels such as "stress" or "boredom."

[1347] Step 4:

[1348] The device sends the question content converted into JSON format and the generated emotion data to the server as an HTTP POST request. Specifically, the data format is a JSON object containing the question content and emotion data, which is sent in the body of the HTTP request.

[1349] Step 5:

[1350] The server analyzes the received request, extracts the question content and emotion data, and uses this data to analyze the question content using a natural language processing engine and generate an appropriate answer.

[1351] Step 6:

[1352] The server adjusts the generated response taking into account the emotional data. For example, if the user is feeling "stressed," the server adjusts the response to be gentler and more reassuring.

[1353] Step 7:

[1354] The server constructs the tailored answer in JSON format and sends it to the device as an HTTP response, which includes the answer to the question and a sentiment-sensitive message.

[1355] Step 8:

[1356] The terminal analyzes the received response and presents the answer data to the user visually or audibly, and the user views the presented answer and solves the homework problem.

[1357] Step 9:

[1358] If the user enters another question or chat message, the process repeats, for example, "Tell me what you'd like to do today."

[1359] Step 10:

[1360] The server generates an appropriate reply from the chatbot based on the new message entered and adjusts it again taking into account the emotional data. The device then displays the adjusted reply to the user.

[1361] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1362] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1363] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1364] [Third embodiment]

[1365] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1366] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1367] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1368] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1369] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1370] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1371] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1372] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1373] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1374] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1375] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1376] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1377] The present invention provides a system that uses a smartphone application to provide homework help and a communication partner for children in dual-income households who are left home alone. Specific embodiments of the system are described below.

[1378] Homework Help

[1379] overview

[1380] Users (children) can input questions about their homework through a smartphone app and receive appropriate answers. The system analyzes the questions entered by the user and generates and provides appropriate answers.

[1381] Specific operation example

[1382] 1. The user launches the smartphone app and selects the "Homework Help" menu.

[1383] 2. The user types the text "I don't know how to solve the equation" and taps the "Submit" button.

[1384] 3. The device receives the user's question and converts it into JSON format.

[1385] 4. The device sends the converted question to the server as an HTTP POST request.

[1386] 5. The server receives the request and analyzes the question, for example, using natural language processing technology to understand the question and search for related articles or calculation methods.

[1387] 6. Based on the analysis results, the server generates JSON format data including the URL of an appropriate explanatory article and specific calculation methods.

[1388] 7. The server sends the generated JSON data to the terminal as an HTTP response.

[1389] 8. The device receives the response and displays it visually to the user.

[1390] 9. The user taps on the link to the explanatory article displayed to view a detailed explanation.

[1391] Communication partner

[1392] overview

[1393] While a user is at home alone, they can chat with a virtual communication partner through a smartphone app. The system analyzes the user's input messages and generates and provides appropriate replies.

[1394] Specific operation example

[1395] 1. The user launches the smartphone app and selects the "Communication Partner" menu.

[1396] 2. The user types the text "Tell me what you'd like to do today" and taps the "Send" button.

[1397] 3. The device converts the message into JSON format and sends it to the server as an HTTP POST request.

[1398] 4. The server receives the request and analyzes the message content, for example using natural language processing techniques to understand the intent of the message and suggest appropriate activities.

[1399] 5. Based on the analysis results, the server generates JSON format data containing the suggestion, "How about playing Animal Crossing?"

[1400] 6. The server sends the generated JSON data to the terminal as an HTTP response.

[1401] 7. The device receives the response and displays or plays it back to the user visually or audibly.

[1402] 8. The user sees the suggestions and decides to play Animal Crossing.

[1403] This system allows children who are alone at home to receive support with their schoolwork and avoid feeling lonely by interacting with virtual communication partners.

[1404] The processing flow will be explained below.

[1405] Homework Help

[1406] Program processing

[1407] Step 1:

[1408] The user launches the smartphone app and selects the "Homework Help" menu.

[1409] Step 2:

[1410] A user texts in a homework question, for example, "I don't know how to solve the equation."

[1411] Step 3:

[1412] The user taps the "Send" button.

[1413] Step 4:

[1414] The terminal receives the user's question.

[1415] Step 5:

[1416] The question received by the device is converted into JSON format.

[1417] Step 6:

[1418] The device sends the converted JSON-formatted question to the server as an HTTP POST request.

[1419] Step 7:

[1420] The server receives the HTTP POST request and analyzes the question, specifically using natural language processing technology to understand the intent of the question.

[1421] Step 8:

[1422] The server searches a database or calls an external API to generate an answer that is appropriate for the question.

[1423] Step 9:

[1424] The server generates JSON data containing the URL of the appropriate explanatory article and the specific answer.

[1425] Step 10:

[1426] The server sends the generated JSON data to the terminal as an HTTP response.

[1427] Step 11:

[1428] The device receives the HTTP response and parses the JSON data.

[1429] Step 12:

[1430] The terminal displays the analyzed data in a format that the user can visually confirm.

[1431] Step 13:

[1432] For example, the user taps the link of the displayed explanatory article to check the detailed content.

[1433] Communication partner

[1434] Program processing

[1435] Step 1:

[1436] The user launches the smartphone app and selects the "Communication Partner" menu.

[1437] Step 2:

[1438] The user types the text "Tell me what you'd like to do today" into the chat window.

[1439] Step 3:

[1440] The user taps the "Send" button.

[1441] Step 4:

[1442] The terminal receives the user's message.

[1443] Step 5:

[1444] The terminal converts the received message into JSON format.

[1445] Step 6:

[1446] The device sends the converted JSON formatted message to the server as an HTTP POST request.

[1447] Step 7:

[1448] The server receives the HTTP POST request and analyzes the message content. Specifically, it uses natural language processing technology to understand the intent of the message.

[1449] Step 8:

[1450] The server searches a relevant database to generate an appropriate reply, or generates suitable reply candidates.

[1451] Step 9:

[1452] The server generates a JSON response containing the suggestion, "How about playing Animal Crossing?"

[1453] Step 10:

[1454] The server sends the generated JSON data to the terminal as an HTTP response.

[1455] Step 11:

[1456] The device receives the HTTP response and parses the JSON data.

[1457] Step 12:

[1458] The terminal presents the analyzed data to the user visually or audibly.

[1459] Step 13:

[1460] The user sees the suggestions and chooses to play Animal Crossing.

[1461] Example 1

[1462] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1463] In situations where children from dual-income households are left home alone, it is difficult to provide appropriate communication methods to support their schoolwork and alleviate loneliness. Furthermore, in order for children to efficiently complete their homework at home, they need immediate, easy-to-understand answers, but there is a lack of technology to adequately resolve this issue. Furthermore, there is a need for a system that facilitates smooth dialogue with virtual communication partners to alleviate children's feelings of loneliness.

[1464] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1465] In this invention, the server includes a means for inputting a question, a means for analyzing the input question, a means for generating an appropriate answer based on the analysis results, and a means for providing the generated answer to the user. This allows a child to efficiently complete their homework at home alone. The server also includes a means for inputting a message while the user is at home alone, a means for analyzing the input message, a means for generating an appropriate reply based on the analysis results, and a means for providing the generated reply to the user. This allows a child to feel less alone by interacting with a virtual communication partner. The server also includes a means for converting the input question and message into a data format and transmitting it to the server, and a means for analyzing the data-formatted answers and replies from the server and displaying or playing them back to the user. This allows for efficient and accurate responses to the user's questions and messages and for immediate information provision.

[1466] The "means for inputting questions" is an interface that allows the user to provide questions about homework to the system in the form of input such as text.

[1467] "Means for analyzing the content of the entered question" refers to technology that uses natural language processing technology to understand the question entered by the user and extract appropriate keywords and context.

[1468] "Means for generating appropriate answers" are algorithms and databases that provide relevant information and explanations based on the analyzed question content.

[1469] "Means for providing the generated answer to the user" refers to an interface for visually or audibly displaying and playing the answer generated by the system on the user's terminal.

[1470] The "means for inputting a message while the user is away" is an interface through which the user inputs a message to converse with the communication partner.

[1471] The "means for analyzing an input message" is a technology that analyzes the contents of the message using natural language processing technology and understands the intent and important elements of the message.

[1472] The "means for generating an appropriate reply" refers to an algorithm or database for generating an appropriate reply to the user based on the analysis results.

[1473] "Means for providing the generated reply to the user" means an interface for visually or audibly displaying and playing the reply generated by the system on the user's terminal.

[1474] "Means for converting the input question and message into a data format and sending it to the server" refers to a technology that converts the user's input into a data format such as JSON and sends it to the server via a network.

[1475] "Means for analyzing answers and replies in data format from the server and displaying or playing them back to the user" refers to technology that analyzes answer and reply data returned from the server on the user's terminal and displays or plays them back visually or audibly.

[1476] This invention is a system that provides learning support and communication partners for children in dual-income households who are left home alone. In particular, it is a smartphone application that combines an algorithm for analyzing questions and providing appropriate answers with technology for generating appropriate replies to user messages.

[1477] Homework Help

[1478] overview

[1479] Users (children) can input questions about their homework through a smartphone app and receive appropriate answers. The system analyzes the questions entered by the user and generates and provides appropriate answers.

[1480] Specific Embodiments

[1481] 1. The user launches the smartphone app and selects the "Homework Help" menu.

[1482] 2. The user enters the question in text format and taps the "Submit" button. For example, the user might enter "I don't know how to solve the equation."

[1483] 3. The device receives the question and converts it into JSON format.

[1484] 4. The device sends the converted question to the server as an HTTP POST request.

[1485] 5. The server uses the Google Cloud NLP API to analyze the question, extracting keywords such as "equation" and "solution method."

[1486] 6. Based on the analysis results, the server generates a response in the form of data, including appropriate explanatory articles and specific calculation methods.

[1487] 7. The server sends the generated response in the data format to the terminal as an HTTP response.

[1488] 8. The device receives the response and displays it visually to the user, for example, a link to an explanatory article or a calculation method.

[1489] 9. The user taps the displayed link to view a detailed explanation.

[1490] Specific examples

[1491] Example prompt: "I don't know how to solve a linear equation like x + 3 = 7. Please help."

[1492] Communication partner

[1493] overview

[1494] While a user is at home alone, they can chat with a virtual communication partner through a smartphone app. The system analyzes the user's input messages and generates and provides appropriate replies.

[1495] Specific Embodiments

[1496] 1. The user launches the smartphone app and selects the "Communication Partner" menu.

[1497] 2. The user types a message and taps the "Send" button. For example, "Tell me what you'd like to do today."

[1498] 3. The device converts the message into JSON format and sends it to the server as an HTTP POST request.

[1499] 4. The server uses OpenAI's GPT-3 model to analyze the message content and suggest appropriate activities.

[1500] 5. Based on the analysis results, the server generates a response in the form of data containing the suggestion, "How about playing Animal Crossing?"

[1501] 6. The server sends the reply in the generated data format to the terminal as an HTTP response.

[1502] 7. The device receives the response and displays or plays it back to the user visually or audibly.

[1503] 8. The user sees the suggestions and decides to play Animal Crossing.

[1504] Specific examples

[1505] Example prompt: "I'm alone and bored today. Can you suggest some fun activities?"

[1506] Hardware and software used

[1507] This system is realized using a smartphone, a server, an internet connection, the Google Cloud NLP API, and OpenAI's GPT-3 model. Specifically, the smartphone receives input from the user, converts it into a data format, and sends it to the server via the internet connection. The server analyzes the received data and generates and returns an appropriate answer or reply. The smartphone displays or plays the returned data to the user.

[1508] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1509] Homework Help

[1510] Processing steps and specific explanations

[1511] Step 1:

[1512] A user launches a smartphone app and selects the "Homework Help" menu, which displays the in-app "Homework Help" interface.

[1513] Input: Launching the smartphone app

[1514] Output: Homework help interface displayed

[1515] Step 2:

[1516] The user enters the question in text format, such as "I don't know how to solve the equation," and taps the "Send" button. This allows the input content to be managed within the app.

[1517] Input: User text input

[1518] Output: Keep input

[1519] Step 3:

[1520] The device receives the user's question and converts it into JSON format, which structures the data and makes it ready for transmission.

[1521] Input: User text input

[1522] Output: JSON format data

[1523] Specific operation:Conversion process to { "question": "I don't know how to solve the equation"}

[1524] Step 4:

[1525] The terminal sends the converted JSON data to the server as an HTTP POST request. The data is sent using the communication protocol.

[1526] Input: JSON format data

[1527] Output: An HTTP POST request is made to the server.

[1528] Step 5:

[1529] The server receives the HTTP request and retrieves the JSON data. The server then uses the Google Cloud NLP API to analyze the question, tokenizing the content and extracting keywords.

[1530] Input: JSON data from an HTTP POST request

[1531] Output: Analysis results (keywords and context)

[1532] Specific operation: Using natural language processing technology, important keywords such as "equation" and "solution method" are extracted from the question.

[1533] Step 6:

[1534] The server searches for appropriate explanatory articles and specific calculation methods based on the analysis results and generates an appropriate answer, which is then converted back into JSON format.

[1535] Input: Keywords of analysis results

[1536] Output: JSON formatted response data

[1537] Specific operation: Generate data including the URL of the related explanatory article and the calculation method { "url": "https: / / example.com / article", "method": "x = 7 - 3"}

[1538] Step 7:

[1539] The server sends the generated JSON data to the terminal as an HTTP response.

[1540] Input: JSON formatted response data

[1541] Output: Sent as an HTTP response

[1542] Step 8:

[1543] The device receives the response, parses the JSON data, and displays it visually to the user, such as links to explanatory articles and calculation instructions.

[1544] Input: JSON data of the HTTP response from the server

[1545] Output: A visual display to the user

[1546] Specific operation: Display an explanatory article link and calculation method { "url": "https: / / example.com / article", "method": "x = 7 - 3"}

[1547] Step 9:

[1548] Users can tap the link to the displayed explanatory article to view a detailed explanation, and by accessing specific learning content, they can solve the problem.

[1549] Input: Tap the displayed link

[1550] Output: View detailed explanation

[1551] Communication partner

[1552] Processing steps and specific explanations

[1553] Step 1:

[1554] The user starts the smartphone app and selects the "Communication Partner" menu, which displays a chat interface with the communication partner.

[1555] Input: Launching the smartphone app

[1556] Output: Display of communication partner interface

[1557] Step 2:

[1558] The user enters a message such as "Tell me what you'd like to do today" and taps the "Send" button. The input content is then managed within the app.

[1559] Input: User text input

[1560] Output: Keep input

[1561] Step 3:

[1562] The device receives the user's message content and converts it to JSON format, which structures the data and makes it ready for transmission.

[1563] Input: User text input

[1564] Output: JSON format data

[1565] Specific operation: Conversion process to { "message": "Tell me what you'd like to do today"}

[1566] Step 4:

[1567] The terminal sends the converted JSON data to the server as an HTTP POST request. The data is sent using the communication protocol.

[1568] Input: JSON format data

[1569] Output: An HTTP POST request is made to the server.

[1570] Step 5:

[1571] The server receives the HTTP request and retrieves the JSON data. The server then analyzes the message content using OpenAI's GPT-3 model. This analysis allows the server to understand the user's intent and derive an appropriate response.

[1572] Input: JSON data from an HTTP POST request

[1573] Output: Analysis results (user intent and appropriate response)

[1574] Specific operation: Using natural language processing technology, the system analyzes the message content and suggests appropriate activities in response to the question, "What would you like to do today?"

[1575] Step 6:

[1576] The server generates a response based on the parsing results, including the suggestion, "How about playing Animal Crossing?" The response is then converted back to JSON format.

[1577] Input: Intention of analysis results and proposals

[1578] Output: Reply data in JSON format

[1579] Specific behavior: Generate reply data such as "How about playing Animal Crossing?" { "reply": "How about playing Animal Crossing?"}

[1580] Step 7:

[1581] The server sends the generated JSON data to the terminal as an HTTP response.

[1582] Input: Reply data in JSON format

[1583] Output: Sent as an HTTP response

[1584] Step 8:

[1585] The device receives the response, parses the JSON data, and displays or plays the suggestion to the user visually or audibly.

[1586] Input: JSON data of the HTTP response from the server

[1587] Output: Visual or audio display / playback to the user

[1588] Specific behavior: Display or play a suggestion saying "How about playing Animal Crossing?"

[1589] Step 9:

[1590] The user sees the suggestions and decides to play Animal Crossing. By actually doing the suggested activities, they spend their time meaningfully while at home.

[1591] Input: Confirm and select the proposed content

[1592] Output: Activity execution

[1593] In this way, the system of the present invention allows users to efficiently complete their homework alone at home and also provides support for learning and alleviates feelings of loneliness by interacting with appropriate communication partners.

[1594] (Application example 1)

[1595] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1596] When children in dual-income households are left at home alone, they need help with their homework and appropriate communication partners to help them spend their alone time meaningfully. However, current systems lack the means to quickly and accurately obtain information when customers ask questions about products, especially when shopping in physical stores. This leaves many customers, including children, with difficulty in selecting and making decisions about products.

[1597] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1598] In this invention, the server includes a means for inputting homework questions and product information, a means for analyzing the input questions and product information and generating appropriate answers and information, and a means for providing the generated answers and information to a user. This not only enables children in dual-income households to receive help with their homework, but also enables customers in physical stores to quickly and accurately obtain appropriate information when asking questions about products.

[1599] The "means for inputting homework questions" refers to an interface that allows a user to input homework questions in text format using a device such as a smartphone or tablet.

[1600] The "means for analyzing the content of the entered question and generating an appropriate answer" refers to an algorithm and process that uses natural language processing technology to analyze the content of the question entered by the user and generate an answer that is highly relevant to the question.

[1601] The "means for providing the generated answer to the user" is an interface that displays or plays the analyzed and generated answer visually or audibly on the user's device.

[1602] The "means for inputting product information" is an interface through which customers in a physical store can input questions about product details and characteristics.

[1603] "Means for analyzing input product information and generating appropriate information" refers to algorithms and processes that use natural language processing technology to analyze questions and requests about products input by customers and generate relevant product information.

[1604] The "means for providing the generated product information to the user" is an interface that displays or plays back the analyzed and generated product information on the customer's device.

[1605] The "means for inputting chat messages while the user is at home alone" refers to an interface that allows the user to input messages in chat format using a smartphone or tablet while at home alone.

[1606] The "means for analyzing input messages and generating appropriate replies" refers to algorithms and processes that use natural language processing technology to analyze chat messages input by users and generate highly relevant replies to those messages.

[1607] The "means for providing the generated reply to the user" is an interface that displays or plays the parsed and generated reply on the user's device.

[1608] "Means for converting questions and product information into JSON format and sending it to a server" refers to the process and means for converting questions and product information entered by a user into a standardized JSON format and sending it to a server via a network as an HTTP request.

[1609] "Means for analyzing JSON format responses, product information, and replies from the server and displaying or playing them back to the user" refers to an interface that analyzes JSON format data received from the server and displays or plays it back on the device in a form that is easily understandable to the user.

[1610] "Means for analyzing question content and product information through natural language processing using a generative AI model" refers to algorithms and processes that use a generative AI model (e.g., GPT-3) to analyze question content and product information using advanced natural language processing technology.

[1611] This invention is a system that provides homework help for children in dual-income households who are left home alone, and also provides fast and accurate information when customers in physical stores have questions about products. The system consists of an application installed on a smartphone or tablet and a server that processes requests from these applications.

[1612] Hardware and software used

[1613] Hardware:

[1614] A smartphone or tablet as the user device

[1615] Data center or cloud computing infrastructure as a server

[1616] software:

[1617] Client side: Mobile application developed using React Native

[1618] Server side: Server application developed using Python and Flask

[1619] Natural language processing engine: OpenAI's GPT-3 model

[1620] Data processing and calculation flow

[1621] When a user types a question about homework or product information, the application installed on the smartphone or tablet receives the question. The application then converts the input question into JSON format and sends it as an HTTP POST request to a server. The server receives the request and uses a generative AI model (e.g., GPT-3) to analyze the question.

[1622] The server searches a database or knowledge base for appropriate answers and product information for the question, and returns the generated answers and information in JSON format to the application. The application then analyzes the received JSON data and displays or plays it back to the user visually or audibly. This process allows the user to obtain information quickly and accurately.

[1623] Specific examples

[1624] Example questions for user homework:

[1625] When a user submits a question by entering text such as "I don't know how to solve an equation," the server analyzes the question, searches for appropriate explanatory articles and calculation methods, and provides them to the user.

[1626] Example prompt sentence:

[1627] User asked: Are these sneakers waterproof?

[1628] AI Response:

[1629] Based on this, the server provides information such as, "These sneakers are made of Gore-Tex material and are waterproof."

[1630] Example prompts for the generative AI model used

[1631] User asked: Are these sneakers waterproof?

[1632] AI Response:

[1633] Specific server processing

[1634] The server utilizes natural language processing technology and generative AI models to generate appropriate answers to user questions and product information. Specifically, it analyzes the user's text input and understands their intent. It then retrieves the information most relevant to that intent from the database and generative AI model and provides it to the user in an appropriate format.

[1635] This system not only allows users to receive help with their homework, but also provides quick and accurate information for selecting products in physical stores, allowing children in dual-income households and customers in physical stores to spend their time more meaningfully.

[1636] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1637] Step 1:

[1638] The user launches the application on their smartphone or tablet and enters a question about homework or a product. The entered question is recorded in text format on the device.

[1639] input:

[1640] Questions typed in by users (e.g., "I don't know how to solve an equation" or "Are these sneakers waterproof?")

[1641] output:

[1642] Text questions recorded on the device

[1643] Specific behavior:

[1644] Using the application's interface, users enter their homework or product question and tap the "Submit" button.

[1645] Step 2:

[1646] The terminal converts the questions entered by the user into JSON format.

[1647] input:

[1648] Text-based questions

[1649] output:

[1650] Question content converted to JSON format

[1651] Specific behavior:

[1652] A script in the application converts the text question into JSON format (e.g., {"question": "I don't know how to solve the equation"}).

[1653] Step 3:

[1654] The device sends the converted JSON-formatted question to the server as an HTTP POST request.

[1655] input:

[1656] JSON formatted question

[1657] output:

[1658] HTTP POST request sent to the server

[1659] Specific behavior:

[1660] The device issues an HTTP request to the server and sends the question (e.g., a POST request with a JSON payload).

[1661] Step 4:

[1662] The server receives the HTTP POST request and parses the JSON-formatted question.

[1663] input:

[1664] Received question in JSON format

[1665] output:

[1666] Analyzed questions

[1667] Specific behavior:

[1668] The server parses the request and gets the question (e.g., parses the JSON payload using a JSON parsing library).

[1669] Step 5:

[1670] The server inputs the analyzed question into a generative AI model (e.g., GPT-3) to generate an appropriate answer.

[1671] input:

[1672] Analyzed questions

[1673] output:

[1674] Answers generated by generative AI models

[1675] Specific behavior:

[1676] The server issues an API request to the generated AI model and generates an answer based on the question (e.g., an API call to GPT-3).

[1677] Step 6:

[1678] The server generates a JSON response containing the generated answer and sends it to the device as an HTTP response.

[1679] input:

[1680] Generated Answer

[1681] output:

[1682] JSON-formatted response

[1683] Specific behavior:

[1684] The server converts the answer into JSON format and sends it to the terminal as an HTTP response (e.g., {"answer": "To solve the equation..."}).

[1685] Step 7:

[1686] The device receives and parses the JSON response from the server.

[1687] input:

[1688] JSON format response received from the server

[1689] output:

[1690] Analyzed answers

[1691] Specific behavior:

[1692] The device parses the received JSON data and formats it into the format required for display to the user.

[1693] Step 8:

[1694] The terminal displays the analyzed answer content to the user visually or audibly.

[1695] input:

[1696] Analyzed answers

[1697] output:

[1698] Answers presented to the user

[1699] Specific behavior:

[1700] The device displays the answer in a user interface so that the user can review it (e.g., displaying the answer in a text view).

[1701] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1702] The present invention combines a smartphone application system that provides homework help and communication partners for children in dual-income households who are left home alone with an emotion engine that recognizes the user's emotions. Specific embodiments of the present invention are described below.

[1703] Homework Help

[1704] overview

[1705] Users (children) can input questions about their homework through a smartphone app and receive appropriate answers. The system analyzes the questions entered by the user and generates and provides appropriate answers. It also utilizes an emotion engine that recognizes the user's emotions and responds based on the user's emotional state.

[1706] Specific operation example

[1707] 1. The user launches the smartphone app and selects the "Homework Help" menu.

[1708] 2. The user types the text "I don't know how to solve the equation" and taps the "Submit" button.

[1709] 3. The device receives the user's question and converts it into JSON format.

[1710] 4. The emotion engine recognizes emotions from the user's facial expressions and voice and generates emotion data.

[1711] 5. The device sends the question and emotion data to the server as an HTTP POST request.

[1712] 6. The server receives the request and analyzes the question and sentiment data, for example using natural language processing to understand the question and tailor the answer taking into account the sentiment data.

[1713] 7. The server generates JSON data including the URL of the appropriate explanatory article and specific calculation methods.

[1714] 8. The server sends the generated JSON data to the terminal as an HTTP response.

[1715] 9. The device receives the response and displays it visually to the user.

[1716] 10. The user taps on the link to the explanatory article displayed to view a detailed explanation.

[1717] Communication partner

[1718] overview

[1719] While the user is home alone, they can chat with a virtual communication partner through a smartphone app. The system analyzes the user's input messages, generates and provides appropriate replies, and uses an emotion engine to respond based on the user's emotional state.

[1720] Specific operation example

[1721] 1. The user launches the smartphone app and selects the "Communication Partner" menu.

[1722] 2. The user types the text "Tell me what you'd like to do today" and taps the "Send" button.

[1723] 3. The device converts the message into JSON format and sends it to the server as an HTTP POST request.

[1724] 4. The emotion engine recognizes emotions from the user's facial expressions and voice and generates emotion data.

[1725] 5. The device sends the message content and emotion data to the server.

[1726] 6. The server receives the request and analyzes the message content and emotional data, for example using natural language processing techniques to generate an appropriate reply and adjust the reply content taking the emotional data into account.

[1727] 7. The server generates a JSON response containing the suggestion, "How about playing Animal Crossing?"

[1728] 8. The server sends the generated JSON data to the terminal as an HTTP response.

[1729] 9. The device receives the response and presents it to the user visually or audibly.

[1730] 10. The user sees the suggestions and chooses to play Animal Crossing.

[1731] This system allows children who are alone at home to receive support with their schoolwork and avoid feeling lonely through conversations with virtual communication partners. Furthermore, the introduction of an emotion engine enables flexible responses based on the user's emotional state, providing more personalized support.

[1732] The processing flow will be explained below.

[1733] Homework Help

[1734] Program processing

[1735] Step 1:

[1736] The user launches the smartphone app and selects the "Homework Help" menu.

[1737] Step 2:

[1738] A user texts in a homework question, for example, "I don't know how to solve the equation."

[1739] Step 3:

[1740] The user taps the "Send" button.

[1741] Step 4:

[1742] The terminal receives the user's question.

[1743] Step 5:

[1744] In order for the device to analyze the user's facial expressions and voice and recognize emotions, it activates an emotion engine and acquires emotion data.

[1745] Step 6:

[1746] The device converts the question content and emotion data into JSON format.

[1747] Step 7:

[1748] The device sends the converted JSON-formatted question and emotion data to the server as an HTTP POST request.

[1749] Step 8:

[1750] The server receives the HTTP POST request, analyzes the question and emotional data, and uses natural language processing technology to understand the intent of the question and generate an answer that takes the emotional data into account.

[1751] Step 9:

[1752] The server searches a database or calls external APIs to generate an answer that fits the question, for example adding detailed explanations or encouraging messages if the user is confused.

[1753] Step 10:

[1754] The server generates JSON data containing the URL of the appropriate explanatory article and the specific answer.

[1755] Step 11:

[1756] The server sends the generated JSON data to the terminal as an HTTP response.

[1757] Step 12:

[1758] The device receives the HTTP response and parses the JSON data.

[1759] Step 13:

[1760] The device displays the analyzed data in a format that the user can visually confirm, such as a link to an explanatory article or a message of support.

[1761] Step 14:

[1762] The user taps the link to the explanatory article displayed to check the detailed content.

[1763] Communication partner

[1764] Program processing

[1765] Step 1:

[1766] The user launches the smartphone app and selects the "Communication Partner" menu.

[1767] Step 2:

[1768] The user types the text "Tell me what you'd like to do today" into the chat window.

[1769] Step 3:

[1770] The user taps the "Send" button.

[1771] Step 4:

[1772] The terminal receives the user's message.

[1773] Step 5:

[1774] In order for the device to analyze the user's facial expressions and voice and recognize emotions, it activates an emotion engine and acquires emotion data.

[1775] Step 6:

[1776] The device converts the message content and emotion data into JSON format.

[1777] Step 7:

[1778] The device sends the converted JSON-formatted message content and emotion data to the server as an HTTP POST request.

[1779] Step 8:

[1780] The server receives the HTTP POST request, analyzes the message content and emotional data, and uses natural language processing technology to understand the intent of the message and generate a reply that takes the emotional data into account.

[1781] Step 9:

[1782] The server searches a relevant database to generate an appropriate reply, and generates suitable reply suggestions, such as activity suggestions if the user is bored.

[1783] Step 10:

[1784] The server generates a JSON response containing the suggestion, "How about playing Animal Crossing?"

[1785] Step 11:

[1786] The server sends the generated JSON data to the terminal as an HTTP response.

[1787] Step 12:

[1788] The device receives the HTTP response and parses the JSON data.

[1789] Step 13:

[1790] The device presents the analyzed data to the user visually or audibly, such as suggestion messages or specific steps for an activity.

[1791] Step 14:

[1792] The user sees the suggestions and chooses to play Animal Crossing.

[1793] Example 2

[1794] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1795] In situations where children in dual-income households are left home alone, there is a need to provide appropriate support for homework questions and communication methods to reduce feelings of loneliness. It is also important that this support and communication take into account the child's emotional state, and personalized responses are required.

[1796] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for inputting homework questions; means for analyzing the input questions and generating appropriate answers; means for providing the generated answers to the user; means having an engine for recognizing the user's emotions and generating emotion data; means for transmitting the questions and emotion data and receiving answers from the server; means for inputting chat messages while the user is at home alone; means for analyzing the input messages and generating appropriate replies; means for providing the generated replies to the user; means for transmitting the message contents and emotion data and receiving the replies from the server; means for converting the questions and messages into a data format and transmitting them; and means for analyzing the answers and replies in data format from the server and displaying or playing them back to the user. This allows children to receive support with their homework while at home alone and helps them feel less lonely through virtual communication.

[1797] The "means for inputting homework questions" is a system component that provides an interface for users to input homework questions as text.

[1798] The "means for analyzing the entered question and generating an appropriate answer" is a component of the system that uses natural language processing technology to analyze the homework question entered by the user and generate the optimal answer to that question.

[1799] A "means for providing a generated answer to a user" is a system component that provides an interface for presenting the system-generated answer to a user visually or audibly.

[1800] "Means having an engine for recognizing a user's emotions and generating emotional data" refers to a system component that includes an engine that uses a camera or microphone to analyze a user's facial expressions and tone of voice, identify their emotional state, and generate emotional data.

[1801] The "means for sending question content and emotion data to a server and receiving a response from the server" refers to a system component that sends a question input by a user and generated emotion data to a server via an HTTP POST request and receives the response data received from the server.

[1802] The "means for inputting chat messages while the user is away" is a system component that provides an interface for the user to input chat messages in text format.

[1803] The "means for analyzing input messages and generating appropriate replies" is a component of a system that uses natural language processing technology to analyze chat messages input by users and generate optimal replies to those messages.

[1804] A "means for providing a generated reply to a user" is a system component that provides an interface for presenting a system-generated reply to a user visually or audibly.

[1805] "Means for transmitting message content and emotion data to a server and receiving a reply from the server" refers to a system component for transmitting message content entered by a user and generated emotion data to a server and receiving reply data received from the server.

[1806] "Means for converting questions and messages into a data format and sending them to the server" refers to a system component that converts questions and messages entered by users into a data format such as JSON and sends them to the server.

[1807] "Means for analyzing answers and replies in data format from the server and displaying or playing them back to the user" refers to a system component that analyzes answers and replies provided in data format such as JSON received from the server and displays or plays them back to the user visually or audibly.

[1808] The present invention combines a smartphone application system that provides homework help and communication partners for children in dual-income households who are left home alone with an emotion engine that recognizes the user's emotions. Specific embodiments of the present invention are described below.

[1809] Homework help feature

[1810] Basic configuration

[1811] This system uses a smartphone app to input questions related to homework and provides a series of functions to obtain appropriate answers. The main elements are a means for inputting the user's question, a means for analyzing the question and generating an appropriate answer, a means for providing the generated answer to the user, and a means for generating emotion data using an emotion engine. This system uses HTTPS communication to send the user's question and emotion data to the server and receive the answer from the server. It is also configured to convert the question and emotion data into JSON format and send it, and to analyze the answer received from the server and display it to the user.

[1812] Operation overview

[1813] 1. The user launches the smartphone app and selects the "Homework Help" menu.

[1814] 2. The user types the text "I don't know how to solve the equation" and taps the "Submit" button.

[1815] 3. The device converts the input question into JSON format and uses an emotion engine to recognize emotions from the user's facial expressions and voice, generating emotion data. The emotion engine can use, for example, Microsoft Azure's emotion analysis API.

[1816] 4. The device sends the question and emotion data to the server as an HTTP POST request, using a communication library such as OkHttp.

[1817] 5. The server analyzes the received request and generates an answer appropriate to the question. For natural language processing, it uses an NLP model, such as Hugging Face.

[1818] 6. The server generates a response and sends it to the device in JSON format.

[1819] 7. The terminal analyzes the received response and displays it visually to the user.

[1820] Examples of concrete examples and prompts

[1821] Examples:

[1822] User types, "Help me with my math homework today!"

[1823] The device uses an emotion engine to recognize when the user is feeling stressed.

[1824] The server provides detailed explanatory articles in a gentle tone.

[1825] Example prompt sentence:

[1826] If your child types "Help me with my math homework today" into your app, generate an appropriate response that takes into account the stress your child may be under.

[1827] Communication partner function

[1828] Basic configuration

[1829] This system provides a function that enables users to chat with virtual communication partners through a smartphone app while they are home alone. The main elements are a means for inputting messages, a means for analyzing the message content and generating an appropriate reply, a means for providing the generated reply to the user, and a means for generating emotion data using an emotion engine. The system is configured to send the message content and emotion data to a server, analyze the replies received from the server, and display or play them back to the user.

[1830] Operation overview

[1831] 1. The user launches the smartphone app and selects the "Communication Partner" menu.

[1832] 2. The user types the text "Tell me what you'd like to do today" and taps the "Send" button.

[1833] 3. The device converts the message content into JSON format and sends it to the server as an HTTP POST request.

[1834] 4. The emotion engine generates emotion data from the user's facial expressions and voice. For emotion analysis, it uses the IBM Watson API, for example.

[1835] 5. The device sends the message content and emotion data to the server.

[1836] 6. The server analyzes the received request and generates an appropriate reply, for example using OpenAI GPT-3.

[1837] 7. The server sends the appropriate reply in JSON format to the device.

[1838] 8. The device analyzes the reply received and presents it to the user visually or audibly.

[1839] Examples of concrete examples and prompts

[1840] Examples:

[1841] A user types, "What would be fun to do today?"

[1842] The device uses its emotion engine to recognize that the user is feeling a little lonely.

[1843] "Animal Crossing is a fun game," the server suggested.

[1844] Example prompt sentence:

[1845] When a child types into your app, "What would be fun to do today?", consider that they might be feeling a little lonely and generate fun suggestions. Include popular games and activities in your suggestions.

[1846] This system allows children who are alone at home to receive support with their schoolwork and avoid feeling lonely by interacting with virtual communication partners. The introduction of an emotion engine also enables flexible responses based on the user's emotional state, providing more personalized support.

[1847] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1848] Homework help feature

[1849] Step 1:

[1850] The user launches the smartphone app and selects the "Homework Help" menu.

[1851] Specific operation: The user taps the app from the smartphone home screen to launch it, and selects the "Homework Help" menu on the initial screen.

[1852] Input: User taps.

[1853] Output: Homework help menu will be displayed.

[1854] Step 2:

[1855] The user types the text "I don't know how to solve the equation" and taps the "Submit" button.

[1856] Specific actions: The user uses the smartphone keyboard to enter a question into the text box and taps the "Send" button.

[1857] Input: The question entered by the user is "I don't know how to solve an equation."

[1858] Output: The question is sent to the app.

[1859] Step 3:

[1860] The device converts the entered question into JSON format.

[1861] What it does: The app on the device reads the user's text input and converts it to JSON format.

[1862] Input: The question entered by the user is "I don't know how to solve an equation."

[1863] Data processing: Converting to JSON format (e.g., {"question": "I don't know how to solve the equation"}).

[1864] Output: Data converted to JSON format.

[1865] Step 4:

[1866] The emotion engine recognizes emotions from the user's facial expressions and voice and generates emotion data.

[1867] Specific operation: Using the device's camera and microphone, the system analyzes the user's facial expressions and voice to generate emotion data. For example, it uses Microsoft Azure's emotion analysis API as an emotion engine.

[1868] Input: User facial and voice data obtained from the camera and microphone.

[1869] Data operations: Identify emotional states and generate emotion data (e.g., {"emotion": "frustrated"}).

[1870] Output: Emotion data.

[1871] Step 5:

[1872] The device sends the question and emotion data to the server as an HTTP POST request.

[1873] Specific operation: The device sends the generated question and emotion data together to the server. A communication library (e.g., OkHttp) is used for communication.

[1874] Input: Question and emotion data in JSON format (e.g., {"question": "I don't know how to solve the equation", "emotion": "frustrated"}).

[1875] Output: An HTTP POST request is sent to the server.

[1876] Step 6:

[1877] The server receives the request and analyzes the question and emotion data.

[1878] What happens: The server receives the request and uses natural language processing techniques (e.g., the Hugging Face NLP model) to analyze the question and sentiment data.

[1879] Input: Submitted question and sentiment data.

[1880] Data Computing: Uses natural language processing to understand questions and tailor answers based on sentiment data.

[1881] Output: The appropriate answer data (e.g., an explanation of how to solve the equation).

[1882] Step 7:

[1883] The server generates JSON-formatted data including the URL of the appropriate explanatory article and specific calculation methods.

[1884] Specific operation: The server generates an appropriate answer and creates JSON-formatted data including the URL of the explanatory article and the specific calculation method.

[1885] Input: Analysis result data.

[1886] Data processing: Generate answer data in JSON format (e.g., {"answer": "To solve the equation...", "url": "http: / / example.com / solution"}).

[1887] Output: Response data in JSON format.

[1888] Step 8:

[1889] The server sends the generated JSON data to the terminal as an HTTP response.

[1890] Specific operation: The server generates data and sends it to the terminal as an HTTP response.

[1891] Input: Response data in JSON format.

[1892] Output: The data sent as the HTTP response.

[1893] Step 9:

[1894] The terminal receives the response and visually displays it to the user.

[1895] Specific operation: Analyzes the data received by the device and displays it in an easy-to-read format for the user. Uses a UI library (e.g., Android's RecyclerView).

[1896] Input: The JSON data sent as an HTTP response.

[1897] Data processing: Parsing and visualizing JSON data.

[1898] Output: The explanation or calculation displayed to the user.

[1899] Step 10:

[1900] The user taps the link of the displayed explanatory article to view a detailed explanation.

[1901] What happens: The user taps on a link to open a browser.

[1902] Input: User taps.

[1903] Output: A detailed explanatory article displayed in your browser.

[1904] Communication partner function

[1905] Step 1:

[1906] The user launches the smartphone app and selects the "Communication Partner" menu.

[1907] Specific behavior: The user opens the app and selects the "People to communicate with" menu.

[1908] Input: User taps.

[1909] Output: The communication partner menu is displayed.

[1910] Step 2:

[1911] The user types the text "Tell me what you'd like to do today" and taps the "Send" button.

[1912] Specific behavior: The user types "What would you like to do for fun today?" into the text box and taps the "Send" button.

[1913] Input: User-entered message: "Tell me what you'd like to do today."

[1914] Output: The message content is sent to the app.

[1915] Step 3:

[1916] The device converts the message content into JSON format and sends it to the server as an HTTP POST request.

[1917] Specific operation: The device converts the input message into JSON format and sends it to the server.

[1918] Input: User-entered message: "Tell me what you'd like to do today."

[1919] Data processing: Converting to JSON format (e.g., {"message": "Tell me what you'd like to do today"}).

[1920] Output: The data is converted to JSON format and sent to the server.

[1921] Step 4:

[1922] The emotion engine generates emotion data from the user's facial expressions and voice.

[1923] Specific operation: The emotion engine recognizes emotions through the camera and microphone and generates data on them.

[1924] Input: User facial and voice data obtained from the camera and microphone.

[1925] Data operations: Identify emotional states and generate emotion data (e.g., {"emotion": "lonely"}).

[1926] Output: Emotion data.

[1927] Step 5:

[1928] The device sends the message content and emotion data to the server.

[1929] Specific operation: The device sends the acquired message content and emotion data to the server.

[1930] Input: Message content and emotion data in JSON format (e.g., {"message": "Tell me what you'd like to do today", "emotion": "lonely"}).

[1931] Output: An HTTP POST request is sent to the server.

[1932] Step 6:

[1933] The server receives the request and analyzes the message content and emotion data.

[1934] Specific operation: The server analyzes the received data and generates appropriate suggestions. For natural language processing, OpenAI GPT-3 is used as an example.

[1935] Input: Message content and sentiment data.

[1936] Data Computation: Analysis and response generation using natural language processing.

[1937] Output: Appropriate reply data (e.g., fun activity suggestions).

[1938] Step 7:

[1939] The server generates a JSON representation of the appropriate response.

[1940] Specific operation: The server converts the proposal based on the analysis results into JSON format data.

[1941] Input: Analysis results.

[1942] Data processing: Generate reply data in JSON format (e.g., {"reply": "How about playing Animal Crossing?"}).

[1943] Output: Reply data in JSON format.

[1944] Step 8:

[1945] The server sends the generated JSON data to the terminal as an HTTP response.

[1946] Specific operation: The data generated by the server is sent to the terminal as an HTTP response.

[1947] Input: Reply data in JSON format.

[1948] Output: The data sent as the HTTP response.

[1949] Step 9:

[1950] The terminal receives the response and presents it to the user visually or audibly.

[1951] Specific operation: The device analyzes the reply data received and displays it on the screen or reads it aloud. It uses a UI library or text-to-speech library.

[1952] Input: The JSON data sent as an HTTP response.

[1953] Data processing: Parse JSON data and display or output it as audio.

[1954] Output: Screen display and audio notification.

[1955] Step 10:

[1956] The user sees the suggestions and chooses to play Animal Crossing.

[1957] Specific behavior: The user accepts the suggestion and selects an activity.

[1958] Input: User's choice.

[1959] Output: The execution of the selected activity.

[1960] Through the above processing steps, children can receive help with their homework while at home alone, and can avoid feeling lonely through virtual communication.

[1961] (Application example 2)

[1962] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1963] In recent years, the number of dual-income households has increased, resulting in more opportunities for children to be left home alone. However, in these situations, children may have questions or feel anxious about their homework. In particular, there is a need for communication methods to receive appropriate answers to questions about homework and to reduce feelings of loneliness. There is also a need for systems that can be used safely for small payments. It is hoped that these issues can be resolved and an environment in which children can spend their time with peace of mind will be provided.

[1964] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting homework questions, means for analyzing the input questions and generating appropriate answers, means for providing the generated answers to the user, means for recognizing the user's emotions and generating emotion data, and means for adjusting a response based on the generated emotion data. This makes it possible to provide a sense of security to children who are home alone through support with homework and communication, and to enable safe small-amount payments.

[1965] "Means for inputting homework questions" refers to a function that allows users to input questions about homework via devices such as smartphones or personal computers.

[1966] "Means for analyzing the content of the question entered and generating an appropriate answer" refers to a function that analyzes the content of the question entered by the user using natural language processing technology, etc., and generates an appropriate answer.

[1967] The "means for providing the generated answer to the user" is a function for presenting the answer generated through analysis to the user visually or audibly.

[1968] The "means for recognizing the user's emotions and generating emotion data" is a function for recognizing emotions from the user's facial expressions, voice, etc., and generating emotion data based on the emotions.

[1969] The "means for adjusting a response based on the generated emotion data" is a function for taking the optimal response to the user's emotional state based on the generated emotion data.

[1970] "Means for inputting chat messages while the user is at home alone" is a function for inputting messages to chat with a virtual communication partner while the user is at home alone.

[1971] The "means for analyzing input messages and generating appropriate replies" refers to a function that analyzes chat messages input by users using natural language processing technology, etc., and generates appropriate replies.

[1972] The "means for providing the generated reply to the user" is a function for presenting the generated reply to the user and continuing the dialogue with the virtual communication partner.

[1973] The "means for generating emotion data and transmitting the emotion data to a server" is a function for transmitting emotion data generated by an emotion engine to a server.

[1974] "Means for converting questions and messages into JSON format and sending them to the server" is a function for converting questions and messages entered by the user into JSON format and sending them to the server.

[1975] "Means for analyzing JSON format answers and replies from the server and displaying or playing them back to the user" refers to a function that analyzes JSON format answers and replies received from the server and provides them to the user visually or audibly.

[1976] This invention combines a smartphone application system that provides homework help and communication partners for children in dual-income households while they are home alone with an emotion engine that recognizes the user's emotions. Specific embodiments will be described in detail below.

[1977] 1. Homework Help

[1978] Program processing

[1979] The terminal accepts questions about homework entered by the child. The questions are entered in text format through the "means for entering homework question content." The terminal analyzes the questions using a natural language processing engine, generates an appropriate answer, and provides it to the user through the "means for analyzing entered question content and generating an appropriate answer." In doing so, the terminal uses an emotion engine to analyze the user's facial expressions and voice and generate emotion data. The emotion data generated by the "means for recognizing user emotions and generating emotion data" is sent to the server, and the answer is adjusted based on the analysis results. The resulting answer is displayed to the user through the "means for providing the generated answer to the user."

[1980] Hardware and software used

[1981] Devices: Smartphones, tablets

[1982] Natural language processing engines: TensorFlow, e.g., GPT-based models

[1983] Emotion Engine: EmotionRecognition Library

[1984] Specific examples

[1985] The user types, "I don't know how to solve the equation."

[1986] The emotion engine analyzes the user's facial expressions and detects stress.

[1987] The server generates an answer presented in a gentler tone based on the question content and stress emotional data.

[1988] 2. Communication Partner

[1989] Program processing

[1990] The terminal accepts chat messages entered by the user, and the messages entered through the "means for entering chat messages while the user is away" are analyzed using a natural language processing engine. The server generates an appropriate reply and provides it to the user through the "means for analyzing the entered message and generating an appropriate reply." At this time, an emotion engine recognizes emotions from the user's facial expressions and voice and generates emotion data. The reply content is adjusted based on the generated emotion data. The reply is presented to the user through the "means for providing the generated reply to the user."

[1991] Hardware and software used

[1992] Devices: Smartphones, tablets

[1993] Natural language processing engines: TensorFlow, e.g., GPT-based models

[1994] Emotion Engine: EmotionRecognition Library

[1995] Specific examples

[1996] The user types, "Tell me what you'd like to do today."

[1997] The emotion engine detects boredom from the user's tone of voice.

[1998] The server uses the suggestion to generate a reply: "How about playing Animal Crossing?"

[1999] 3. Examples of Electronic Payment Services

[2000] Program processing

[2001] The terminal accepts a payment request entered by the user. The payment details are converted to JSON format and sent to the server using the "means for converting questions and messages into JSON format and sending them to the server." The server then analyzes the user's emotions using an emotion engine and takes appropriate action. For example, if the user is feeling anxious, a notification is sent to the parent. Finally, the payment result is presented to the user using the "means for analyzing JSON-formatted answers and replies from the server and displaying or playing them back to the user."

[2002] Hardware and software used

[2003] Devices: Smartphones, tablets

[2004] Emotion Engine: EmotionRecognition Library

[2005] Notification System: Email Service API

[2006] Specific examples

[2007] The user types, "Please tell me which math textbook you recommend that costs less than 500 yen."

[2008] The emotion engine recognizes the user's tension.

[2009] The server uses emotional data from stressful situations to suggest recommended textbooks using reassuring language.

[2010] The above is an embodiment of the present invention. This system allows children in dual-income households to safely make small payments when they are left at home alone. Furthermore, by utilizing an emotion engine, personalized support is provided based on the emotional state of the user (child).

[2011] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2012] Step 1:

[2013] The user starts up the smartphone and asks for help with their homework or enters a chat message. Specifically, the user types, "I don't know how to solve this equation," and this question is sent to the device.

[2014] Step 2:

[2015] The terminal converts the input question content into JSON format for data analysis, structuring the user's input data and facilitating processing on the server side.

[2016] Step 3:

[2017] The emotion engine generates emotion data in real time from the user's facial expressions and voice. For example, it receives the user's facial expression images and voice data as input, performs emotion analysis based on them, and outputs emotion labels such as "stress" or "boredom."

[2018] Step 4:

[2019] The device sends the question content converted into JSON format and the generated emotion data to the server as an HTTP POST request. Specifically, the data format is a JSON object containing the question content and emotion data, which is sent in the body of the HTTP request.

[2020] Step 5:

[2021] The server analyzes the received request, extracts the question content and emotion data, and uses this data to analyze the question content using a natural language processing engine and generate an appropriate answer.

[2022] Step 6:

[2023] The server adjusts the generated response taking into account the emotional data. For example, if the user is feeling "stressed," the server adjusts the response to be gentler and more reassuring.

[2024] Step 7:

[2025] The server constructs the tailored answer in JSON format and sends it to the device as an HTTP response, which includes the answer to the question and a sentiment-sensitive message.

[2026] Step 8:

[2027] The terminal analyzes the received response and presents the answer data to the user visually or audibly, and the user views the presented answer and solves the homework problem.

[2028] Step 9:

[2029] If the user enters another question or chat message, the process repeats, for example, "Tell me what you'd like to do today."

[2030] Step 10:

[2031] The server generates an appropriate reply from the chatbot based on the new message entered and adjusts it again taking into account the emotional data. The device then displays the adjusted reply to the user.

[2032] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[2033] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2034] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[2035] [Fourth embodiment]

[2036] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[2037] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[2038] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[2039] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[2040] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[2041] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[2042] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[2043] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[2044] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[2045] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[2046] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[2047] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[2048] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2049] The present invention provides a system that uses a smartphone application to provide homework help and a communication partner for children in dual-income households who are left home alone. Specific embodiments of the system are described below.

[2050] Homework Help

[2051] overview

[2052] Users (children) can input questions about their homework through a smartphone app and receive appropriate answers. The system analyzes the questions entered by the user and generates and provides appropriate answers.

[2053] Specific operation example

[2054] 1. The user launches the smartphone app and selects the "Homework Help" menu.

[2055] 2. The user types the text "I don't know how to solve the equation" and taps the "Submit" button.

[2056] 3. The device receives the user's question and converts it into JSON format.

[2057] 4. The device sends the converted question to the server as an HTTP POST request.

[2058] 5. The server receives the request and analyzes the question, for example, using natural language processing technology to understand the question and search for related articles or calculation methods.

[2059] 6. Based on the analysis results, the server generates JSON format data including the URL of an appropriate explanatory article and specific calculation methods.

[2060] 7. The server sends the generated JSON data to the terminal as an HTTP response.

[2061] 8. The device receives the response and displays it visually to the user.

[2062] 9. The user taps on the link to the explanatory article displayed to view a detailed explanation.

[2063] Communication partner

[2064] overview

[2065] While a user is at home alone, they can chat with a virtual communication partner through a smartphone app. The system analyzes the user's input messages and generates and provides appropriate replies.

[2066] Specific operation example

[2067] 1. The user launches the smartphone app and selects the "Communication Partner" menu.

[2068] 2. The user types the text "Tell me what you'd like to do today" and taps the "Send" button.

[2069] 3. The device converts the message into JSON format and sends it to the server as an HTTP POST request.

[2070] 4. The server receives the request and analyzes the message content, for example using natural language processing techniques to understand the intent of the message and suggest appropriate activities.

[2071] 5. Based on the analysis results, the server generates JSON format data containing the suggestion, "How about playing Animal Crossing?"

[2072] 6. The server sends the generated JSON data to the terminal as an HTTP response.

[2073] 7. The device receives the response and displays or plays it back to the user visually or audibly.

[2074] 8. The user sees the suggestions and decides to play Animal Crossing.

[2075] This system allows children who are alone at home to receive support with their schoolwork and avoid feeling lonely by interacting with virtual communication partners.

[2076] The processing flow will be explained below.

[2077] Homework Help

[2078] Program processing

[2079] Step 1:

[2080] The user launches the smartphone app and selects the "Homework Help" menu.

[2081] Step 2:

[2082] A user texts in a homework question, for example, "I don't know how to solve the equation."

[2083] Step 3:

[2084] The user taps the "Send" button.

[2085] Step 4:

[2086] The terminal receives the user's question.

[2087] Step 5:

[2088] The question received by the device is converted into JSON format.

[2089] Step 6:

[2090] The device sends the converted JSON-formatted question to the server as an HTTP POST request.

[2091] Step 7:

[2092] The server receives the HTTP POST request and analyzes the question, specifically using natural language processing technology to understand the intent of the question.

[2093] Step 8:

[2094] The server searches a database or calls an external API to generate an answer that is appropriate for the question.

[2095] Step 9:

[2096] The server generates JSON data containing the URL of the appropriate explanatory article and the specific answer.

[2097] Step 10:

[2098] The server sends the generated JSON data to the terminal as an HTTP response.

[2099] Step 11:

[2100] The device receives the HTTP response and parses the JSON data.

[2101] Step 12:

[2102] The terminal displays the analyzed data in a format that the user can visually confirm.

[2103] Step 13:

[2104] For example, the user taps the link of the displayed explanatory article to check the detailed content.

[2105] Communication partner

[2106] Program processing

[2107] Step 1:

[2108] The user launches the smartphone app and selects the "Communication Partner" menu.

[2109] Step 2:

[2110] The user types the text "Tell me what you'd like to do today" into the chat window.

[2111] Step 3:

[2112] The user taps the "Send" button.

[2113] Step 4:

[2114] The terminal receives the user's message.

[2115] Step 5:

[2116] The terminal converts the received message into JSON format.

[2117] Step 6:

[2118] The device sends the converted JSON formatted message to the server as an HTTP POST request.

[2119] Step 7:

[2120] The server receives the HTTP POST request and analyzes the message content. Specifically, it uses natural language processing technology to understand the intent of the message.

[2121] Step 8:

[2122] The server searches a relevant database to generate an appropriate reply, or generates suitable reply candidates.

[2123] Step 9:

[2124] The server generates a JSON response containing the suggestion, "How about playing Animal Crossing?"

[2125] Step 10:

[2126] The server sends the generated JSON data to the terminal as an HTTP response.

[2127] Step 11:

[2128] The device receives the HTTP response and parses the JSON data.

[2129] Step 12:

[2130] The terminal presents the analyzed data to the user visually or audibly.

[2131] Step 13:

[2132] The user sees the suggestions and chooses to play Animal Crossing.

[2133] Example 1

[2134] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2135] In situations where children from dual-income households are left home alone, it is difficult to provide appropriate communication methods to support their schoolwork and alleviate loneliness. Furthermore, in order for children to efficiently complete their homework at home, they need immediate, easy-to-understand answers, but there is a lack of technology to adequately resolve this issue. Furthermore, there is a need for a system that facilitates smooth dialogue with virtual communication partners to alleviate children's feelings of loneliness.

[2136] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2137] In this invention, the server includes a means for inputting a question, a means for analyzing the input question, a means for generating an appropriate answer based on the analysis results, and a means for providing the generated answer to the user. This allows a child to efficiently complete their homework at home alone. The server also includes a means for inputting a message while the user is at home alone, a means for analyzing the input message, a means for generating an appropriate reply based on the analysis results, and a means for providing the generated reply to the user. This allows a child to feel less alone by interacting with a virtual communication partner. The server also includes a means for converting the input question and message into a data format and transmitting it to the server, and a means for analyzing the data-formatted answers and replies from the server and displaying or playing them back to the user. This allows for efficient and accurate responses to the user's questions and messages and for immediate information provision.

[2138] The "means for inputting questions" is an interface that allows the user to provide questions about homework to the system in the form of input such as text.

[2139] "Means for analyzing the content of the entered question" refers to technology that uses natural language processing technology to understand the question entered by the user and extract appropriate keywords and context.

[2140] "Means for generating appropriate answers" are algorithms and databases that provide relevant information and explanations based on the analyzed question content.

[2141] "Means for providing the generated answer to the user" refers to an interface for visually or audibly displaying and playing the answer generated by the system on the user's terminal.

[2142] The "means for inputting a message while the user is away" is an interface through which the user inputs a message to converse with the communication partner.

[2143] The "means for analyzing an input message" is a technology that analyzes the contents of the message using natural language processing technology and understands the intent and important elements of the message.

[2144] The "means for generating an appropriate reply" refers to an algorithm or database for generating an appropriate reply to the user based on the analysis results.

[2145] "Means for providing the generated reply to the user" means an interface for visually or audibly displaying and playing the reply generated by the system on the user's terminal.

[2146] "Means for converting the input question and message into a data format and sending it to the server" refers to a technology that converts the user's input into a data format such as JSON and sends it to the server via a network.

[2147] "Means for analyzing answers and replies in data format from the server and displaying or playing them back to the user" refers to technology that analyzes answer and reply data returned from the server on the user's terminal and displays or plays them back visually or audibly.

[2148] This invention is a system that provides learning support and communication partners for children in dual-income households who are left home alone. In particular, it is a smartphone application that combines an algorithm for analyzing questions and providing appropriate answers with technology for generating appropriate replies to user messages.

[2149] Homework Help

[2150] overview

[2151] Users (children) can input questions about their homework through a smartphone app and receive appropriate answers. The system analyzes the questions entered by the user and generates and provides appropriate answers.

[2152] Specific Embodiments

[2153] 1. The user launches the smartphone app and selects the "Homework Help" menu.

[2154] 2. The user enters the question in text format and taps the "Submit" button. For example, the user might enter "I don't know how to solve the equation."

[2155] 3. The device receives the question and converts it into JSON format.

[2156] 4. The device sends the converted question to the server as an HTTP POST request.

[2157] 5. The server uses the Google Cloud NLP API to analyze the question, extracting keywords such as "equation" and "solution method."

[2158] 6. Based on the analysis results, the server generates a response in the form of data, including appropriate explanatory articles and specific calculation methods.

[2159] 7. The server sends the generated response in the data format to the terminal as an HTTP response.

[2160] 8. The device receives the response and displays it visually to the user, for example, a link to an explanatory article or a calculation method.

[2161] 9. The user taps the displayed link to view a detailed explanation.

[2162] Specific examples

[2163] Example prompt: "I don't know how to solve a linear equation like x + 3 = 7. Please help."

[2164] Communication partner

[2165] overview

[2166] While a user is at home alone, they can chat with a virtual communication partner through a smartphone app. The system analyzes the user's input messages and generates and provides appropriate replies.

[2167] Specific Embodiments

[2168] 1. The user launches the smartphone app and selects the "Communication Partner" menu.

[2169] 2. The user types a message and taps the "Send" button. For example, "Tell me what you'd like to do today."

[2170] 3. The device converts the message into JSON format and sends it to the server as an HTTP POST request.

[2171] 4. The server uses OpenAI's GPT-3 model to analyze the message content and suggest appropriate activities.

[2172] 5. Based on the analysis results, the server generates a response in the form of data containing the suggestion, "How about playing Animal Crossing?"

[2173] 6. The server sends the reply in the generated data format to the terminal as an HTTP response.

[2174] 7. The device receives the response and displays or plays it back to the user visually or audibly.

[2175] 8. The user sees the suggestions and decides to play Animal Crossing.

[2176] Specific examples

[2177] Example prompt: "I'm alone and bored today. Can you suggest some fun activities?"

[2178] Hardware and software used

[2179] This system is realized using a smartphone, a server, an internet connection, the Google Cloud NLP API, and OpenAI's GPT-3 model. Specifically, the smartphone receives input from the user, converts it into a data format, and sends it to the server via the internet connection. The server analyzes the received data and generates and returns an appropriate answer or reply. The smartphone displays or plays the returned data to the user.

[2180] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2181] Homework Help

[2182] Processing steps and specific explanations

[2183] Step 1:

[2184] A user launches a smartphone app and selects the "Homework Help" menu, which displays the in-app "Homework Help" interface.

[2185] Input: Launching the smartphone app

[2186] Output: Homework help interface displayed

[2187] Step 2:

[2188] The user enters the question in text format, such as "I don't know how to solve the equation," and taps the "Send" button. This allows the input content to be managed within the app.

[2189] Input: User text input

[2190] Output: Keep input

[2191] Step 3:

[2192] The device receives the user's question and converts it into JSON format, which structures the data and makes it ready for transmission.

[2193] Input: User text input

[2194] Output: JSON format data

[2195] Specific operation:Conversion process to { "question": "I don't know how to solve the equation"}

[2196] Step 4:

[2197] The terminal sends the converted JSON data to the server as an HTTP POST request. The data is sent using the communication protocol.

[2198] Input: JSON format data

[2199] Output: An HTTP POST request is made to the server.

[2200] Step 5:

[2201] The server receives the HTTP request and retrieves the JSON data. The server then uses the Google Cloud NLP API to analyze the question, tokenizing the content and extracting keywords.

[2202] Input: JSON data from an HTTP POST request

[2203] Output: Analysis results (keywords and context)

[2204] Specific operation: Using natural language processing technology, important keywords such as "equation" and "solution method" are extracted from the question.

[2205] Step 6:

[2206] The server searches for appropriate explanatory articles and specific calculation methods based on the analysis results and generates an appropriate answer, which is then converted back into JSON format.

[2207] Input: Keywords of analysis results

[2208] Output: JSON formatted response data

[2209] Specific operation: Generate data including the URL of the related explanatory article and the calculation method { "url": "https: / / example.com / article", "method": "x = 7 - 3"}

[2210] Step 7:

[2211] The server sends the generated JSON data to the terminal as an HTTP response.

[2212] Input: JSON formatted response data

[2213] Output: Sent as an HTTP response

[2214] Step 8:

[2215] The device receives the response, parses the JSON data, and displays it visually to the user, such as links to explanatory articles and calculation instructions.

[2216] Input: JSON data of the HTTP response from the server

[2217] Output: A visual display to the user

[2218] Specific operation: Display an explanatory article link and calculation method { "url": "https: / / example.com / article", "method": "x = 7 - 3"}

[2219] Step 9:

[2220] Users can tap the link to the displayed explanatory article to view a detailed explanation, and by accessing specific learning content, they can solve the problem.

[2221] Input: Tap the displayed link

[2222] Output: View detailed explanation

[2223] Communication partner

[2224] Processing steps and specific explanations

[2225] Step 1:

[2226] The user starts the smartphone app and selects the "Communication Partner" menu, which displays a chat interface with the communication partner.

[2227] Input: Launching the smartphone app

[2228] Output: Display of communication partner interface

[2229] Step 2:

[2230] The user enters a message such as "Tell me what you'd like to do today" and taps the "Send" button. The input content is then managed within the app.

[2231] Input: User text input

[2232] Output: Keep input

[2233] Step 3:

[2234] The device receives the user's message content and converts it to JSON format, which structures the data and makes it ready for transmission.

[2235] Input: User text input

[2236] Output: JSON format data

[2237] Specific operation: Conversion process to { "message": "Tell me what you'd like to do today"}

[2238] Step 4:

[2239] The terminal sends the converted JSON data to the server as an HTTP POST request. The data is sent using the communication protocol.

[2240] Input: JSON format data

[2241] Output: An HTTP POST request is made to the server.

[2242] Step 5:

[2243] The server receives the HTTP request and retrieves the JSON data. The server then analyzes the message content using OpenAI's GPT-3 model. This analysis allows the server to understand the user's intent and derive an appropriate response.

[2244] Input: JSON data from an HTTP POST request

[2245] Output: Analysis results (user intent and appropriate response)

[2246] Specific operation: Using natural language processing technology, the system analyzes the message content and suggests appropriate activities in response to the question, "What would you like to do today?"

[2247] Step 6:

[2248] The server generates a response based on the parsing results, including the suggestion, "How about playing Animal Crossing?" The response is then converted back to JSON format.

[2249] Input: Intention of analysis results and proposals

[2250] Output: Reply data in JSON format

[2251] Specific behavior: Generate reply data such as "How about playing Animal Crossing?" { "reply": "How about playing Animal Crossing?"}

[2252] Step 7:

[2253] The server sends the generated JSON data to the terminal as an HTTP response.

[2254] Input: Reply data in JSON format

[2255] Output: Sent as an HTTP response

[2256] Step 8:

[2257] The device receives the response, parses the JSON data, and displays or plays the suggestion to the user visually or audibly.

[2258] Input: JSON data of the HTTP response from the server

[2259] Output: Visual or audio display / playback to the user

[2260] Specific behavior: Display or play a suggestion saying "How about playing Animal Crossing?"

[2261] Step 9:

[2262] The user sees the suggestions and decides to play Animal Crossing. By actually doing the suggested activities, they spend their time meaningfully while at home.

[2263] Input: Confirm and select the proposed content

[2264] Output: Activity execution

[2265] In this way, the system of the present invention allows users to efficiently complete their homework alone at home and also provides support for learning and alleviates feelings of loneliness by interacting with appropriate communication partners.

[2266] (Application example 1)

[2267] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2268] When children in dual-income households are left at home alone, they need help with their homework and appropriate communication partners to help them spend their alone time meaningfully. However, current systems lack the means to quickly and accurately obtain information when customers ask questions about products, especially when shopping in physical stores. This leaves many customers, including children, with difficulty in selecting and making decisions about products.

[2269] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2270] In this invention, the server includes a means for inputting homework questions and product information, a means for analyzing the input questions and product information and generating appropriate answers and information, and a means for providing the generated answers and information to a user. This not only enables children in dual-income households to receive help with their homework, but also enables customers in physical stores to quickly and accurately obtain appropriate information when asking questions about products.

[2271] The "means for inputting homework questions" refers to an interface that allows a user to input homework questions in text format using a device such as a smartphone or tablet.

[2272] The "means for analyzing the content of the entered question and generating an appropriate answer" refers to an algorithm and process that uses natural language processing technology to analyze the content of the question entered by the user and generate an answer that is highly relevant to the question.

[2273] The "means for providing the generated answer to the user" is an interface that displays or plays the analyzed and generated answer visually or audibly on the user's device.

[2274] The "means for inputting product information" is an interface through which customers in a physical store can input questions about product details and characteristics.

[2275] "Means for analyzing input product information and generating appropriate information" refers to algorithms and processes that use natural language processing technology to analyze questions and requests about products input by customers and generate relevant product information.

[2276] The "means for providing the generated product information to the user" is an interface that displays or plays back the analyzed and generated product information on the customer's device.

[2277] The "means for inputting chat messages while the user is at home alone" refers to an interface that allows the user to input messages in chat format using a smartphone or tablet while at home alone.

[2278] The "means for analyzing input messages and generating appropriate replies" refers to algorithms and processes that use natural language processing technology to analyze chat messages input by users and generate highly relevant replies to those messages.

[2279] The "means for providing the generated reply to the user" is an interface that displays or plays the parsed and generated reply on the user's device.

[2280] "Means for converting questions and product information into JSON format and sending it to a server" refers to the process and means for converting questions and product information entered by a user into a standardized JSON format and sending it to a server via a network as an HTTP request.

[2281] "Means for analyzing JSON format responses, product information, and replies from the server and displaying or playing them back to the user" refers to an interface that analyzes JSON format data received from the server and displays or plays it back on the device in a form that is easily understandable to the user.

[2282] "Means for analyzing question content and product information through natural language processing using a generative AI model" refers to algorithms and processes that use a generative AI model (e.g., GPT-3) to analyze question content and product information using advanced natural language processing technology.

[2283] This invention is a system that provides homework help for children in dual-income households who are left home alone, and also provides fast and accurate information when customers in physical stores have questions about products. The system consists of an application installed on a smartphone or tablet and a server that processes requests from these applications.

[2284] Hardware and software used

[2285] Hardware:

[2286] A smartphone or tablet as the user device

[2287] Data center or cloud computing infrastructure as a server

[2288] software:

[2289] Client side: Mobile application developed using React Native

[2290] Server side: Server application developed using Python and Flask

[2291] Natural language processing engine: OpenAI's GPT-3 model

[2292] Data processing and calculation flow

[2293] When a user types a question about homework or product information, the application installed on the smartphone or tablet receives the question. The application then converts the input question into JSON format and sends it as an HTTP POST request to a server. The server receives the request and uses a generative AI model (e.g., GPT-3) to analyze the question.

[2294] The server searches a database or knowledge base for appropriate answers and product information for the question, and returns the generated answers and information in JSON format to the application. The application then analyzes the received JSON data and displays or plays it back to the user visually or audibly. This process allows the user to obtain information quickly and accurately.

[2295] Specific examples

[2296] Example questions for user homework:

[2297] When a user submits a question by entering text such as "I don't know how to solve an equation," the server analyzes the question, searches for appropriate explanatory articles and calculation methods, and provides them to the user.

[2298] Example prompt sentence:

[2299] User asked: Are these sneakers waterproof?

[2300] AI Response:

[2301] Based on this, the server provides information such as, "These sneakers are made of Gore-Tex material and are waterproof."

[2302] Example prompts for the generative AI model used

[2303] User asked: Are these sneakers waterproof?

[2304] AI Response:

[2305] Specific server processing

[2306] The server utilizes natural language processing technology and generative AI models to generate appropriate answers to user questions and product information. Specifically, it analyzes the user's text input and understands their intent. It then retrieves the information most relevant to that intent from the database and generative AI model and provides it to the user in an appropriate format.

[2307] This system not only allows users to receive help with their homework, but also provides quick and accurate information for selecting products in physical stores, allowing children in dual-income households and customers in physical stores to spend their time more meaningfully.

[2308] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2309] Step 1:

[2310] The user launches the application on their smartphone or tablet and enters a question about homework or a product. The entered question is recorded in text format on the device.

[2311] input:

[2312] Questions typed in by users (e.g., "I don't know how to solve an equation" or "Are these sneakers waterproof?")

[2313] output:

[2314] Text questions recorded on the device

[2315] Specific behavior:

[2316] Using the application's interface, users enter their homework or product question and tap the "Submit" button.

[2317] Step 2:

[2318] The terminal converts the questions entered by the user into JSON format.

[2319] input:

[2320] Text-based questions

[2321] output:

[2322] Question content converted to JSON format

[2323] Specific behavior:

[2324] A script in the application converts the text question into JSON format (e.g., {"question": "I don't know how to solve the equation"}).

[2325] Step 3:

[2326] The device sends the converted JSON-formatted question to the server as an HTTP POST request.

[2327] input:

[2328] JSON formatted question

[2329] output:

[2330] HTTP POST request sent to the server

[2331] Specific behavior:

[2332] The device issues an HTTP request to the server and sends the question (e.g., a POST request with a JSON payload).

[2333] Step 4:

[2334] The server receives the HTTP POST request and parses the JSON-formatted question.

[2335] input:

[2336] Received question in JSON format

[2337] output:

[2338] Analyzed questions

[2339] Specific behavior:

[2340] The server parses the request and gets the question (e.g., parses the JSON payload using a JSON parsing library).

[2341] Step 5:

[2342] The server inputs the analyzed question into a generative AI model (e.g., GPT-3) to generate an appropriate answer.

[2343] input:

[2344] Analyzed questions

[2345] output:

[2346] Answers generated by generative AI models

[2347] Specific behavior:

[2348] The server issues an API request to the generated AI model and generates an answer based on the question (e.g., an API call to GPT-3).

[2349] Step 6:

[2350] The server generates a JSON response containing the generated answer and sends it to the device as an HTTP response.

[2351] input:

[2352] Generated Answer

[2353] output:

[2354] JSON-formatted response

[2355] Specific behavior:

[2356] The server converts the answer into JSON format and sends it to the terminal as an HTTP response (e.g., {"answer": "To solve the equation..."}).

[2357] Step 7:

[2358] The device receives and parses the JSON response from the server.

[2359] input:

[2360] JSON format response received from the server

[2361] output:

[2362] Analyzed answers

[2363] Specific behavior:

[2364] The device parses the received JSON data and formats it into the format required for display to the user.

[2365] Step 8:

[2366] The terminal displays the analyzed answer content to the user visually or audibly.

[2367] input:

[2368] Analyzed answers

[2369] output:

[2370] Answers presented to the user

[2371] Specific behavior:

[2372] The device displays the answer in a user interface so that the user can review it (e.g., displaying the answer in a text view).

[2373] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2374] The present invention combines a smartphone application system that provides homework help and communication partners for children in dual-income households who are left home alone with an emotion engine that recognizes the user's emotions. Specific embodiments of the present invention are described below.

[2375] Homework Help

[2376] overview

[2377] Users (children) can input questions about their homework through a smartphone app and receive appropriate answers. The system analyzes the questions entered by the user and generates and provides appropriate answers. It also utilizes an emotion engine that recognizes the user's emotions and responds based on the user's emotional state.

[2378] Specific operation example

[2379] 1. The user launches the smartphone app and selects the "Homework Help" menu.

[2380] 2. The user types the text "I don't know how to solve the equation" and taps the "Submit" button.

[2381] 3. The device receives the user's question and converts it into JSON format.

[2382] 4. The emotion engine recognizes emotions from the user's facial expressions and voice and generates emotion data.

[2383] 5. The device sends the question and emotion data to the server as an HTTP POST request.

[2384] 6. The server receives the request and analyzes the question and sentiment data, for example using natural language processing to understand the question and tailor the answer taking into account the sentiment data.

[2385] 7. The server generates JSON data including the URL of the appropriate explanatory article and specific calculation methods.

[2386] 8. The server sends the generated JSON data to the terminal as an HTTP response.

[2387] 9. The device receives the response and displays it visually to the user.

[2388] 10. The user taps on the link to the explanatory article displayed to view a detailed explanation.

[2389] Communication partner

[2390] overview

[2391] While the user is home alone, they can chat with a virtual communication partner through a smartphone app. The system analyzes the user's input messages, generates and provides appropriate replies, and uses an emotion engine to respond based on the user's emotional state.

[2392] Specific operation example

[2393] 1. The user launches the smartphone app and selects the "Communication Partner" menu.

[2394] 2. The user types the text "Tell me what you'd like to do today" and taps the "Send" button.

[2395] 3. The device converts the message into JSON format and sends it to the server as an HTTP POST request.

[2396] 4. The emotion engine recognizes emotions from the user's facial expressions and voice and generates emotion data.

[2397] 5. The device sends the message content and emotion data to the server.

[2398] 6. The server receives the request and analyzes the message content and emotional data, for example using natural language processing techniques to generate an appropriate reply and adjust the reply content taking the emotional data into account.

[2399] 7. The server generates a JSON response containing the suggestion, "How about playing Animal Crossing?"

[2400] 8. The server sends the generated JSON data to the terminal as an HTTP response.

[2401] 9. The device receives the response and presents it to the user visually or audibly.

[2402] 10. The user sees the suggestions and chooses to play Animal Crossing.

[2403] This system allows children who are alone at home to receive support with their schoolwork and avoid feeling lonely through conversations with virtual communication partners. Furthermore, the introduction of an emotion engine enables flexible responses based on the user's emotional state, providing more personalized support.

[2404] The processing flow will be explained below.

[2405] Homework Help

[2406] Program processing

[2407] Step 1:

[2408] The user launches the smartphone app and selects the "Homework Help" menu.

[2409] Step 2:

[2410] A user texts in a homework question, for example, "I don't know how to solve the equation."

[2411] Step 3:

[2412] The user taps the "Send" button.

[2413] Step 4:

[2414] The terminal receives the user's question.

[2415] Step 5:

[2416] In order for the device to analyze the user's facial expressions and voice and recognize emotions, it activates an emotion engine and acquires emotion data.

[2417] Step 6:

[2418] The device converts the question content and emotion data into JSON format.

[2419] Step 7:

[2420] The device sends the converted JSON-formatted question and emotion data to the server as an HTTP POST request.

[2421] Step 8:

[2422] The server receives the HTTP POST request, analyzes the question and emotional data, and uses natural language processing technology to understand the intent of the question and generate an answer that takes the emotional data into account.

[2423] Step 9:

[2424] The server searches a database or calls external APIs to generate an answer that fits the question, for example adding detailed explanations or encouraging messages if the user is confused.

[2425] Step 10:

[2426] The server generates JSON data containing the URL of the appropriate explanatory article and the specific answer.

[2427] Step 11:

[2428] The server sends the generated JSON data to the terminal as an HTTP response.

[2429] Step 12:

[2430] The device receives the HTTP response and parses the JSON data.

[2431] Step 13:

[2432] The device displays the analyzed data in a format that the user can visually confirm, such as a link to an explanatory article or a message of support.

[2433] Step 14:

[2434] The user taps the link to the explanatory article displayed to check the detailed content.

[2435] Communication partner

[2436] Program processing

[2437] Step 1:

[2438] The user launches the smartphone app and selects the "Communication Partner" menu.

[2439] Step 2:

[2440] The user types the text "Tell me what you'd like to do today" into the chat window.

[2441] Step 3:

[2442] The user taps the "Send" button.

[2443] Step 4:

[2444] The terminal receives the user's message.

[2445] Step 5:

[2446] In order for the device to analyze the user's facial expressions and voice and recognize emotions, it activates an emotion engine and acquires emotion data.

[2447] Step 6:

[2448] The device converts the message content and emotion data into JSON format.

[2449] Step 7:

[2450] The device sends the converted JSON-formatted message content and emotion data to the server as an HTTP POST request.

[2451] Step 8:

[2452] The server receives the HTTP POST request, analyzes the message content and emotional data, and uses natural language processing technology to understand the intent of the message and generate a reply that takes the emotional data into account.

[2453] Step 9:

[2454] The server searches a relevant database to generate an appropriate reply, and generates suitable reply suggestions, such as activity suggestions if the user is bored.

[2455] Step 10:

[2456] The server generates a JSON response containing the suggestion, "How about playing Animal Crossing?"

[2457] Step 11:

[2458] The server sends the generated JSON data to the terminal as an HTTP response.

[2459] Step 12:

[2460] The device receives the HTTP response and parses the JSON data.

[2461] Step 13:

[2462] The device presents the analyzed data to the user visually or audibly, such as suggestion messages or specific steps for an activity.

[2463] Step 14:

[2464] The user sees the suggestions and chooses to play Animal Crossing.

[2465] Example 2

[2466] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2467] In situations where children in dual-income households are left home alone, there is a need to provide appropriate support for homework questions and communication methods to reduce feelings of loneliness. It is also important that this support and communication take into account the child's emotional state, and personalized responses are required.

[2468] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for inputting homework questions; means for analyzing the input questions and generating appropriate answers; means for providing the generated answers to the user; means having an engine for recognizing the user's emotions and generating emotion data; means for transmitting the questions and emotion data and receiving answers from the server; means for inputting chat messages while the user is at home alone; means for analyzing the input messages and generating appropriate replies; means for providing the generated replies to the user; means for transmitting the message contents and emotion data and receiving the replies from the server; means for converting the questions and messages into a data format and transmitting them; and means for analyzing the answers and replies in data format from the server and displaying or playing them back to the user. This allows children to receive support with their homework while at home alone and helps them feel less lonely through virtual communication.

[2469] The "means for inputting homework questions" is a system component that provides an interface for users to input homework questions as text.

[2470] The "means for analyzing the entered question and generating an appropriate answer" is a component of the system that uses natural language processing technology to analyze the homework question entered by the user and generate the optimal answer to that question.

[2471] A "means for providing a generated answer to a user" is a system component that provides an interface for presenting the system-generated answer to a user visually or audibly.

[2472] "Means having an engine for recognizing a user's emotions and generating emotional data" refers to a system component that includes an engine that uses a camera or microphone to analyze a user's facial expressions and tone of voice, identify their emotional state, and generate emotional data.

[2473] The "means for sending question content and emotion data to a server and receiving a response from the server" refers to a system component that sends a question input by a user and generated emotion data to a server via an HTTP POST request and receives the response data received from the server.

[2474] The "means for inputting chat messages while the user is away" is a system component that provides an interface for the user to input chat messages in text format.

[2475] The "means for analyzing input messages and generating appropriate replies" is a component of a system that uses natural language processing technology to analyze chat messages input by users and generate optimal replies to those messages.

[2476] A "means for providing a generated reply to a user" is a system component that provides an interface for presenting a system-generated reply to a user visually or audibly.

[2477] "Means for transmitting message content and emotion data to a server and receiving a reply from the server" refers to a system component for transmitting message content entered by a user and generated emotion data to a server and receiving reply data received from the server.

[2478] "Means for converting questions and messages into a data format and sending them to the server" refers to a system component that converts questions and messages entered by users into a data format such as JSON and sends them to the server.

[2479] "Means for analyzing answers and replies in data format from the server and displaying or playing them back to the user" refers to a system component that analyzes answers and replies provided in data format such as JSON received from the server and displays or plays them back to the user visually or audibly.

[2480] The present invention combines a smartphone application system that provides homework help and communication partners for children in dual-income households who are left home alone with an emotion engine that recognizes the user's emotions. Specific embodiments of the present invention are described below.

[2481] Homework help feature

[2482] Basic configuration

[2483] This system uses a smartphone app to input questions related to homework and provides a series of functions to obtain appropriate answers. The main elements are a means for inputting the user's question, a means for analyzing the question and generating an appropriate answer, a means for providing the generated answer to the user, and a means for generating emotion data using an emotion engine. This system uses HTTPS communication to send the user's question and emotion data to the server and receive the answer from the server. It is also configured to convert the question and emotion data into JSON format and send it, and to analyze the answer received from the server and display it to the user.

[2484] Operation overview

[2485] 1. The user launches the smartphone app and selects the "Homework Help" menu.

[2486] 2. The user types the text "I don't know how to solve the equation" and taps the "Submit" button.

[2487] 3. The device converts the input question into JSON format and uses an emotion engine to recognize emotions from the user's facial expressions and voice, generating emotion data. The emotion engine can use, for example, Microsoft Azure's emotion analysis API.

[2488] 4. The device sends the question and emotion data to the server as an HTTP POST request, using a communication library such as OkHttp.

[2489] 5. The server analyzes the received request and generates an answer appropriate to the question. For natural language processing, it uses an NLP model, such as Hugging Face.

[2490] 6. The server generates a response and sends it to the device in JSON format.

[2491] 7. The terminal analyzes the received response and displays it visually to the user.

[2492] Examples of concrete examples and prompts

[2493] Examples:

[2494] User types, "Help me with my math homework today!"

[2495] The device uses an emotion engine to recognize when the user is feeling stressed.

[2496] The server provides detailed explanatory articles in a gentle tone.

[2497] Example prompt sentence:

[2498] If your child types "Help me with my math homework today" into your app, generate an appropriate response that takes into account the stress your child may be under.

[2499] Communication partner function

[2500] Basic configuration

[2501] This system provides a function that enables users to chat with virtual communication partners through a smartphone app while they are home alone. The main elements are a means for inputting messages, a means for analyzing the message content and generating an appropriate reply, a means for providing the generated reply to the user, and a means for generating emotion data using an emotion engine. The system is configured to send the message content and emotion data to a server, analyze the replies received from the server, and display or play them back to the user.

[2502] Operation overview

[2503] 1. The user launches the smartphone app and selects the "Communication Partner" menu.

[2504] 2. The user types the text "Tell me what you'd like to do today" and taps the "Send" button.

[2505] 3. The device converts the message content into JSON format and sends it to the server as an HTTP POST request.

[2506] 4. The emotion engine generates emotion data from the user's facial expressions and voice. For emotion analysis, it uses the IBM Watson API, for example.

[2507] 5. The device sends the message content and emotion data to the server.

[2508] 6. The server analyzes the received request and generates an appropriate reply, for example using OpenAI GPT-3.

[2509] 7. The server sends the appropriate reply in JSON format to the device.

[2510] 8. The device analyzes the reply received and presents it to the user visually or audibly.

[2511] Examples of concrete examples and prompts

[2512] Examples:

[2513] A user types, "What would be fun to do today?"

[2514] The device uses its emotion engine to recognize that the user is feeling a little lonely.

[2515] "Animal Crossing is a fun game," the server suggested.

[2516] Example prompt sentence:

[2517] When a child types into your app, "What would be fun to do today?", consider that they might be feeling a little lonely and generate fun suggestions. Include popular games and activities in your suggestions.

[2518] This system allows children who are alone at home to receive support with their schoolwork and avoid feeling lonely by interacting with virtual communication partners. The introduction of an emotion engine also enables flexible responses based on the user's emotional state, providing more personalized support.

[2519] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2520] Homework help feature

[2521] Step 1:

[2522] The user launches the smartphone app and selects the "Homework Help" menu.

[2523] Specific operation: The user taps the app from the smartphone home screen to launch it, and selects the "Homework Help" menu on the initial screen.

[2524] Input: User taps.

[2525] Output: Homework help menu will be displayed.

[2526] Step 2:

[2527] The user types the text "I don't know how to solve the equation" and taps the "Submit" button.

[2528] Specific actions: The user uses the smartphone keyboard to enter a question into the text box and taps the "Send" button.

[2529] Input: The question entered by the user is "I don't know how to solve an equation."

[2530] Output: The question is sent to the app.

[2531] Step 3:

[2532] The device converts the entered question into JSON format.

[2533] What it does: The app on the device reads the user's text input and converts it to JSON format.

[2534] Input: The question entered by the user is "I don't know how to solve an equation."

[2535] Data processing: Converting to JSON format (e.g., {"question": "I don't know how to solve the equation"}).

[2536] Output: Data converted to JSON format.

[2537] Step 4:

[2538] The emotion engine recognizes emotions from the user's facial expressions and voice and generates emotion data.

[2539] Specific operation: Using the device's camera and microphone, the system analyzes the user's facial expressions and voice to generate emotion data. For example, it uses Microsoft Azure's emotion analysis API as an emotion engine.

[2540] Input: User facial and voice data obtained from the camera and microphone.

[2541] Data operations: Identify emotional states and generate emotion data (e.g., {"emotion": "frustrated"}).

[2542] Output: Emotion data.

[2543] Step 5:

[2544] The device sends the question and emotion data to the server as an HTTP POST request.

[2545] Specific operation: The device sends the generated question and emotion data together to the server. A communication library (e.g., OkHttp) is used for communication.

[2546] Input: Question and emotion data in JSON format (e.g., {"question": "I don't know how to solve the equation", "emotion": "frustrated"}).

[2547] Output: An HTTP POST request is sent to the server.

[2548] Step 6:

[2549] The server receives the request and analyzes the question and emotion data.

[2550] What happens: The server receives the request and uses natural language processing techniques (e.g., the Hugging Face NLP model) to analyze the question and sentiment data.

[2551] Input: Submitted question and sentiment data.

[2552] Data Computing: Uses natural language processing to understand questions and tailor answers based on sentiment data.

[2553] Output: The appropriate answer data (e.g., an explanation of how to solve the equation).

[2554] Step 7:

[2555] The server generates JSON-f...

Claims

1. a means for inputting homework questions; A means for analyzing the input question content and generating an appropriate answer; means for providing the generated answer to a user; A system including:

2. means for inputting chat messages while the user is away from home; means for parsing the input message and generating an appropriate reply; means for providing the generated reply to the user; The system of claim 1 further comprising:

3. A means of converting questions and messages into JSON format and sending them to the server; A means for parsing JSON responses and replies from the server and displaying or playing them back to the user; The system of claim 1 or 2, further comprising:

Images